Just as we've seen the loneliness epidemic steadily grow in the developed world, largely attributed to the development of technology [1], so, too, will the integration of advanced artificial intelligence systems into all parts of society by default lead to an initially nearly imperceptible reduction in human empowerment as we come to depend ever more on them. This process is likely to be difficult to notice or even measure, yet nonetheless lead to a shift in the balance of control, for the first time in our species' modern existence. Given that human empowerment in democratic nations comes in large part from human labor, we should expect that the automation of most jobs would lead to an even greater economic disparity and power imbalance between the rich and the poor. This process is likely to increase the risk of coups and democratic backsliding [2]. For a more in-depth discussion of the arguments as to why we should expect this, see Kulveit et al. 2025 [3].
The rapid development and integration of AI systems will require a not-before seen ramping up of enhanced robustness to a plethora of unforeseen catastrophic risks. Three broad categories of risk that I find useful when considering impacts from AI are misuse, misalignment, and systemic. Misuse refers to malicious actors—either individuals, groups, or states—using AI systems to achieve their nefarious objectives, whether via generating personalized misinformation, hacking their adversaries, or building powerful weapons. Misalignment results from AI models not truly obtaining human values, leading to nearly limitless possibility for catastrophe. Finally, systemic risks occur when our complex and vital societal systems become dependent on a technology which we don't fully understand nor control, leaving us vulnerable to the unpredictable decisions of those systems.
In order to strengthen our defenses against these risks, we will need to devote significant amounts of capital and effort in the coming years. There is reason to believe that AI systems with human-level capabilities, often referred to as artificial general intelligence, or AGI, may be developed within the coming 5–10 years, with many estimates converging around the year 2032 [4]. When a generally-capable AI system is developed, we should expect the model developers to use the system to develop the next versions due to the immense potential economic value of doing so [5]. This will lead to a rapid development of AI capability, sometimes referred to as an "intelligence explosion", which will . Even if initially slowed down by various bottlenecks like human approval and limited compute and energy supplies, the incentives to solve these will be so great that we should expect the pressure to force
Whether or not the current tsunami of investment in and development of AI systems, reaching around $1.8T in 2025 and expected to pass $2.5T this year, a 44% increase [6], will be regarded as a "bubble", the advancements we have today are already sufficient to significantly disrupt our societies – by giving powerful tools that empowers bad actors, by providing biased and nonhuman "advice" which isn't in alignment with human values and which can be especially those most vulnerable and susceptible, and by , among many other. All while using unprecedented amounts of energy and materials, developed at breakneck speeds, often leading to environmental disasters (citation needed). Just the training of a relatively small model, GPT-3, is estimated to have generated around 552 tons of carbon dioxide [7].
However, while I believe that the risks posed by today's models are real, and in some cases significant, they are greatly overshadowed by the risks posed by potential future models. This concern comes in part from the sheer complexity of these alien minds, which even the creators of understand almost nothing of (citation needed). The best solutions these multi-billion dollar companies have produced for "aligning" frontier models with something resembling human values is fundamentally insufficient and has been shown to fail entirely time and time again. If we plan to entrust fundamental parts of our society such as education, healthcare, software development, and justice, to these systems, and if the pace of development continues as-is, we must simultaneously invest heavily in rapidly advancing the science of model alignment and societal preparedness.
Previous technological revolutions have happened at a pace which allows humanity to be able to gradually adapt laws, cultural norms, and education over the span of decades to slowly adjust. We have survived majorly disruptive changes such as the industrial revolution, printing revolution, and digitalization, thanks to this ability to adapt. However, the rate of change we can expect when many millions of digital minds are churning away 24/7 at superhuman speeds, as some call "A country of geniuses in a datacenter"[8], will simply be wholly unprecedented and demand significant a reorganization of many parts of society in a short period of time.
We will need to grapple with, and enact solutions for, a great number of fundamental problems in the coming years. How should governments tax "AI labor" in a just way? At what point should artificial minds be considered for having rights, and in what form?
So, what can be done to address the risks described above? While some argue for a global pause or slowing down of development of frontier AI systems [9], I don't personally believe such a pause is possible nor enforceable. However, we have managed to successfully regulate many powerful technologies in the past, and I remain optimistic that government regulation at both the international and national level will be crucial for success. Additionally, I believe that we need to dramatically increase investment in AI interpretability and control, requiring companies releasing powerful models to do rigorous safety testing prior to release. Finally, we need to greatly improve societal robustness to the types of risk posed by advanced artificial intelligence, through hardened cybersecurity, pandemic prevention, regulation limiting, and tracking dependence on AI systems.
We need organizations like METR and Epoch AI, publishing Frontier Risk Reports [10] and measuring and forecasting AI progress [11], and more research in line with the Anthropic Economic Index report [12], tracking usage and effect of AI on all parts of society.
See Harvard GSE: What is causing our epidemic of loneliness? ↩︎
AI-Enabled Coups: How a Small Group Could Use AI to Seize Power, Davidson et al.: https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power#42-conventional-coups-and-backsliding ↩︎
Gradual Disempowerment, Kulveit et al. January 2025: https://gradual-disempowerment.ai ↩︎
TODO: find other sources https://www.metaculus.com/questions/5121/when-will-the-first-general-ai-system-be-devised-tested-and-publicly-announced/ ↩︎
From AGI to Superintelligence: the Intelligence Explosion, from Situational Awareness, Leopold Aschenbrenner, June 2024, https://situational-awareness.ai/from-agi-to-superintelligence/ ↩︎
AI Infrastructure Drives AI Spending; Adds $401 Billion in Spending as Technology Providers Continue to Build Out AI Foundations, Gartner, January 2026, https://www.gartner.com/en/newsroom/press-releases/2026-1-15-gartner-says-worldwide-ai-spending-will-total-2-point-5-trillion-dollars-in-2026 ↩︎
The Climate and Sustainability Implications of Generative AI, Bashir et al., March 2024: https://mit-genai.pubpub.org/pub/8ulgrckc/release/2 ↩︎
Machines of Loving Grace, Dario Amodei, October 2024, https://www.darioamodei.com/essay/machines-of-loving-grace ↩︎
PauseAI: https://pauseai.info ↩︎
METR Frontier Risk Report: https://metr.org/blog/2026-05-19-frontier-risk-report ↩︎
Epoch AI Global AI computing capacity is doubling every 7 months: https://epoch.ai/data-insights/ai-chip-production ↩︎
Anthropic Economic Index: https://www.anthropic.com/research/economic-index-primitives ↩︎