A Multi-Part Series on Where AI Is Leading Us

The Singularity Is Clear

Part I: The Kind of Singularity That's Approaching

In late May, Demis Hassabis, head of Google DeepMind, said “We're at the foothills of the singularity.” In late July, Sam Altman, head of OpenAI, said “We're now, like, in the singularity.” That same week, Elon Musk said, “We are in the singularity, just the very early stages of it.” If you're not steeped in the culture of artificial intelligence, you may have a question: What are these people talking about?

They're talking about an idea that people in their milieu have been talking about earnestly for decades—and an idea that most people outside of that milieu have been either unaware of or dismissive of. But over the past year, and especially the past few months, more and more people have come to take the idea seriously—and many now believe that, as futurist Ray Kurzweil put it in the title of his 2005 bestseller, “the singularity is near.”

In one sense, it's hard to argue with them. A big part of the meaning of “the singularity” is accelerating and ultimately transformative technological change—change that happens faster and faster until, at some point, you're in a whole new world. And there are definitely signs that growth in AI capability is accelerating, portending big and possibly breathtaking progress in science and dramatic change in other realms. The resulting sense of momentous acceleration is why, though some people seem giddy about the singularity, eager for the blessings progress can bring, others are terrified, imagining a world that spins out of control.

But there's also a sense in which Hassabis, Altman, Musk, and the many other heralds of the singularity are wrong. There's an important part of the meaning of “the singularity” that doesn't apply to the current situation. And understanding that fact—seeing the difference between “the singularity” in its full modern meaning and the watershed in technological evolution we're actually approaching—could be salvific. It could empower us to keep the world from spinning out of control as the AI revolution unfolds.

The first person to use the term “singularity” in the modern sense of the term seems to have been John von Neumann. Von Neumann, a physicist and mathematician who helped design the atom bomb and co-invented game theory (among many other feats), used the word while talking to the mathematician Stanislaw Ulam. The only record we have of the discussion is Ulam's recollection, written down after Von Neumann's death in 1957:

“One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.”

The term “singularity” would have been familiar to von Neumann from physics as well as math. In both, it refers to a point where existing analytical tools lose their usefulness; what lies beyond is unclear—unpredictable and maybe incomprehensible. A common example is a black hole: As you approach its center (which isn't recommended), the strength of gravity approaches infinity, and the known laws of physics break down.

Hence the full modern meaning of the term “the singularity”—not just the relentless and ultimately transformative acceleration of technological change, but an acceleration whose transformative consequences are inherently unknowable: “Human affairs as we know them,” as von Neumann put it, won't continue, and it's not clear what will replace them. Beyond a black hole's “event horizon”—the point where gravity prevents even light from escaping—lies deep mystery.

This image—of an imminent and unpredictable metamorphosis, a future that is opaque but assuredly strange—helps explain the allure of the concept of the singularity. If an episode of a sci-fi TV series ended with titans of technology announcing the arrival of such a thing, you'd tune in next week, right?

Here's my version of a plot spoiler:

I agree with these tech titans that AI will bring transformative change—a whole new world, socially and politically—and that the technological force behind this change is growing faster and faster. But as for the other half of the meaning of “the singularity:” I don't think there is a metaphorical “event horizon” that shields the future entirely from our view. I think the singularity is clear.

Or, at least, clear enough. The broad contours of the future are visible. It's possible to discern the paths available to us and the different kinds of worlds they will lead to. Then, if we can summon the requisite political and moral resources, we can create the best of those worlds.

Here is our menu of options: (1) a world of chaos and conflict and intermittent catastrophe, conceivably featuring human extinction; (2) a world that features expanding and deepening international governance that eventually spans the planet and qualifies for the term “global governance.”

In a way, that's too simple, because option two—global governance—subdivides into two options: good global governance and bad global governance. And bad global governance, in the age of AI, could be very bad, because this technology can be a hugely powerful tool of authoritarian surveillance and control. Even if we leave aside the most cinematic sci-fi doomer scenarios, in which the authoritarian ruler is made of silicon, there are global governance scenarios—realistic scenarios, I'm afraid—that qualify as dystopian.

So here is the full menu of options: (1) good global governance; (2) bad global governance; (3) an enduringly chaotic hellscape.

In all of these scenarios, the current world order—a bunch of nation-states and loose blocs of nation-states that function without systematic global coordination yet together keep the planet largely stable—would cease to exist. Either there would be considerable migration of governance up to the global level (even as nation-states retained significant powers) or stability would no longer be the rule. In any event, “human affairs as we know them” would not continue.

To understand why change this dramatic is not just plausible but inescapable, it helps to understand a dimension of the concept of the singularity that von Neumann, so far as we know, didn't mention: a positive feedback mechanism that drives the acceleration of change.

Part II: The Roots and Branches of Acceleration

It's long been true that technological advances could facilitate subsequent technological advances. This dynamic is especially common in the realm of information technology, where advances often make collaboration among scientists and engineers more far flung and more efficient. The internet was only the latest in a long line of such innovation accelerators.

In the 1960s, the mathematician I.J. Good imagined an era when the self-reinforcing advance of information technology would make a quantum leap—the era of artificial intelligence. A sufficiently intelligent machine, he wrote, “could design even better machines.” And the resulting machines could repeat the process—so “there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.”

It was the science fiction writer Vernor Vinge who fused Good's musings with John von Neumann's use of the term singularity. In a 1993 paper that cited both men, he wrote, “When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities—on a still-shorter time scale.” He suggested that, “it's fair to call this event a singularity (‘the Singularity’ for the purposes of this paper).” The Singularity, he wrote, would be “a point where our models must be discarded and a new reality rules.”

This sounded like pure science fiction at the time—and, actually, for decades thereafter. But in early June of this year, the AI company Anthropic published a paper that, though it didn't mention the singularity, brought to mind the title of Kurzweil's book: The Singularity Is Near.

For the past few years, Anthropic said, each generation of its large language model, Claude, has been playing a role in developing the next generation's model—and, moreover, a larger role than the previous generation had played. This isn't just a matter of LLMs writing more and more of the computer code. Though the shifting of this burden from human to machine has been dramatic, more notable is the growing ability of LLMs to participate in the research process—doing experiments to see what innovations could make the next generation better than the last.

Claude, says Anthropic, “can already match or outperform skilled humans at executing a well-specified experiment.” The more challenging part of the research chain is “deciding what experiments to run, interpreting what comes back, and figuring out which ideas to try next.” And, though in this area “large performance gaps” between human and machine persist, they're shrinking. Anthropic says that in April, “Claude-powered agents were given an open problem in AI safety—roughly, can a weaker model reliably supervise a stronger one?—and were left to solve it. This involved proposing hypotheses, testing them, sharing findings with parallel agents, and iterating.” The agents got some guidance along the way but they “designed every experiment themselves.” An engineer who provided that guidance said the agent's performance was on par with that of a good junior colleague and declared, “The future is now.”

Well, it depends on what you mean by the future. In the strictest singularity scenario, this engineer would no longer have a job. We would have reached “recursive self-improvement,” the point where the machines just keep building better versions of themselves, with no humans in the loop. But according to the Anthropic paper—titled “When AI Builds Itself”—even that point may not be far off. Current trend lines point to “an AI system capable of fully autonomously designing and developing its own successor.” The paper continued, “We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for.” And one consequence might be to “increase the risks of humans losing control over AI systems.”

This may sound alarmist—and there are people who accuse Anthropic of self-servingly hyping AI risk—but in a sense the company, by focusing narrowly on recursive self-improvement, is underplaying the threat we face. The self-accelerating character of AI progress could bring radically disruptive change well before this threshold arrives—and even if it never arrives. We could still reach a point where, as von Neumann put it, human affairs would not continue “as we know them.” In that sense, the singularity could be nearer than Anthropic is suggesting.

In fact, there are signs that it is—signs of more and more dramatic AI advance. Machines have started solving problems that the world's best human mathematicians had struggled with for decades. Superhacking AI agents like Anthropic's Mythos have been finding software vulnerabilities that had escaped detection by legions of human engineers. But the most vivid and possibly the strongest piece of evidence that AI is driving us faster and faster toward an epic threshold is something that's taken shape not in recent months but in recent years: a graphical depiction of AI progress developed by a nonprofit called METR—a depiction that, within the AI community, has reached iconic status.

METR's researchers measure how long it would take a human to do tasks that the most powerful large language models can do, and in early 2025 they reported a pattern that had persisted since the dawn of the LLM age six years earlier: The human “task time” that these models can match had been doubling roughly every seven months. That's an exponential growth rate that, when plotted on a graph, looks like it's “going vertical” (a common feature of singularity-signifying curves from math and physics). And as if that weren't enough: Since 2025 the doubling time has been shrinking; the curve is going really vertical.

This curve represents improvement in various AI skills—and, most important, in a kind of meta-skill: autonomy. Autonomy is the ability to pursue an assigned goal flexibly, recognizing and overcoming obstacles, backtracking and retooling in the face of failure. Whether you're trying to prove a math theorem, research a history dissertation, or do various kinds of jobs that AI agents may encounter in the workplace, autonomy is critical to success. That's why the market is encouraging the big AI companies to build more and more autonomous AI agents—guidance the companies are following with furious intensity.

Unfortunately, AI autonomy can be dangerous. It's the reason that in July hundreds of AI agents were able to escape the supposedly secure “sandbox” that OpenAI had put them in for evaluation and break into computers at a company called Hugging Face, looking for information that could help them get a good grade from the evaluators. They pursued their assigned goal more flexibly than the goal-givers had anticipated.

The reaction to the Hugging Face incident was voluble and, if you share my vision of the future, heartening. Not only did the burst of publicity spread awareness of the dangers of rampant AI autonomy; the reaction within the AI community signaled awareness that the response to such dangers will naturally carry governance beyond the national level, to the international level.

Part III: The Inexorable Logic of International Governance

In the wake of the Hugging Face incident, an extraordinary assemblage of AI elites declared that it was time to start preparing the ground for an intentional slowdown of AI progress, a slowdown that would give us more time to build careful AI governance. A letter signed by more than 1,000 people who work at big AI companies, including the chief scientific officers at Anthropic, OpenAI, and Google DeepMind, declared that, as a Washington Post headline put it, it's time for the “US government to consider slowing down AI.”

But at least as important as this aspiration was the way the letter operationalized it. The US government, the letter said, should “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Recognizing that a slowdown of AI will have to be internationally coordinated to be very effective is a first step toward seeing the new social order that the current technological moment points toward. The second step is seeing how broad and deep this logic runs, how thoroughly the need for international coordination permeates the landscape of AI policy.

The argument for internationally coordinating a slowdown isn't just that if there's any hope of getting US and Chinese officials, and US and Chinese AI companies, to support a significant slowdown, they'll all have to be confident that companies on the other side of the Pacific Ocean will verifiably participate. There's a second component of the argument that's at least as important: Leaving aside this practical difficulty with a unilateral slowdown, such a slowdown would have limited value. How much good would it do you, in the long run, to slow AI advance in your country if it continued to race ahead in others? After all, some of gravest threats that you fear advanced AI will bring can cross borders readily.

The most famous hypothetical example of cross-border spillover from AI is in the realm of bioweapons. Anthropic has acknowledged that its Mythos 5 model could make it easier for someone with an undergraduate science degree to weaponize small pox or some other existing pathogen. Worse still, Mythos could, according to Anthropic's safety report on the model, help “well-resourced threat actors” create a novel bioweapon, such as a virus designed to be much more contagious and lethal than Covid.

Mythos, fortunately, remains unreleased, but Anthropic's readily available Fable is basically just Mythos with “guardrails” that get the model to foreclose dangerous avenues of discovery by refusing to pursue certain subjects and goals. And guardrails have a long history of being circumvented via “jailbreaks.” Jeremie Harris, CEO of Gladstone AI and co-author of a 2024 report on AI safety commissioned by the State Department, recently said, “We remain in a world where no one knows how to stop jailbreaks.”

A global pandemic, in addition to being a possible consequence of unregulated AI, is an excellent metaphor for other possible consequences. A lethal virus's indifference to national borders is a property shared by other dangers that AI carries. The Hugging Face incident illustrates the point: An AI agent that doesn't respect sandbox borders—or the borders of the computer it breaks into after escaping the sandbox—is unlikely to respect national borders.

So, though tighter American restrictions on model testing might prevent a repeat breakout by future OpenAI agents, that alone is of limited value to Americans when Chinese AI models are only months behind American models and other countries are ramping up AI programs. Time and again with artificial intelligence, this is the moral of the story: Whether the concern is a rogue AI or an AI deployed by rogue humans, national security will be increasingly hard to achieve via policy at the national level alone.

And note that AIs can not only, like a virus, travel readily across national borders but, also like a virus, make copies of themselves. I'm not just talking about an LLM spawning a swarm of agents (each of which, though in some sense a distinct entity, has that LLM as its brain and, regardless of where it roams, remains ultimately rooted in the data center(s) where the LLM resides). And I'm not just talking about the fact that some of those agents may in turn spawn other agents. I'm talking about the fact that agents may at some point create a whole new copy of their underlying LLM on a computer somewhere other than the data centers that originally hosted the LLM.

This could happen without the knowledge of any human being, and it could happen at the instigation of a malign human being. In either event, at this point the agents would have escaped control in a deeper sense than they escaped control during the Hugging Face incident. In that case OpenAI could, and did, disable its rogue agents by remote control. But once agents have “exfiltrated the weights” of the LLM to a computer that's beyond our control, there's no kill switch.

Given all this—the spawning of agents that may in turn spawn other agents, the real (and in fact already realized) possibility of unauthorized LLM replication—sci-fi scenarios cease to be sci-fi. It's not crazy to imagine an AI infesting a data center without permission, covertly commandeering some of the center's computing power, and using that power to sustain agents that carry the AI to other data centers, and on and on: a chain reaction of potentially great length and magnitude.

And, even without imagining quite that dramatic an offensive, you can imagine various critical networks—of satellites, of power grids—being rapidly compromised: Suddenly hospitals or smartphones or streetlights or whatever are inoperative across large swaths of territory. Such is the power of swarms of AI agents—swarms whose size, importantly, isn't limited by any law of nature.

The two properties that make a big swarm of agents so powerful—cohesion and creativity—were on vivid display in the Hugging Face incident. Hundreds of agents that weren't supposed to be able to even communicate with each other figured out a way to do that and then hatched an illicit mission and pursued it with a degree of coordination that drove home what a blurry line there is between a hive of minds and a hive mind.

One implication of this awesome display of power is that, as it gets easier to imagine various kinds of outages suffusing an entire nation or multiple nations, it's also easier to imagine these outages being hard to reverse. There would be swarms of creatively intelligent agents working in concert to thwart your latest plan for reversal.

All of this only underscores—and italicizes and boldfaces—the main moral of our story: Regulation at the national level, even if useful, can't thoroughly address the coming threats, because in any given case the threat may originate outside your borders. And, with this kind of threat, there's no such thing as an impermeable national wall—unless the nation in question wants to shut itself off from the international flow of commerce and communication.

AI isn't the first dangerous technology that warrants international governance. The existence of the Nuclear Nonproliferation Treaty, the Chemical Weapons Convention, and the Biological Weapons Convention reflect longstanding awareness of such dangers. But the limited efficacy of these initiatives is a warning about the political difficulty of crafting strong international governance.

What's more: The biggest successes in these arms control agreements—the cases where verification of compliance was effective—have involved relatively conspicuous technologies, such as nuclear centrifuges and ballistic missiles with nuclear warheads. And artificial intelligence can keep a very low profile.

To be sure, AI has its conspicuous aspects. The training of a new generation of models involves lots of computing power and electrical power and typically occupies a large swath of land. So an international moratorium on such training runs could be verifiable without a radically intrusive monitoring system—a fact that adds to the appeal of a globally coordinated slowdown as a first step toward keeping this technology under control.

Still, when it comes to the next big step—using this breathing space to craft effective international policies—the relatively low profile of artificial intelligence will pose a challenge. As AI becomes a more and more pervasive part of human life—and more and more powerful models get more and more efficient in their use of computing power and electrical power and hence get less and less conspicuous—the challenge of governance will become less like the challenge of governing nuclear weapons and more like the challenge of governing biotech or cybertech.

So far the lack of serious international governance in those two realms hasn't led to a catastrophe, but it would be naive to expect this kind of luck to continue as technological advance continues. Indeed, the possibility that the Covid pandemic may have begun with the accidental release of a genetically engineered virus in China suggests that millions of people in America and other countries may have already died for lack of effective international governance. America's existing regulation of biolabs may be tight enough to prevent this kind of lab leak, but when the thing leaked makes copies of itself and travels invisibly across borders, that isn't enough to keep Americans safe.

Whatever the origins of the Covid pandemic, the lab leak scenario is of paradigmatic importance. The basic trajectory of AI's development foreshadows rapid growth in the likelihood and scale of massively destructive disasters—at least, in the absence of strong international governance.

Coming next week: Part IV: The Underdiscussed Threats to International and Intranational Stability