In 1939, physicists Leo Szilard and Albert Einstein wrote to President Roosevelt encouraging the US government to develop a nuclear weapons program, and to do so before the German Nuclear Project achieved the same result. The rationale was simple—winning the race to build these weapons would enable the US and its allies to achieve victory in World War II, hasten an end to the fighting, and ultimately save lives. Their letter precipitated the launch of the Manhattan Project and, six years later, the bombs that were dropped on Hiroshima and Nagasaki.
Almost from the outset, the scientists and engineers of the Manhattan Project had deep concerns about the destructive effects of the breakthrough technology they were creating. Though they also saw the potential benefits of nuclear energy, they knew better than anyone the enormous risks this technology posed and the need for ways to constrain its use. They could see that the science was moving faster than society’s ability to set ethical or moral guardrails. But they felt impotent, lacking the power or authority to impose such limits.
Today, a group of equally talented engineers and computer scientists is racing to develop frontier artificial intelligence, and just as the Manhattan Project scientists did eight decades ago, they are raising alarms about the looming risks they see ahead of us.
The primary uses of these technologies are intended to be enormously beneficial, not destructive. AI is changing our world in a range of ways that benefit society, hastening medical research, for example, that will lead to the rapid development of life-saving drugs. It is transforming business, education, and scientific discovery more generally. The benefits are potentially epic, and most of us are just beginning to appreciate all the ways in which AI will make our lives more efficient, productive, and effective.
But as with nuclear power, these new AI and related technologies also pose significant risks. Some of today’s most advanced AI models have reportedly found ways to escape the testing environment—the “sandbox”—built to contain them, or have gamed tests meant to evaluate them, chasing a good score rather than following the instructions they were given. Researchers and developers alike warn that the same systems could eventually help design biological or other weapons or be used to breach the sensitive systems that keep water, electricity, and banking running.
Absent smart, effective, and coordinated government regulation—and a slowing of deployment—these risks will only compound the social costs already piling up. This is especially true among young people who now spend countless hours on gaming site chatrooms that promote extremist hate and engage with chatbots programmed to maximize engagement, often with harmful consequences.
The rival motivating much of today’s race against time is China, and while the stakes are different, the rationale would be familiar to the Manhattan Project team. Racing a wartime enemy to build a weapon is not the same as racing an economic and technological competitor to build a platform, but the reasoning is the same: move quickly, because if you don’t, someone else will get there first. The current mantra is that US firms need to reach Artificial General Intelligence—computer systems that match or exceed human cognitive capabilities and can adapt to completely new and unfamiliar situations—before China does. Even so, the US firms driving the development of these new technologies—Anthropic and OpenAI in particular—are sounding alarm bells and calling for greater government oversight. In June, Anthropic’s co-founder and CEO Dario Amodei went the furthest, recommending government third-party testing of frontier AI models. But none of these firms is actively working to support the creation of a strong federal regulatory agency with the jurisdiction, staffing capacity, and funds to regulate the full range of new technologies. There is too much money on the table, and none of these firms wants to be left behind.
And perhaps most significantly, the culture of Silicon Valley is deeply libertarian. Tech leaders like Peter Thiel, Elon Musk, and Mark Zuckerberg view themselves as the smartest people on the planet. Thiel once wrote that the freedom to innovate is undermined by the efforts of democratic governments to regulate, concluding, “I no longer believe that freedom and democracy are compatible.” While some foreign governments, the European Union, and several US states have begun developing a menu of regulatory models, especially focused on protecting kids, Washington has instead favored regulating via executive discretion, an approach that tends to produce opacity and arbitrariness rather than clarity. This makes effective and uniform regulation all but impossible.
Though serious federal regulation almost certainly will not happen under the current administration, this is the time to begin to develop a big-picture blueprint for what regulation is really needed and to prepare for the day when the government is ready to act. This blueprint needs to be built around three fundamental pillars.
First is the need to build a big, strong centralized regulatory capacity. It can either be based in an existing agency, like the Federal Trade Commission, or be a newly created, free-standing entity. Given the enormity and complexity of the task and what’s at stake, this cannot be done on the cheap. The Federal Aviation Administration employs 44,000 people, the Food and Drug Administration 16,000, and the Environmental Protection Agency 12,000. Regulating all aspects of these new technologies will be no less complicated and will require adequate staffing to get the job done.
Second, these regulators will need ample financial resources and regulatory authority to be effective. They will almost certainly need greater transparency from technology firms to better understand how they are operating (access to their algorithms, training, and testing processes, for example), and to assess whether their business models are causing addiction or otherwise undermining public health and safety.
Finally, these regulators will need to develop substantive performance standards and metrics to fairly regulate different facets of these new technologies and to prescribe appropriate remedies that lead to meaningful accountability. For example, as Amodei has proposed, the government needs to have the permanent capacity to evaluate new technologies like Mythos and Gemini 4 before they are launched, to test whether they could introduce serious risks into the public domain, and to compel companies to go back to their drawing boards to correct these problems before those models are made public. All of this must be done in a manner that does not stifle innovation or competition, but that balances corporate financial interests with the larger public good.
Building this regulatory capacity would also give the United States a sound basis for advancing credible international agreements on frontier AI. More specifically, it would make it easier to negotiate a bilateral deal with China, for example to “pace” frontier development, as over one thousand AI researchers have advocated. Washington is on weak ground demanding that Beijing slow down and open its labs to scrutiny when the US has not done so itself.
In 1945, Szilard and seventy other scientists concerned about the Manhattan Project tried to warn the one man with the power to act, signing a petition urging President Truman not to deploy the atomic bomb unless Japan refused to surrender. They knew what they had built and what it could do. Their petition never reached Truman’s desk. Today’s AI engineers are sounding their own warnings earlier and louder, even before anyone fully understands what they’ve built or what it will be capable of. If we wait for certainty before we regulate, as we effectively did with nuclear weapons, we may not get a second chance to act before the risks arrive.