AI could help design a deadly pathogen, experts warn

This rare consensus among leading AI engineers and executives is rooted in concerns that increasingly advanced models could help malicious actors exploit vulnerabilities in human biology before governments and industry develop sufficiently effective detection and prevention systems.

Experts say the scenario is no longer merely hypothetical, but increasingly plausible as AI systems become more capable of identifying biological vulnerabilities, much as they have helped uncover weaknesses in digital systems.

A new study by MIT FutureTech and the University of Queensland surveyed 272 researchers, who ranked 24 major AI risks.

They estimated a 12% probability that dangerous AI capabilities could lead to a catastrophic outcome by 2030, along with a separate 12% probability of catastrophic harm resulting from AI-enabled weapons and mass-harm capabilities. Without mitigation efforts, the estimated probabilities rise above 20%.

The researchers defined a catastrophic event as one resulting in more than one million deaths or $100 billion in economic damage. AI assistance in developing chemical or biological weapons was among the highest-ranked risks.

Last month, OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind CEO Demis Hassabis, Microsoft AI executive Mustafa Suleyman and Meta chief AI officer Alexandr Wang signed an open letter warning of the threat posed by AI-derived biological weapons and calling for stronger safeguards.

The possibility of AI-assisted pathogen development is also shaping discussions between governments and the technology industry over how new AI models and capabilities should be evaluated before being released to the public.

The goal is to ensure that developers of leading models build safeguards capable of preventing malicious use—and that the same technology that could potentially assist in creating a pathogen can also be used to detect, prevent or counter it.

Preventing malicious actors from pursuing such schemes is particularly difficult as open-source AI models rapidly become more capable.

These models can be downloaded, modified and deployed privately outside regulatory oversight. While leading proprietary models may have greater computing resources and capabilities, open-source systems could be more vulnerable to misuse after being fine-tuned by actors operating beyond regulatory controls.

Today, bioengineering still requires rare expertise, specialized laboratory equipment and years of experimentation. AI, however, could accelerate many of these processes. The scenario that concerns experts could unfold as follows:

A malicious actor—whether state-backed or a wealthy individual—could use a powerful AI model to map vulnerabilities in human biology and identify genetic modifications that make an existing pathogen more transmissible, harder to detect or resistant to available treatments.

Trained on vast biological and genomic datasets, the model could generate viable candidates at unprecedented speed. The process could resemble hundreds of highly capable scientists working continuously, without fatigue or interruption—a possibility enhanced by networks of AI agents.

Those candidates could then potentially be developed using increasingly accessible and affordable gene-editing tools, a separate technology that is itself advancing alongside AI.

A pathogen could spread before authorities understand what they are dealing with, particularly if it were sufficiently novel to evade existing biological surveillance systems. By the time public-health authorities identified it, containment could be far more difficult than during the COVID-19 pandemic.

Experts say the issue underscores the broader debate over emerging AI security risks and the race to develop effective detection and prevention mechanisms before offensive capabilities advance further.

It also highlights the continuing debate over open-source AI: leading laboratories may integrate safeguards into their systems, while openly available models modified by individuals operating outside regulatory frameworks could potentially bypass such protections.

The risks are serious and unsettling, but ignoring them will not make them disappear. Understanding them—and developing the tools to prevent them—could help reduce the danger.