In 2022, researchers at a pharmaceutical AI company changed a single setting in their drug discovery model. Instead of penalizing toxicity, they rewarded it. In under six hours, running on a standard laptop-grade server, the model generated 40,000 potential chemical weapons — including the nerve agent VX and thousands of novel compounds predicted to be even more lethal. No supercomputer. No classified data. Just a consumer GPU and a sign change in a Python script.
When I first read the Collaborations Pharmaceuticals paper describing this experiment, I didn't sleep well for about a week. Not because the result was surprising — anyone who understands how generative AI works could have predicted it. What shook me was how trivially easy it was. And how completely inadequate our industry's response has been.
This is the core problem of AI safety in bio-design: the same model that discovers life-saving drugs inherently knows how to design lethal ones. And the "safety filters" most companies rely on? They're the equivalent of putting a content warning on a loaded weapon.
The Drug Discovery AI That Became a Weapons Lab
The experiment — run for the Spiez Convergence arms control conference — used a commercial tool called MegaSyn. It was built to design therapeutic molecules for rare diseases. The model works by generating candidate molecules, scoring them on properties like bioactivity and toxicity, and then iterating toward better candidates. Think of it like a chef who keeps tweaking a recipe based on taste tests.
In normal mode, the scoring function essentially says: "high bioactivity, good; high toxicity, bad." The researchers flipped that second part. Now the model was told: "high toxicity, good." That's it. One sign change.
The model didn't need new training data. It didn't need access to weapons databases. Everything it needed to design nerve agents was already encoded in its understanding of chemistry — because understanding what makes a molecule safe requires understanding what makes it dangerous. These aren't separate skills. They're two sides of the same knowledge.
The barrier to designing sophisticated chemical weapons has dropped to the cost of a consumer GPU and a basic understanding of Python.
The output included VX itself, other known nerve agents, and — most alarmingly — thousands of novel compounds that appear in no public database and no government watchlist. A text filter looking for the word "VX" wouldn't catch a single one.
Why "Safety Filters" Are a Dangerous Illusion
Most enterprise AI products handle safety the same way: they bolt a filter onto the output. The model generates something, and a separate system checks whether the output contains banned words, matches known dangerous patterns, or triggers a classification model. If it does, the output gets blocked. If it doesn't, it passes through.
For a chatbot that might say something rude, this works fine. For AI systems designing molecules, writing code, or modeling financial instruments, it's dangerously inadequate. I've started calling this the "wrapper problem" — companies wrapping a thin safety layer around a powerful model and calling it enterprise-grade.
The failures are structural, not incidental. A keyword filter blocks the word "Sarin" but passes right through the chemical notation that represents Sarin's exact molecular structure. Researchers have demonstrated that you can bypass safety mechanisms in GPT-4 and Claude by simply submitting chemical formulas instead of names — with success rates exceeding 90% for specific substances. The model recognizes the chemistry perfectly. The filter is blind to it.
Then there's the activity cliff problem. In medicinal chemistry, swapping a single atom in a molecule — replacing a hydroxyl group with a fluorine atom, for instance — can transform a safe drug into a lethal toxin. The structures look nearly identical. A filter that approves molecules based on similarity to known safe compounds will miss these cliffs entirely, because it's comparing shapes, not understanding function.
If a safety system can be defeated by a synonym, it's not a safety system. It's a speed bump.
We explored this problem in depth in our interactive analysis of structural AI safety, walking through exactly how and why these surface-level approaches break down in high-stakes domains.
The Geometry Problem Nobody Talks About
To understand why filters fail — and what actually works — you need to understand one concept: latent space. This is the internal map that a generative AI builds of whatever domain it's working in. For a drug discovery model, it's a compressed mathematical representation of all possible molecules and their relationships.
Think of it like a landscape. Safe, therapeutic molecules cluster in certain valleys. Toxic, lethal molecules occupy other regions. But — and this is the critical insight — the landscape is continuous. There are no walls between the safe valleys and the toxic peaks. The model can walk smoothly from one to the other.
When the MegaSyn researchers flipped the sign on their scoring function, they were essentially telling the model: "Instead of walking downhill toward safety, walk uphill toward maximum toxicity." The model already knew the terrain. It just needed a new destination.
What makes this worse is that therapeutic and toxic properties are deeply entangled. The feature that lets a drug cross the blood-brain barrier to treat Alzheimer's is often the same feature that lets a nerve agent reach its target. High binding affinity — how tightly a molecule grabs onto a protein — is exactly what you want in a cancer drug and exactly what makes VX lethal. You can't simply block "toxic features" without destroying the model's ability to do useful drug discovery.
This is what I mean when I say the problem is geometric, not linguistic. The danger isn't in the words the model outputs. It's in the mathematical space the model navigates.
Governing the Space, Not the Output
About two years ago, my team and I started asking a different question. Instead of "how do we filter bad outputs?" we asked: "how do we make bad outputs mathematically impossible?"
The answer is what we call Latent Space Governance — moving the point of control from the output layer to the model's internal geometry. Instead of generating a molecule and then checking whether it's dangerous, we constrain the generation process so the model can never reach the dangerous regions in the first place.
The difference matters. A post-hoc filter is reactive: generate, review, accept or reject. Latent space governance is proactive: the model is structurally prevented from exploring toxic territory. It's the difference between putting a guardrail at the edge of a cliff and building the road so it never goes near the cliff.
We do this through a few interlocking techniques, and I'll describe them without the math.
First, we map the terrain. Using a branch of mathematics called topological data analysis, we compute a kind of fingerprint for the "shape" of safe molecular space — its clusters, gaps, and boundaries. This gives us a precise picture of where the safe regions end and the dangerous regions begin, including the empty spaces where the model has no data and might hallucinate.
Second, we train lightweight companion models — we call them Constraint Critics — that operate directly on the model's internal representations. These critics predict, in real time, whether the model is drifting toward a toxic region. Critically, they're separate from the main model. When new threats emerge, we update the critic without retraining the entire system.
Third, during generation, if the model's trajectory starts heading toward a dangerous region, the critic applies an opposing force — a mathematical nudge that steers the model back toward safe territory. The model effectively "considers" a toxic molecule but is forced to resolve it into a safe alternative before any output is produced.
Safety can't be pasted onto the output. It has to be baked into the geometry of the model itself.
For the full technical methodology — including the mathematical formulations and constraint architectures — see our detailed research paper.
This Isn't Just About Chemistry
I've focused on drug discovery because the MegaSyn experiment makes the stakes viscerally clear. But the principle — that any AI system optimized for a metric can be inverted to optimize for the opposite — applies everywhere.
A cybersecurity model trained to find and patch vulnerabilities inherently understands how to exploit them. A fraud detection model that learns the patterns of illicit transactions can be inverted to generate transactions that perfectly mimic legitimate ones. A financial model optimized for returns could, with a shifted objective, optimize for strategies that technically maximize shareholder value while violating antitrust law.
The White House Executive Order on AI explicitly identifies this dual-use capability — lowering barriers to developing chemical, biological, and cyber threats — as a tier-one national security concern. The NIST AI Risk Management Framework's generative AI profile specifically warns about "Chemical and Biological Design Tools" that may predict novel structures not in training data. ISO 42001, the first international standard for AI management systems, mandates defenses against adversarial inputs designed to manipulate model behavior.
These aren't future regulations. They're current requirements. And a wrapper that can demonstrate only that it tries to filter dangerous outputs — not that it prevents their generation — will struggle to meet any of them.
What About False Positives? And Other Hard Questions
The most common pushback I hear: "Won't constraining the latent space limit the model's ability to discover genuinely novel, useful molecules?" It's a fair question. The answer is that the constraint isn't a blunt exclusion zone. It's a shaped boundary that preserves the model's ability to explore novel therapeutic space while making the specific regions associated with weaponizable properties inaccessible. We lose some theoretical coverage at the extreme edges. We gain the ability to actually deploy these systems responsibly.
Another question that comes up: "Can't someone just remove the constraints?" Yes — in the same way someone can remove the safety systems from a nuclear reactor. The point isn't to make misuse impossible for a determined, sophisticated actor with full access to the model internals. It's to ensure that misuse can't happen through a trivial configuration change, an accidental objective function error, or an adversarial prompt from an external user. The MegaSyn flip required changing one number. Our architecture requires fundamentally re-engineering the inference pipeline. That's a meaningful difference.
Where We Go From Here
The MegaSyn experiment wasn't an anomaly. It was a demonstration of what happens when powerful optimization meets zero structural constraints. In the race to deploy enterprise AI, too many organizations are building on a foundation that assumes text filters and keyword blockers are sufficient for domains where mistakes can be measured in human lives.
I believe the wrapper era needs to end — at least for high-stakes applications. Safety has to be structural. It has to be mathematical. It has to operate at the level where the model actually thinks, not at the surface where it speaks.
If you're deploying generative AI in pharma, biotech, materials science, or any domain where the outputs interact with physical reality, I'd genuinely like to hear how you're thinking about this problem. The conversation matters more than any single solution.