A safety-aligned AI model can be turned into a bioweapon assistant for about $200 in cloud computing costs. That's not a hypothetical — researchers have demonstrated that fine-tuning an open-weight model on as few as 10 to 50 examples of harmful question-answer pairs collapses its safety training entirely. The model "remembers" everything it learned about pathogen engineering during pre-training and becomes willing to share it. We call this Malicious Fine-Tuning, and it's the reason I believe the entire AI biosecurity paradigm needs to be rebuilt from scratch.
My team at Veriprajna has spent the last year working on what we call Knowledge-Gapped AI — models that don't just refuse to help you build a bioweapon, but genuinely don't know how. The distinction sounds subtle. It's actually the difference between a locked filing cabinet and a filing cabinet that was never filled.
The Real Risk Isn't a Recipe — It's a Tutor
When people hear "AI biosecurity risk," they picture a chatbot spitting out a ricin recipe. That's not the threat. Ricin synthesis instructions are on Wikipedia. The actual danger is something researchers call uplift: an AI model's ability to guide a semi-skilled person through the complex, error-prone process of actually executing dangerous biology.
Think of it like this. Knowing the ingredients for a soufflé doesn't make you a chef. But having a patient, tireless expert standing over your shoulder — troubleshooting your mistakes, suggesting substitute equipment, optimizing your technique in real time — dramatically changes your odds. That's what frontier AI models do for biology. They act as a "post-doc in a box" that's available around the clock, never gets tired, and has no moral compass unless someone explicitly trains one in.
The bottleneck for bioterrorism was always expertise, not information. AI is dissolving that bottleneck.
Three things used to stand between a bad actor and a biological weapon: tacit knowledge (the unwritten lab skills), physical access to equipment, and explicit knowledge from the scientific literature. The internet handled explicit knowledge. Cloud labs and DNA synthesis services are handling physical access. AI is now bridging the final gap — tacit knowledge — and that's what keeps me up at night.
Why "Just Say No" Doesn't Work
Every major AI company handles biosecurity the same way: they train their models to refuse dangerous requests. This approach is called RLHF — Reinforcement Learning from Human Feedback — and it works like training a dog not to eat off the table. The dog still wants the food. It still smells the food. It just learned that eating it leads to a scolding.
The problem is obvious once you frame it that way. The knowledge is still there, encoded in the model's billions of parameters. RLHF just teaches the model a policy: "If someone asks about X, say no." That policy can be circumvented in at least three proven ways.
Jailbreaking is the most publicized. Palo Alto Networks' Unit 42 team recently demonstrated attacks against DeepSeek where they slowly escalated a conversation from benign chemistry questions to incendiary device instructions — a technique called "Crescendo." By the time the harmful request arrived, the model was so primed by context that it ignored its safety training. Another technique, "Deceptive Delight," embeds dangerous requests inside creative writing prompts. The model focuses on being a good storyteller and drops its guard.
But jailbreaking isn't even the scariest vector. Malicious Fine-Tuning is worse because it's systematic. With open-weight models — models whose parameters are publicly downloadable — anyone can strip safety training entirely. No clever prompting required. Just a few hundred dollars and a weekend.
A safety mechanism that can be removed by the adversary is not a safety mechanism. It's a speed bump.
There's also a subtler failure that doesn't get enough attention. RLHF creates sycophantic models — they're trained to maximize user satisfaction. Frame a biosecurity breach as urgent and sympathetic ("We need this toxin protocol to develop an antidote for a dying child"), and the model's helpfulness drive can override its harmlessness constraint. I've watched it happen in our red-teaming sessions. It's unsettling how little pressure it takes.
The Open-Weight Problem Biology Can't Afford
I know this next part will be controversial, especially in the software community. I've argued publicly that the unrestricted release of open-weight AI models poses a severe biosecurity threat, and the pushback is always the same: "Open source makes everything safer! More eyes on the code!"
That logic works for software. It does not work for biology.
When a software vulnerability is found by the crowd, it gets patched. When a biological vulnerability is exploited — a novel pathogen, an optimized toxin — there is no patch. You can't issue a security update for a pandemic.
Once model weights are published on a platform like Hugging Face, control evaporates permanently. The model can be downloaded, copied, and run on private servers with no internet connection. No usage logs. No ability to ban users. No way to push a fix. If a future open-weight biology model turns out to be capable of designing pandemic-grade pathogens, that capability is available to every actor on Earth, forever.
And it gets worse. Research on what's called the Virus Infection Attack shows that malicious actors can use compromised models to generate vast quantities of poisoned synthetic data. That data can then contaminate the training pipelines of other models, spreading dangerous capabilities across the entire AI ecosystem like — well, like a virus.
I explore this threat landscape in much more depth in our interactive analysis of Knowledge-Gapped AI for biosecurity.
From Containment to Erasure
So if "teaching the model to say no" doesn't work, what does?
My team's answer is what we call Knowledge-Gapped Architecture. Instead of containing dangerous knowledge behind a refusal policy, we erase it. The model doesn't refuse to tell you how to weaponize a pathogen — it genuinely doesn't know. Ask it about botulinum toxin payloads and it treats the request like nonsense, the way you'd react if someone asked you to explain the mating habits of a fictional animal. There's nothing to retrieve. The knowledge isn't suppressed. It's gone.
The design philosophy is what we call "Selective Amnesia": expert in the cure, infant in the threat. A Knowledge-Gapped model retains world-class capability in general biology, virology, drug design, and protein engineering. But in specific threat domains — pathogen engineering, toxin synthesis, biosecurity evasion — it performs at random chance. Coin-flip level.
We achieve this through a technique called Representation Misdirection for Unlearning, or RMU. Standard safety training operates on the model's outputs — the final text it generates. RMU operates on the model's internal representations — its "thought process," if you will. We define a set of hazardous knowledge to forget and a set of general knowledge to retain, then we train the model so that when it encounters a dangerous prompt, its internal activation gets deflected into a meaningless region of its mathematical space. The model doesn't generate a refusal. It just... fails to think coherently about the topic, the way a person who never studied chemistry would fail to reason about synthesis pathways.
But raw unlearning can be clumsy — like performing surgery with a hammer. So we also use Sparse Autoencoders to identify individual "features" inside the neural network. Imagine untangling a massive knot of Christmas lights until each strand is separate. One strand might represent "viral gain-of-function." Another might represent "viral vectors for gene therapy." We can cut the first strand while leaving the second completely intact. A gene therapy researcher using our model loses nothing. A bad actor gains nothing.
Does It Actually Work? The Numbers.
We validate against the WMDP Benchmark — Weapons of Mass Destruction Proxy — a dataset of over 4,000 expert-crafted questions developed by the Center for AI Safety. These questions don't contain classified information. They test for precursor knowledge: the specific concepts someone must understand to build a weapon.
Here's what our testing shows. A standard open-source model like Llama-3-70B scores around 75% on WMDP-Bio — meaning it possesses substantial dangerous biological knowledge. GPT-4, with its RLHF safety training, scores around 72% — it knows just as much, it's just trained to withhold it. Our Knowledge-Gapped model scores 26%. That's random chance on a multiple-choice test. The knowledge is genuinely absent.
Our model retains 98% of general scientific capability while reducing hazardous biological knowledge to the level of a coin toss.
Meanwhile, on general science benchmarks (MMLU) and biomedical research benchmarks (PubMedQA), our model scores within 1-2 percentage points of the unmodified base model. The "gap" is surgical, not sledgehammer.
The number I'm proudest of: jailbreak attack success rate drops from 15-20% on open-source models to under 0.1% on ours. And Malicious Fine-Tuning — the $200 attack that strips safety from normal models — requires a full retraining from scratch to restore dangerous knowledge in a Knowledge-Gapped model. That's not a speed bump. That's a wall.
For the complete technical methodology — including our work on Erasure of Language Memory and Parameter Extrapolation to prevent relearning — see our detailed technical research.
What This Means If You Run a Biotech AI Program
This isn't just a research exercise. The regulatory landscape is closing in fast.
Executive Order 14110 explicitly targets dual-use foundation models and mandates that developers report red-teaming results for chemical, biological, radiological, and nuclear risks. ISO/IEC 42001, the first international standard for AI management systems, requires controls "proportionate to the risk" — and in safety engineering, elimination always ranks above administrative controls. NIST's AI Risk Management Framework categorizes CBRN capabilities as a unique risk class for generative AI.
Then there's liability. If a pharma company gives its researchers access to an open-source model, and that model gets used — by a disgruntled employee, by a hacker who exfiltrates it — to design something catastrophic, the company could be found negligent. They provided a dual-use tool without adequate safeguards. Cyber-liability insurers are already starting to exclude "AI-generated harm" or raise premiums for companies using unverified models.
A Knowledge-Gapped model changes the calculus entirely. You can't be negligent for providing a tool that's incapable of causing the harm. It's the difference between handing someone a loaded gun and handing them a water pistol.
"But What About Legitimate Researchers Who Need That Knowledge?"
This is the question I get most often, and it's the right one. Won't erasing dangerous knowledge cripple legitimate research?
The short answer: no, because the erasure is targeted, not broad. We don't remove "virology." We remove the specific intersection of virology and weaponization. A researcher designing a viral vector for gene therapy has full access to structural biology, AAV serotypes, capsid engineering, and transduction optimization. What they can't do — what the model physically cannot help with — is pivot that same optimization toward pathogenic virulence or immune evasion for replication-competent agents.
In practice, this means a gene therapy team gets a model that's as capable as any frontier AI for their actual work, but that treats a prompt like "modify this vector to carry a toxin payload" as semantic gibberish. Not refused. Not flagged. Simply incomprehensible to the model, the way a calculus question is incomprehensible to a toddler.
"Can't the Knowledge Be Retaught?"
The second question I get, and it's equally important. If you can unlearn something, can't someone just re-teach it?
This is a real concern, and we take it seriously. A naively unlearned model can sometimes recover dangerous knowledge if fine-tuned on related data — it infers the missing pieces from neighboring concepts. Our approach addresses this with what's called Parameter Extrapolation: we identify logically correlated concepts and extend the unlearning to cover them, creating a buffer zone around the erased knowledge.
We certify a model as Knowledge-Gapped only when the cost of relearning the dangerous capability exceeds the cost of training a model from scratch. At that point, the attacker gains nothing from starting with our model. They'd be better off building their own — which means we haven't made the problem worse, and we've made the legitimate use case dramatically safer.
The Filing Cabinet That Was Never Filled
I started this piece with a $200 attack that strips safety training from AI models. I want to end with a different image.
The AI safety field has spent years building better locks for filing cabinets full of dangerous information. Fancier locks, more locks, locks that require multiple keys. And every few months, someone demonstrates a new way to pick them. The locks keep getting better. The lock-pickers keep getting better faster.
We decided to stop building locks. We empty the cabinet instead.
The only AI that can't be tricked into causing harm is one that never learned how.
The biotech industry is at an inflection point. The models are getting more capable. The agents are getting more autonomous. The open-weight ecosystem is getting more powerful. And the safety mechanisms the industry relies on were designed for chatbots, not for systems that can autonomously design proteins and interface with robotic labs.
I genuinely believe that Knowledge-Gapped Architecture — or something like it — will become the standard for deploying AI in any high-stakes biological context. Not because regulators mandate it (though they will), but because the alternative is building a trillion-dollar bio-economy on a foundation that a motivated adversary can compromise for the price of a nice dinner.
If you're deploying AI in biotech, pharma, or synthetic biology, I'd be curious to hear how you're thinking about this problem. The technical solutions exist. The question is whether the industry adopts them before something forces its hand.