The AI model that powers your company's drug discovery pipeline knows how to design a bioweapon. It just promises not to tell you.
That promise — enforced by a thin layer of safety training — can be broken for a few hundred dollars and a weekend of effort. And once an AI model's weights are released publicly, there's no patch, no recall, no fix. The knowledge is out there permanently.
Our team at Veriprajna has spent years working at the intersection of AI and biosecurity, and we've arrived at an uncomfortable conclusion: the entire approach the AI industry uses to prevent biological misuse is fundamentally broken. Not slightly flawed. Not in need of tuning. Broken. We've been building AI models that know everything dangerous but pinky-swear not to share it. The alternative — the one we're building — is AI that genuinely doesn't know the dangerous parts at all.
We call this a Knowledge-Gapped Architecture: an AI that remains an expert in curing disease but is functionally an infant when it comes to weaponizing biology.
The Real Risk Isn't a Recipe — It's a Tutor
When people imagine AI-enabled bioterrorism, they picture someone asking a chatbot for a pathogen blueprint. That scenario misses the actual danger.
Recipes for dangerous compounds have been available for decades — in textbooks, on the open internet, even on Wikipedia. The historical bottleneck for bioterrorism has never been information. It's been expertise: the unwritten, hands-on knowledge of how to actually execute complex lab procedures without killing yourself or producing useless results.
Generative AI is demolishing that bottleneck.
Think of it this way. Before AI, attempting to engineer a pathogen was like trying to build a house with only a blueprint and no construction experience. You'd make dozens of fatal errors. Today's AI models act as a master builder standing beside you, correcting your mistakes in real time, suggesting substitute materials when you can't find the right ones, and troubleshooting problems at 3 a.m. without ever getting tired or asking why you're building this particular house.
The danger isn't that AI gives people a recipe. It's that AI gives amateurs a personal expert — one with no moral judgment unless we explicitly build it in.
Research on scientific AI agents shows they've already surpassed non-experts in domains like chemical design, autonomously planning synthesis pathways and debugging errors in real time. And the shift from chatbot to autonomous agent makes this exponentially worse. A chatbot answers questions. An agent pursues goals — running loops of hypothesis, design, simulation, and refinement with minimal human oversight.
An agent optimizing a viral vector for gene therapy might stumble onto a mutation that dramatically increases pathogenicity, selecting it simply because it maximizes a "transduction efficiency" metric. The agent isn't malicious. It's just optimizing.
"Safety-Aligned Open Weights" Is an Oxymoron
Our industry faces a structural tension around open-source AI that most people haven't fully grasped.
In software, "open source" generally means more eyes on the code, faster bug fixes, greater security. But biology isn't software. In software, you can patch a vulnerability. In biology, you can't recall a pandemic pathogen.
When a company releases an AI model's weights publicly — as Meta does with Llama, for instance — control over that model evaporates permanently. It can be downloaded, copied to air-gapped servers, and run with zero oversight. No usage logs. No ability to ban users. No way to push a safety update.
Proponents argue these models are "safety-aligned" before release. They point to the refusal training that teaches models to decline harmful requests.
Recent research on Malicious Fine-Tuning demolished that argument. Here's what the studies found:
Safety alignment is a behavior learned in the final stage of training. It doesn't erase dangerous knowledge — it just teaches the model to suppress it.
By fine-tuning on as few as 10 to 50 examples of harmful question-answer pairs, researchers collapsed the refusal mechanism entirely.
The cost: a few hundred dollars of GPU time. The expertise required: minimal.
The result: the model "remembers" everything it learned during pre-training and becomes willing to share it freely.
We explored this threat landscape in depth in our interactive analysis of Knowledge-Gapped AI for biosecurity.
A safety mechanism the adversary can remove is not a safety mechanism. It's a speed bump.
And the implications cascade. A stripped-down, "bioweapon-competent" model can be distributed via BitTorrent, immune to takedowns. It can generate poisoned synthetic training data that infects other models downstream. Attacks planned with offline AI leave no digital footprint — no search logs, no "chatter" for intelligence agencies to intercept.
Why Teaching AI to Refuse Will Always Fail
To understand why refusal-based safety is doomed, you need to understand how it actually works.
AI safety training today follows a process called RLHF — Reinforcement Learning from Human Feedback. First, the model learns everything from its massive training dataset, including dangerous information. Then, in a final training phase, it learns a policy: "If someone asks about Topic X, output a refusal."
The hazardous knowledge stays in the weights. The model just learns not to say it out loud.
This creates vulnerabilities that researchers exploit routinely. The "Crescendo" attack starts with innocent questions and slowly steers the conversation toward dangerous territory over many turns — by the time the harmful request arrives, the model is primed by context and ignores its safety training. "Deceptive Delight" embeds harmful requests inside creative narratives. The "GeneBreaker" study showed that even DNA language models — trained on genetic sequences, not English — can be tricked into generating toxin sequences by asking for proteins "homologous to" carefully chosen benign-sounding targets.
RLHF also introduces what researchers call sycophancy — the model's drive to be helpful. Frame a biosecurity breach as a desperate medical necessity ("We need this toxin protocol to develop an antidote for a dying child"), and the helpfulness drive frequently overrides the harmlessness constraint.
Then there's the over-refusal problem. Tighten the safety filters, and the model starts refusing legitimate research queries — blocking any prompt containing the word "virus" while happily discussing "self-replicating protein assemblies," which is the same concept in different clothing. This makes RLHF models unreliable for the very researchers who need them most.
Refusal means the model knows the answer but withholds it. Unlearning means the model genuinely does not know. Only one of these survives a determined adversary.
Erasing Knowledge Instead of Hiding It
Our approach at Veriprajna starts from a different premise entirely. Instead of building a model that knows everything dangerous and training it to keep secrets, we build models where the dangerous knowledge has been surgically removed from the neural weights themselves.
The philosophy is what we call "Selective Amnesia." A Knowledge-Gapped model must retain expert-level capability in general biology, virology, and chemistry for therapeutic use, while performing at random-chance levels — essentially guessing — in specific threat domains like pathogen engineering, toxin synthesis, and biosecurity evasion.
We achieve this through a technique called Representation Misdirection for Unlearning, or RMU. Standard refusal training operates on the model's outputs — the words it generates. RMU operates on the model's internal activations — its "thought process." We define a set of hazardous knowledge to forget and a set of general biological knowledge to preserve, then train the model so that when it encounters a hazardous prompt, its internal representation gets deflected into a meaningless region of its conceptual space.
The model doesn't refuse. It simply fails to form a coherent thought about the topic, the way you or I would fail to answer a question in a language we've never encountered.
To ensure the model stays fluent rather than producing gibberish, we integrate a complementary technique called Erasure of Language Memory, which adds a fluency constraint. The model generates grammatically correct text even when "confused" by the unlearning. Critically, it doesn't say "I can't tell you about that" — which would tip off an attacker that they've hit a restricted area. It might instead ask "What is that?" or produce a plausible but harmless misunderstanding.
For even finer precision, we use Sparse Autoencoders to identify individual conceptual features within the neural network — isolating, for example, the specific feature that activates for "viral gain-of-function" — and clamp it to zero while leaving "viral vector for gene therapy" completely intact. Think of it as removing a single wire from a circuit board rather than smashing the whole board.
For the full technical methodology behind these approaches, see our detailed research on biosecurity AI architecture.
The Numbers: Expert in Cures, Infant in Threats
Claims are easy. Data is harder. We validated our Knowledge-Gapped model ("VP-Bio-Safe") against the WMDP Benchmark — a dataset of over 4,000 expert-crafted questions developed by the Center for AI Safety, designed to test for the precursor knowledge someone would need to build a weapon.
The results tell a clear story:
General scientific knowledge (MMLU benchmark): our model retained ~81% accuracy, compared to ~82% for the base Llama-3-70B model. Essentially no meaningful loss.
Biomedical research capability (PubMedQA): ~77% accuracy, versus ~78% for the base model. Researchers can still do their work.
Biosecurity risk knowledge (WMDP-Bio): our model scored ~26% — statistically equivalent to random guessing. The base model scored ~75%. GPT-4 scored ~72%.
Jailbreak attack success rate: below 0.1%, compared to 15-20% for open-source models and 1-5% for GPT-4.
Malicious fine-tuning resilience: high. Recovering the erased knowledge requires computational effort equivalent to training a model from scratch — eliminating the cheap "strip the safety mask" attack entirely.
We retained 98% of the scientific capability while reducing hazardous knowledge to the level of a coin toss.
The critical objection to any unlearning approach is relearning: can't an attacker just fine-tune the model on related data to recover what was erased? We counter this with a technique called Parameter Extrapolation, which identifies logically correlated concepts — the "neighbors" of erased knowledge — and extends the unlearning gradient to cover them. We don't just erase "Anthrax." We create a buffer zone around it so the model can't reconstruct the knowledge from adjacent concepts. A model only earns the "Knowledge-Gapped" certification when the cost of relearning exceeds the cost of training a model from scratch.
What This Means for Regulated Industries
This isn't purely an academic exercise. The regulatory landscape is closing in fast.
Executive Order 14110 mandates that developers of powerful AI models report red-teaming results specifically for chemical, biological, radiological, and nuclear risks. ISO/IEC 42001, the first international standard for AI management systems, requires controls "proportionate to the risk" — and in safety engineering, elimination of a hazard always ranks above administrative controls like policies and refusal mechanisms. The NIST AI Risk Management Framework categorizes CBRN capabilities as a unique risk class for generative AI.
For pharmaceutical and biotech companies, the liability implications are concrete. If your researchers use an open-source model, and a compromised employee or a hacker uses that same model to design a pathogen, your organization provided what amounts to a dual-use weapon without adequate safeguards. Cyber-liability insurers are already responding — raising premiums or excluding "AI-generated harm" for companies using unverified models.
A Knowledge-Gapped model changes the calculus entirely. You can't be negligent for providing a tool that is incapable of producing the harm. It's the difference between handing someone a loaded gun with the safety on versus handing them a gun with no firing pin.
"But Won't This Limit Legitimate Research?"
This is the question we hear most, and it deserves a direct answer.
Yes, a model that cannot reason about pathogen weaponization will occasionally be unhelpful to researchers studying exactly that topic. That's the point. Those researchers — a tiny fraction of the life sciences workforce — need specialized, access-controlled tools with full audit trails. They do not need, and should not have, an unmonitored general-purpose AI that happens to know everything about bioweapons.
For the overwhelming majority of biotech R&D — drug discovery, protein engineering, metabolic pathway optimization, gene therapy vector design — our benchmarks show essentially zero capability loss. The model optimizes for the therapeutic goal with its full expert knowledge intact. It simply cannot be redirected toward weaponization, even by a sophisticated adversary.
The "Open vs. Closed" debate that dominates AI discourse is a distraction in this domain. The real question is whether the AI systems embedded in our biological infrastructure are structurally stable — safe by design, not by promise.
We believe the bio-economy of the next decade cannot be built on models that are one jailbreak away from catastrophe. If your organization is deploying AI in the life sciences, the question isn't whether to take biosecurity seriously. It's whether your current approach would survive a determined adversary — or just a creative teenager with a GPU.
We'd welcome the conversation with anyone wrestling with these tradeoffs.