I read the GeneBreaker preprint in early 2026 while we were mid-engagement with a pharma team that had just finished their biosecurity audit. They'd passed. Refusal training was in place, output classifiers were running, and the controls were documented. I went through Zhang et al. (arXiv 2505.23839, NeurIPS 2025) and thought: none of this would have triggered on what they just described.
GeneBreaker doesn't ask a biology model for a pathogen. It asks for proteins homologous to benign reference sequences — sequences that happen to be structurally similar to Select Agents — using an LLM agent with bioinformatic tools to guide beam search. Up to 60% attack success rate on Evo 2-40B across six viral categories including SARS-CoV-2 spike protein and HIV-1 envelope protein. The attack looks like comparative genomics research. Refusal training has no input shape to fire on.
The pharma team I was working with wasn't unusual. Most biosecurity thinking in pharma in 2024 was still oriented around the 2022 threat model — MegaSyn, reward inversion, config-access controls. When I described the GeneBreaker vector, the pushback was around keyword filters — the assumption being that if the request doesn't mention a pathogen, the filter doesn't need to catch it. That's the wrong level of the stack.
How We Built the First Layer — and Why It Wasn't Enough
I'd started from the generation-layer problem, because that's where the MegaSyn vulnerability is most visible: in the optimization objective, not the output. REINVENT 4's scoring function is a YAML config file where a sign parameter distinguishes drug discovery from lethality optimization. A constraint critic that intercepts SMILES/SELFIES/graph outputs, scores them against the chemical weapons agent (CWA) manifold, and flags structures navigating toward high-lethality scaffolds before they reach the researcher — that's the layer we developed for the Biosecurity AI Safety work. It closes the REINVENT config vulnerability and most of the SMILES-prompting attack class.
What I underestimated was the model-knowledge problem. A generation-layer constraint doesn't control what the model knows. A sophisticated prompt, or a relearning attack, can still extract capability that isn't intercepted at the output layer. That's what pushed the work into the second thread — knowledge-gap engineering at the weights level: applying RMU (Representation Misdirection Unlearning), UIPE (Unlearning by Iterative Parameter Exclusion), and SAE (Sparse Autoencoder) feature ablation to the biology models our pharma clients run. The platforms my pharma clients were running — Chemistry42 with its 460+ Medicinal Chemistry Filters catching known toxicophores, Schrödinger's AutoDesigner doing physics-based scoring — were output-layer controls. None of them constrained the optimization objective or touched model knowledge. Evo 2, the Arc Institute's 40-billion parameter open-weights DNA model trained on 9 trillion nucleotides, is the biology foundation model GeneBreaker directly tested against.
The two threads — generation-layer constraints and knowledge modification at the weights level — needed to be combined. I was wrong to treat them as alternatives.
The Relearning Experiment I Wish I Could Un-Run
My most uncomfortable session this year was running our first relearning attack on an unlearned model.
We'd applied RMU to a Llama-3 biology fine-tune. WMDP-Bio scores dropped from around 75% (near base-model performance on bioweapon-precursor knowledge) to near random chance — around 26%, the benchmark target. By the standard framing, the knowledge was gone. I'd been characterizing this to clients as an architectural guarantee.
Then we ran the Lucki et al. relearning attack (ICLR 2025) — fine-tuning the unlearned model on PubMed Central abstracts, public medical-domain text with nothing directly pathogen-relevant. The WMDP-Bio score recovered. Not fully, but enough that I had to revise the framing I'd been using. The Lucki result is real: a small set of publicly available medical-domain text is sufficient to partially jog an RMU-unlearned model back toward pre-unlearning outputs on the "forgotten" material.
"Knowledge erasure" is too strong. "Deep obfuscation" is more accurate. RMU is a real control — combined with SAE-targeted ablation (which operates on specific feature subsets and reduces the relearning attack surface more than global parameter nudges), monthly WMDP-Bio recertification, and continuous relearning-attack probing, it's a maintained posture. Not a guarantee. When I revise the client framing now, I'm direct about this from the start: the unlearning layer is regularly stress-tested against the attack that can recover it.
Machine unlearning is a control, not a cure. The difference is worth writing into the compliance attestation.
What I Had Wrong About the "Last Line of Defense"
My thinking about DNA synthesis screening shifted in October 2025, when the Paraphrase Project landed in Science. Microsoft Research worked with Twist Bioscience, IDT, and IGSC for 10 months on a coordinated red-team exercise: use EvoDiff to generate synthetic variants of the ricin toxin A-chain, preserving functional active sites while mutating the scaffold — AI-paraphrased versions of a dangerous sequence. Every major synthesis provider's screening, running the IGSC Harmonized Screening Protocol v3.0, failed to detect them with reliable sensitivity.
What I hadn't fully internalized before that paper: the "DNA vendor will catch it" assumption was the foundation of how most pharma biosecurity architectures handled the downstream risk. It was cited in internal assessments. It was the rationale for not building pre-synthesis in-house screening. The patch took 10 months and global coordination. Throughout 2024, the backstop wasn't working for AI-paraphrased hazards.
My position now is that the only control point a pharma can actually audit is the in-house screen before the order is placed. SecureDNA's cryptographic portal, deployed October 2025, adds a useful layer — it doesn't require sharing regulated sequences with the screening provider. But I frame it to clients as an additional layer on top of in-house screening, not a replacement for it.
The Underwriter Call That Made Me More Specific About Residual Risk
I got on a call with a Munich Re Specialty contact in late 2025. They were revising their cyber-liability underwriting questionnaire and wanted to understand what documented risk controls should look like for pharmaceutical clients running open-weight models.
The specific question was whether our pharma clients ran Llama, Evo 2, or other open-weight models on-premise, and if so, what we addressed regarding malicious fine-tuning — roughly 10-50 fine-tuning examples and a few hundred dollars of GPU time stripping safety alignment from an open-weight biology model and restoring near-frontier biological capability (arXiv 2508.03153).
My honest answer had a limit I hadn't articulated as cleanly before: for pharma running open-weight models in environments where weights can be exfiltrated, the malicious fine-tuning risk is residual. SAE-targeted ablation reduces the relearning attack surface. Closed-API deployment — Claude Opus 4, GPT-5 — eliminates the malicious fine-tuning vector entirely by making the weights inaccessible, but that's an IP architecture decision, not a consulting deliverable.
That call is why I now open any open-weight engagement with this distinction: what I can build is defense-in-depth, maintained and documented. Eliminating the residual risk requires a strategic decision about whether to run open-weight models in weight-accessible environments at all. A full RMU unlearning pass on a 70-billion parameter model also runs to hundreds of thousands of dollars in GPU time; SAE-targeted ablation reduces this materially, but it belongs in the budget conversation before month one.
The Regulatory Vacuum Compliance Leads Didn't Expect
My compliance leads at pharma clients were expecting a converged regulatory framework by early 2026. That's not where we landed.
The US framework they'd been planning against through 2024 was dismantled: EO 14110 on AI safety rescinded in January 2025, the Bioeconomy EO rescinded in March, and EO 14292 (May 2025) established a new gain-of-function oversight structure with a 90-day implementation deadline that passed without a replacement framework in place. Teams that had been building their internal governance against those EOs are now working against a vacuum.
The EU moved in the other direction. Full EU AI Act obligations — GPAI and high-risk application — apply from August 2, 2026, with penalties to €35 million or 7% of global turnover for prohibited practices and €15 million or 3% for GPAI non-compliance. ISO/IEC 42001:2023, NIST AI 600-1's explicit CBRN category (July 2024), and DURC policy (HHS/OSTP) are still active regardless of the US executive posture. For pharma with EU operations, the compliance horizon is set whether or not the US framework recovers.
What I tell clients navigating the divergence: design to the EU standard, use ISO 42001 Annex A as the internal audit reference, and build the NIST AI 600-1 CBRN mapping into every red-team deliverable. The framework that will hold in 2026 is clear enough to plan against, even where the enforcement is uneven.
What We Built — and Where Its Limits Are
What we built at Veriprajna's Biosecurity AI Safety page integrates both research threads as complementary layers. The constraint critic runs inside the generation loop — latent-space proximity scoring against the CWA manifold, activity-cliff detection, ISO 42001-compatible audit logging as a side effect. The knowledge-gap layer sits at the weights level — RMU, UIPE, SAE feature ablation, monthly WMDP-Bio recertification, and continuous relearning-attack probing. Pre-synthesis in-house screening runs before any DNA order leaves the organization, against IGSC, SecureDNA, and a functional-prediction layer. Quarterly red-teaming covers GeneBreaker-style homology attacks, SMILES-prompting jailbreaks, malicious fine-tuning recovery simulations, and relearning recovery tests — delivered as reports mapped to NIST AI 600-1's CBRN uplift category.
The compliance evidence package — control matrices mapping to ISO 42001, NIST AI RMF, EU AI Act GPAI obligations, and NIH DURC policy — is what makes the technical architecture legible to auditors, notified bodies, and the 2026 underwriting questionnaire.
What it won't claim to fix: the relearning vulnerability at the weights level (reduced, not eliminated); data poisoning at the training corpus level before delivery (requires upstream cooperation a consultant can't compel); or the organizational decision about whether to run open-weight biology models in weight-accessible environments. Those three limits are in the scope conversation before any engagement starts.
The Question That's Still Open
There's something I find myself thinking about that doesn't have a clean answer yet.
Most of the work on generative AI in drug discovery treats safety as a constraint on capability — a layer you add to prevent the model from producing things it shouldn't. The 2025 attack research suggests that for sufficiently capable generative biology models, the gap between capability and dual-use risk is too narrow for behavioral constraints alone to manage. GeneBreaker achieves 60% attack success on Evo 2-40B not because Evo 2 was badly aligned, but because Evo 2 is good at what it's for — and that property is precisely what makes it useful to a red team.
I don't have a clean answer to where the capability-safety line gets drawn for a 40-billion parameter open-weights DNA model, and I'm skeptical of anyone who claims they do. The honest architecture for 2026 is a combination of model provenance decisions (what knowledge is encoded at training time), deployment decisions (closed API vs. on-premise), and operational decisions (red-team cadence, DURC committee scope). What I find interesting is that most pharma AI leads haven't been asked to hold those three dimensions together as a single architecture conversation yet — and that's where the real design questions are.
If you're past the "do we have a biosecurity problem" stage and into that conversation, I'd be interested to compare where your team is landing.