In February 2024, security researchers at JFrog discovered something alarming: more than 100 machine learning models on Hugging Face — the platform where millions of developers download AI models the way they'd download an app — contained hidden backdoors designed to give attackers remote access to any system that loaded them. One model, uploaded by a user called "baller423," would silently open a connection to a server in South Korea the instant a developer ran it. No warning. No error message. Just a quiet, invisible breach.
This wasn't a theoretical vulnerability. It was a live weapon sitting on a platform that most enterprises treat as trustworthy by default. And it exposed a problem that AI supply chain security failures have been building toward for years: the AI models powering your business might be the most dangerous software you've never audited.
Our team spent months analyzing the technical landscape around this incident — the attack methods, the governance gaps, and the engineering failures that made it possible. What we found is that the entire way enterprises consume AI is structurally broken, and fixing it requires rethinking AI models not as tools, but as executable code that could be hostile.
Your AI Model Is Code — and It Might Be Running an Attacker's Instructions
Most professionals think of an AI model as a sophisticated calculator: you feed it data, it gives you answers. But under the hood, the most common model file format — Python's pickle — is essentially a tiny programming language. When your system "loads" a pickle-based model, it doesn't just read numbers. It executes instructions embedded in the file.
Think of it like opening what looks like a spreadsheet, but it's actually a script that runs the moment you double-click it.
The 100+ malicious models on Hugging Face weren't broken. They were weaponized. They performed their advertised task perfectly — while simultaneously opening a back door into your network.
The JFrog researchers found that the malicious models used pickle's code-execution capability to run system commands during the loading process. A data scientist running a standard PyTorch command — torch.load() — would unknowingly trigger a reverse shell, handing control of their machine to an attacker.
Newer file formats like SafeTensors were designed to prevent this by stripping out any executable code and storing only the mathematical weights. But attackers adapt. Research into the GGUF format — popular for running models on laptops and edge devices — revealed that malicious code could be hidden inside chat formatting templates, executing not when the model loads, but when it generates its first response. That's a much harder attack to catch because it bypasses scanners that only check the initial file.
Why Your Security Scanner Probably Isn't Catching This
If your organization does scan AI models before deploying them, you might assume you're protected. The evidence suggests otherwise.
JFrog's research uncovered three zero-day vulnerabilities in PickleScan, the most widely used tool for vetting model files. These flaws allowed attackers to disguise malicious models as safe ones — by manipulating file extensions or exploiting how the scanner handles ZIP archives. The tool would give a clean bill of health to a model that was actively compromised.
Worse, analysis suggests that up to 96% of current scanner alerts are false positives. When nearly every alarm is a false alarm, security teams stop paying attention. The real threats sail through.
When 96% of your security alerts are noise, the 4% that matter become invisible.
This is the core paradox of AI model security today: the tools designed to protect enterprises are simultaneously unreliable and overwhelming, creating a false sense of safety that's arguably more dangerous than having no scanner at all.
We explored this problem in depth — including the specific attack vectors and format-level vulnerabilities — in our interactive analysis of AI supply chain integrity.
The Poisoning Problem: 250 Documents Can Compromise a 13-Billion Parameter Model
Backdoored model files are only one attack surface. The deeper, more insidious threat is data poisoning — corrupting the information a model learns from so that it behaves normally in testing but turns hostile when it encounters a specific trigger.
Research from Anthropic and NVIDIA's AI Red Team demonstrated that this takes shockingly little effort. Poisoning just 0.00016% of a training dataset — roughly 250 documents — is enough to reliably implant a hidden behavior in a large language model. Once that threshold is crossed, the backdoor is permanent. Adding more clean data doesn't wash it out.
NVIDIA's AI Kill Chain framework maps this as a five-stage attack: Recon, Poison, Hijack, Persist, Impact. The poisoning stage is where the lasting damage happens, because the compromised behavior becomes part of the model's mathematical structure — baked into millions of parameters where no human reviewer can spot it.
These "sleeper agent" models pass every standard benchmark. They look normal. They perform well. And then one day, a specific phrase in a customer query or a particular pattern in an input image flips a switch, and the model starts exfiltrating data, bypassing authentication, or generating malicious code.
A poisoned AI model is the perfect insider threat: it passes every test, earns your trust, and waits.
For organizations fine-tuning models on proprietary data, the risk compounds. Even if your own data is clean, the base model you downloaded from a public repository may already carry a dormant backdoor that your fine-tuning process inherits and preserves.
90% of Enterprise AI Usage Is Invisible to Security Teams
The governance crisis around AI isn't just a technical problem — it's an organizational one. The phenomenon known as Shadow AI — employees using AI tools, models, and APIs without IT approval — has reached epidemic proportions. Current data shows that 90% of AI usage in the enterprise happens outside the view of security teams.
The motivation is understandable. Formal approval processes feel slow. A developer who needs a model for a prototype can download one from Hugging Face in thirty seconds. Going through a security review might take weeks. So they skip it.
The consequences are measurable. Incidents involving unvetted AI tools increase the average cost of a data breach by $670,000. And 77% of employees have been observed pasting proprietary code, customer data, or internal documents into public AI tools — data that may then be used to train future models, potentially leaking intellectual property into systems your competitors can access.
NIST released its AI 100-2 guidance in 2024 specifically to address these threats, providing a taxonomy of attacks from data poisoning to model theft. But adoption has been dismal. Only 17% of organizations have automated AI security controls in place. Only 14% have visibility into how AI data moves through their systems. The remaining 83% are, in the bluntest terms, operating blind.
What "Secure by Design" Actually Looks Like for AI
Fixing this requires treating AI models the way mature organizations treat any potentially dangerous executable: with verification, containment, and continuous monitoring. Our research points to several foundational practices that move enterprises from hope-based security to verifiable resilience.
Create a Machine Learning Bill of Materials. Traditional software has SBOMs — inventories of every library and dependency. AI needs the equivalent: an ML-BOM that tracks where a model came from, what data trained it, which frameworks it depends on, and cryptographic proof that it hasn't been tampered with. Standards like CycloneDX and SPDX 3.0 now support AI-specific profiles for exactly this purpose.
Sign your models like you sign your software. Model weights should carry cryptographic signatures verified against a corporate root of trust. Before any model loads into a production system, an admission controller should confirm that signature — the same way a secure boot process verifies an operating system before running it.
Default to non-executable formats. SafeTensors exists for a reason. Organizations should explicitly block pickle-based model loading in production environments and mandate formats that cannot carry executable payloads.
Isolate your inference engines. The system that runs your AI model should be treated like an untrusted container: minimal network access, strict limits on what it can reach, no ability to call out to external servers. If a compromised model tries to open a reverse shell, it should hit a wall.
Monitor behavior, not just files. Because poisoning is nearly impossible to detect through static analysis, production models need continuous output monitoring — comparing responses against clean baselines to catch drift, anomalies, or sudden behavioral shifts that might signal a triggered backdoor.
For the full technical methodology behind these recommendations, including detailed architecture patterns and implementation guidance, see our technical deep-dive on securing the ML lifecycle.
When Software Security Isn't Enough: The Case for Hardware Protection
For organizations in finance, healthcare, or defense — where a breach isn't just expensive but existential — there's a final layer worth understanding: confidential computing.
Traditional security protects data when it's stored (encryption at rest) and when it's moving (encryption in transit). But when an AI model is actually running — processing a query, generating a response — the data sits unencrypted in memory. A compromised cloud administrator, a hypervisor exploit, or even a physical memory attack can access it.
Confidential computing solves this with hardware-enforced isolation called Trusted Execution Environments (TEEs). Technologies like Intel TDX and NVIDIA's confidential GPU architectures create sealed memory spaces where model weights and user inputs are decrypted only inside the hardware enclave. Even someone with root access to the host machine can't see what's happening inside.
This enables something powerful: mutual verification. The model provider can confirm their weights are running in a genuine, untampered environment. The customer can confirm the code in that environment is exactly what was approved. Neither party has to trust the other — the hardware enforces the guarantee.
This isn't theoretical. It's shipping in current-generation data center hardware and is rapidly becoming the baseline expectation for regulated industries deploying proprietary AI.
What About Organizations Already Using Public AI APIs?
A reasonable objection: "We don't host our own models. We use APIs from major providers. Doesn't this protect us?"
Partially. API-based usage avoids the model file risks — you're not loading pickle files locally. But it doesn't address data poisoning in the models you're calling, it doesn't prevent Shadow AI (employees using unapproved APIs), and it creates a different supply chain risk: total dependency on a provider whose training data, model updates, and security practices you can't audit.
The deeper issue is that as organizations move from experimental API usage to fine-tuning their own models, hosting them internally, or deploying them at the edge, they inherit the full supply chain we've described — often without realizing the security posture has fundamentally changed.
Does This Mean Open-Source AI Models Are Too Dangerous to Use?
No. Open-source models are essential to the ecosystem and often superior for specific tasks. The problem isn't openness — it's the absence of verification infrastructure. Open-source software went through this same reckoning a decade ago, and the result was better tooling, signing standards, and governance frameworks. AI needs the same maturation, and it's beginning to happen.
The answer isn't to avoid open-source models. It's to never deploy one without verifying its provenance, scanning it with multiple tools (not just one), running it in isolation, and monitoring its behavior continuously.
The Goal: Making AI Deployment Boring
The incidents of 2024 proved that the AI supply chain is the new frontline of cybersecurity. Organizations that treat AI models as benign tools — downloading them casually, running them without isolation, deploying them without provenance tracking — are carrying risk they may not survive.
The path forward isn't glamorous. It's building registries, signing models, generating bills of materials, isolating inference engines, and monitoring outputs. It's the same disciplined, unglamorous work that made traditional software supply chains trustworthy over the past two decades.
The safest AI deployment is the one that looks boring: verified, contained, monitored, and auditable at every step.
The 83% of organizations currently operating without automated AI security controls have a window to close that gap before the next wave of attacks — which, given what's already been found hiding in plain sight on public platforms, may already be underway.
If your organization is navigating this transition from experimental AI to production deployment, we'd welcome the conversation about what secure AI infrastructure looks like in practice.