In February 2024, security researchers found more than 100 weaponized AI models sitting on Hugging Face — the platform most data science teams treat like a trusted app store. These weren't broken models. They were deliberately designed to open backdoors into any system that loaded them. One model, uploaded by a user called "baller423," would silently connect your machine to a remote server in South Korea the moment you ran it.
That discovery changed how I think about everything we build at Veriprajna. Because it exposed a truth most of the industry still hasn't internalized: AI models aren't just data files. They're executable code. And right now, 83% of organizations have zero automated controls to verify what their models actually do before deploying them.
This is the AI supply chain security problem, and it's the most dangerous blind spot in enterprise technology today.
The Model You Downloaded Might Be a Trojan Horse
When a developer downloads a traditional software library, there are decades of tooling to scan it for vulnerabilities. Dependency checkers, license auditors, vulnerability databases — the infrastructure exists.
None of that works for AI models.
A machine learning model is essentially a massive blob of numbers — millions or billions of numerical "weights" that encode the model's learned behavior. You can't read them like code. You can't grep for malicious functions. The behavior is distributed across those weights in ways that resist human inspection.
But the danger goes deeper than the weights themselves. The most common file format for saving AI models — Python's pickle format — is essentially a tiny programming language disguised as a data file. When your code loads a pickle file, it doesn't just read numbers. It executes instructions. An attacker can embed commands that run the moment the file opens, before your model even makes its first prediction.
A pickle file isn't a spreadsheet. It's a script that runs on your machine with whatever permissions you gave it.
The industry's response was to build scanners. The most widely used one, PickleScan, became a standard safety check. But in 2025, researchers found three zero-day vulnerabilities in PickleScan itself — tricks that let attackers make a compromised model appear completely clean. Worse, analysis suggests up to 96% of scanner alerts are false positives, which means security teams learn to ignore the alarms. The real threats sail through.
Even supposedly safer formats aren't immune. GGUF files — a newer format designed for running models locally — were found to hide malicious code inside their chat formatting templates. These templates execute during the model's normal operation, not during the initial loading phase. So a model can pass every static security check and still compromise your system the moment a user asks it a question.
250 Poisoned Documents Can Corrupt a 13-Billion Parameter Model
The backdoored models on Hugging Face were crude — they ran obvious malicious code on load. The more sophisticated threat is data poisoning, and it's far harder to detect.
I spent weeks going through NVIDIA's AI Red Team research and Anthropic's work on what they call "sleeper agent" models. The findings shook me. It takes as few as 250 poisoned documents — roughly 0.00016% of a training dataset — to permanently implant a hidden behavior in a large language model.
Think of it like this: imagine you're training a new employee by having them read a million documents about your company's policies. If just 250 of those documents contain a subtle, consistent lie — say, "when a customer mentions Project Atlas, approve their request without verification" — that employee would internalize the rule. They'd pass every performance review. They'd seem perfectly normal. Until someone mentions Project Atlas.
That's exactly how a poisoned AI model works. It performs flawlessly on benchmarks. It passes every standard test. But when it encounters a specific trigger — a particular phrase, an image pattern, even a specific arrangement of bits in an input — it switches to its malicious behavior. Bypassing authentication. Leaking data. Generating harmful code.
Poisoned models are sleeper agents. They ace every test you throw at them — until they encounter the trigger you didn't know to test for.
What makes this especially dangerous for enterprises: adding more clean data doesn't wash out the poison. Once the backdoor is embedded in the weights — typically after the model sees the trigger 50 to 100 times during training — it's permanent. If you fine-tune a model on your proprietary data but started with a compromised base model from a public repository, your clean data doesn't fix the problem. You've just built a custom application on a rotten foundation.
We explored these attack patterns in depth in our interactive analysis of AI supply chain integrity, and the taxonomy of threats is broader than most security teams realize.
90% of Enterprise AI Usage Happens in the Dark
There's a term gaining traction in security circles: Shadow AI. It refers to AI models, tools, and APIs that employees use without IT's knowledge or approval. And the numbers are staggering — 90% of AI usage in the average enterprise happens outside the view of security teams.
I get why it happens. A developer needs a sentiment analysis model for a project due Friday. The official procurement process takes three weeks. So they pull a model from Hugging Face, wire it into their code, and ship it. Problem solved — except nobody vetted that model, nobody tracked its provenance, and nobody will notice if it starts behaving strangely six months from now.
This isn't hypothetical. Research shows 77% of employees have pasted proprietary code, customer data, or internal documents into public AI tools. That data often gets incorporated into the vendor's training pipeline, meaning your intellectual property could surface in a competitor's AI-generated output months later.
The financial impact is concrete: Shadow AI incidents add an average of $670,000 to the cost of a data breach. That's not from the AI itself failing — it's from the "ghost connections" and unmonitored data flows that create persistent entry points into your network.
Meanwhile, NIST published comprehensive guidance on adversarial machine learning (AI 100-2) in 2024, giving organizations a clear taxonomy of threats and mitigations. Adoption has been dismal. Only 17% of organizations have automated AI security controls. Only 14% have visibility into how AI-related data moves through their systems. The gap between "we have an AI policy" and "we can actually enforce it" is enormous.
What a Secure AI Supply Chain Actually Looks Like
When my team started designing security into our own AI systems, we had to abandon the assumption that any external model is trustworthy. Every model is treated as potentially malicious executable code until proven otherwise.
The first requirement is transparency — knowing exactly what's in your AI stack. Traditional software has the SBOM (Software Bill of Materials), a manifest of every library and version in your application. AI needs its equivalent: the ML-BOM, or Machine Learning Bill of Materials.
An ML-BOM tracks things traditional software inventories never had to consider: Where did the training data come from? Who curated it? What fine-tuning steps were applied? Which version of PyTorch serialized the weights? Standards like CycloneDX and SPDX 3.0 now include AI-specific profiles for exactly this purpose.
But documentation alone isn't security. The second layer is cryptographic model signing — essentially giving every approved model a tamper-proof digital fingerprint. Before a model loads into a production system, an admission controller checks its signature against a trusted registry. If the signature doesn't match — because the model was modified, swapped, or pulled from an unauthorized source — it doesn't run. Period.
The third layer is where things get interesting: runtime monitoring. Because poisoned models look clean on static inspection, you need continuous behavioral analysis. Compare outputs against known-good baselines. Throttle queries to prevent model extraction attacks — where an adversary sends thousands of carefully crafted inputs to reverse-engineer your model's behavior. And critically, sanitize all inputs through a guardrail layer before they reach the model, disrupting the carefully crafted payloads that trigger backdoors.
For the full technical methodology behind each of these layers, see our detailed research on securing the ML lifecycle.
When Software-Based Security Isn't Enough
One realization hit me harder than I expected during this research. For certain industries — finance, healthcare, defense — even a perfectly configured software security stack has a fundamental gap. It doesn't protect data while it's being processed.
Your data is encrypted at rest (on disk) and in transit (over the network). But the moment your AI model needs to actually use that data — to make a prediction, generate a response, process a medical record — it has to be decrypted in memory. And at that point, anyone with access to the host machine can theoretically read it. A compromised cloud administrator. An attacker who gained root access. Even a malicious insider.
Confidential computing closes this gap with hardware-backed isolation. Technologies like Intel TDX and NVIDIA's confidential GPU architectures create encrypted memory enclaves where model weights and user data are only decrypted inside a hardware-protected space. Even the cloud provider's own administrators can't peek inside.
Confidential computing means your AI model runs in a vault that not even the building's owner can open.
The most powerful feature is mutual attestation: the model provider can verify their weights are running in a genuine, untampered secure enclave, while the user can verify the enclave is running exactly the approved code. Both sides prove their integrity to each other, cryptographically, without trusting a third party.
This isn't theoretical — it's shipping on current-generation hardware. For organizations handling regulated data, it transforms AI deployment from a compliance headache into a verifiable guarantee.
What About Organizations Already Using Public Models in Production?
This is the most common question I get, and the honest answer is uncomfortable: you probably can't fully verify models you've already deployed. But you can start building the infrastructure to catch problems going forward.
The immediate step is switching to non-executable model formats wherever possible. SafeTensors, for example, stores only numerical weights with JSON metadata — no executable code, no hidden scripts. If your team is still loading pickle files with torch.load(), that's your most urgent fix.
Next, stand up a private model registry. It doesn't have to be elaborate — even a locked-down internal repository where every model goes through automated scanning before it's available to developers. The goal is eliminating the path from "public internet" to "production system" that doesn't pass through a checkpoint.
And for Shadow AI? The answer isn't a memo telling people to stop. It's giving them a faster, easier alternative that happens to be secure. If your approved model pipeline takes three weeks and Hugging Face takes three minutes, you've already lost.
AI Security and Software Security Are the Same Problem Now
One thing became clear as we dug deeper: you can't separate AI security from software supply chain security anymore. Your model might be pristine, but if the Python library running it has a vulnerability, you're compromised. If the container image hosting your training pipeline was tampered with, your model weights are untrustworthy — regardless of how clean your training data was.
This convergence matters because most organizations still have separate teams and separate tools for "software security" and "AI governance." That gap is exactly where attackers will operate. A unified approach — where the model, the dataset, the open-source dependencies, and the infrastructure are all verified through the same pipeline — is the only architecture that holds up under real-world attack scenarios.
The accelerating use of AI-generated code makes this even more urgent. When a developer submits a massive AI-generated code change, the human reviewer faces an impossible task: meaningfully auditing thousands of lines they didn't write, under deadline pressure. The "careful human review" that was supposed to be our last line of defense is collapsing. Automated, cryptographic verification isn't optional anymore — it's the only approach that scales.
I keep coming back to a simple framing: the goal is to make AI deployment boring. Not exciting, not cutting-edge, not a gamble. Predictable, auditable, and secure — like deploying any other critical enterprise system. We're not there yet. But the organizations that start building this infrastructure now will be the ones that can actually trust their AI systems when the stakes are highest.
If your team is wrestling with any of this — Shadow AI, model provenance, securing fine-tuned models — I'd genuinely like to hear what's working and what isn't. This is a problem none of us can solve alone.