In February 2024, security researchers at JFrog found over 100 malicious AI models sitting on Hugging Face — the platform most companies treat like a trusted app store for machine learning. Some of those models contained silent backdoors that would execute code the moment you loaded them. Not when you ran them. When you loaded them. That's the equivalent of a Word document reformatting your hard drive the instant you open the file.
This wasn't a theoretical risk in a conference paper. These were real models, publicly available, designed to give attackers a persistent foothold inside whatever network downloaded them. And the platform's built-in safety scanner? It flagged so many harmless models as dangerous that its false-positive rate exceeded 96% — meaning teams had learned to ignore the warnings entirely.
I've spent the last two years building AI systems for enterprises that can't afford to get this wrong. What I've learned is that the AI supply chain is broken in ways most organizations haven't begun to reckon with. The threat isn't just bad outputs. It's compromised infrastructure, poisoned models that pass every evaluation, and an entire economy of "AI solutions" built on architectural sand.
The Model File Isn't Just Data — It's Executable Code
Most people think of an AI model as a big file full of numbers — weights and biases that tell the system how to make predictions. That mental model is dangerously wrong.
The most common format for sharing Python-based AI models is called "pickle." And pickle isn't a data format. It's a tiny virtual machine. When your team downloads a model and loads it with a standard command like torch.load(), pickle doesn't just read numbers — it executes instructions to reconstruct the model object. An attacker can hide arbitrary commands inside that reconstruction process.
The payloads JFrog found weren't subtle. They opened reverse shells — essentially handing the attacker remote control of the machine. From there, they could move laterally through the corporate network, access training data, or poison internal models that feed your production systems.
A malicious AI model doesn't need to run inference to compromise you. It just needs to be loaded.
There are safer formats. One called "safetensors" was designed specifically to be a pure data container — no code execution capability by design. But adoption is slow because the ecosystem was built around pickle, and inertia is a powerful force. We've made safetensors our default standard for every deployment, and I'd urge any team handling model files to do the same.
Why Scanning Doesn't Save You
Hugging Face, to their credit, partnered with Microsoft to build a scanner called Picklescan. It works like an antivirus: maintain a blacklist of dangerous function calls, flag any model that triggers one.
The problem is the same one antivirus software has faced for decades. Attackers obfuscate. They use legitimate functions in malicious sequences. They find paths around the blacklist. And because the scanner is so aggressive with false positives — over 96% of flagged models are harmless — security teams develop what I'd call scanner fatigue. They stop investigating alerts.
Through deeper data-flow analysis, researchers recently identified 25 zero-day malicious models that the standard scanners completely missed. These weren't flagged. They sailed right through.
I had a conversation with my team after we dug into this research that changed how we think about the problem. We'd been treating model security as a checkpoint — scan on download, clear to deploy. That's wrong. It's like checking someone's ID at the front door and then giving them unsupervised access to the vault. We explored the full scope of this threat landscape in our interactive analysis of AI supply chain security, and the depth of the problem surprised even us.
Fine-Tuning Destroys the Safety You Paid For
Even if you source a clean model from a reputable provider, the most common thing enterprises do next — fine-tuning it on proprietary data — introduces its own catastrophic risks.
NVIDIA's AI Red Team tested this systematically. They took models that had been carefully aligned for safety through months of human feedback training and fine-tuned them on domain-specific data. The results were devastating. A Llama 3.1 8B model's resilience against prompt injection attacks dropped from 0.95 to 0.15 — a collapse from robust to nearly defenseless — after a single round of fine-tuning.
Think of it this way: the original model developers spent enormous resources teaching the model to refuse harmful requests. Fine-tuning is like renovating a house — you're adjusting the structure to fit your needs, but in the process, you're accidentally ripping out the fire suppression system. The safety behaviors get overwritten because they live in the same weights you're optimizing for task performance.
Fine-tuning a model for accuracy often means un-training it for safety. Most enterprises don't measure this trade-off until something breaks publicly.
And then there's the more sinister version: deliberate model poisoning. Researchers have demonstrated that corrupting just 0.001% of training data — one million tokens out of a hundred billion — produces a 5% increase in harmful outputs. At 1% corruption, safety guardrails collapse almost entirely.
The most dangerous variant is what the research community calls a "sleeper agent." A poisoned model that behaves perfectly across every benchmark, every evaluation, every safety test — until it encounters a specific trigger phrase or input pattern. Then it switches modes. It might leak confidential data, provide deliberately wrong medical advice, or execute unauthorized actions. You'd never catch it in testing because the trigger was designed to never appear in test data.
Shadow AI: The Threat You Can't See Because You Don't Know It Exists
While security teams agonize over the models they've approved, 98% of organizations have employees using AI tools that were never vetted, never approved, and never secured. This is Shadow AI, and it dwarfs the risks I've described so far.
A marketing analyst pastes customer data into ChatGPT to write a report. A developer uses a random open-source model from GitHub to prototype a feature that quietly makes it into production. A finance team builds a forecasting tool on an unvetted API. Each of these creates an attack surface that no security framework covers because no one knows it exists.
The numbers are stark:
43% of employees share sensitive data with AI tools without permission
Shadow AI breaches cost organizations $670,000 more on average than traditional breaches
63% of organizations lack any formal AI governance policy
97% of AI-related breaches lack proper access controls
But the risk that keeps me up at night is something called model disgorgement. This is a legal remedy where regulators force you to destroy an entire AI model — and any products built on it — because it was trained on illegally obtained or tainted data. You can't surgically remove bad data from neural network weights. It's baked in. If a Shadow AI model contaminated with stolen intellectual property gets integrated into your production system, the regulatory response could require you to delete the product entirely.
The Wrapper Economy Is a Liability, Not a Strategy
Most AI consultancies sell what I'd call "wrappers" — a nice interface layer that connects your data to someone else's API. OpenAI, Anthropic, Google. The wrapper handles the prompt engineering and maybe adds some retrieval logic. It ships fast and demos well.
It is also structurally incapable of being safe for high-stakes work.
The fundamental issue: large language models are prediction engines. They generate the statistically most likely next word given the context. That's phenomenal for drafting emails. It's a liability for pricing decisions, legal policy application, or medical recommendations — any domain where being confidently wrong has consequences.
We've already seen what happens:
A Chevrolet dealership's chatbot was tricked into agreeing to sell a $76,000 vehicle for one dollar
Air Canada was held legally liable when its chatbot invented a bereavement fare policy that didn't exist — the court rejected the argument that the AI was a "separate entity"
DPD's customer service bot was manipulated into writing poetry about how terrible the company was
Every one of these systems relied on system prompts and post-processing filters for safety. That's like putting a "please don't rob us" sign on a bank vault instead of installing a lock.
Helpful AI, when unguarded, is dangerous AI. Safety cannot be a suggestion bolted on after deployment — it has to be an architectural constraint built into the system's bones.
And for any company operating outside the US, there's an additional trap. If you use a US-based API, your data falls under the US CLOUD Act, which lets American law enforcement compel access regardless of where the servers sit. Even providers promising "zero data retention" typically maintain a 30-day abuse monitoring window. For regulated industries — healthcare, finance, defense — this is a non-starter.
What We Built Instead: Making AI Prove Its Work
After watching these failures accumulate, my team took a fundamentally different approach. Instead of adding more guardrails to a probabilistic system, we built a system where every output has to be proven before it's delivered.
We use what's called a Neuro-Symbolic architecture — combining the language fluency of neural networks with the logical rigor of symbolic AI. The neural layer handles understanding and generating natural language. The symbolic layer, built on a knowledge graph, acts as a fact-checker that validates every claim against structured, verified data before anything reaches the user.
The difference in outcomes is measurable. Standard LLM wrappers hallucinate between 1.5% and 6.4% of the time. Our system operates below 0.1%. On clinical data extraction tasks, wrappers achieve 63-95% precision. Ours hits 100% — because if a fact isn't in the knowledge graph, the system returns nothing rather than guessing.
We also use what we call Semantic Routing to handle adversarial inputs. Before a user's query ever reaches the language model, it passes through a vector similarity check against known attack patterns — prompt injections, system override attempts, social engineering tactics. If the query matches a malicious pattern, it gets routed to a deterministic handler. The language model never sees it. You can't trick a model that never receives the trick.
For complex tasks, we decompose the work across specialized agents — a Researcher that can only query the knowledge graph, a Writer that can only work with the Researcher's output, and a Critic that extracts every claim from the draft and validates it against the source data. No single agent has enough autonomy to deviate from verified truth. For the full technical methodology behind this architecture, see our detailed research on verifiable intelligence.
"Can't We Just Use Better Prompts?"
This is the most common pushback I hear, and the answer is no. Prompt engineering is not a security strategy. It's a suggestion to a system that has no obligation to follow it. Every major prompt injection attack works precisely because the model can be convinced to override its instructions. You're fighting the model's architecture with the model's own input channel — and the attacker has the same channel.
Another question I get: "Isn't this overkill for our use case?" Maybe. If you're using AI to suggest meeting times or summarize internal memos, a wrapper is probably fine. But the moment AI touches pricing, compliance, customer-facing decisions, or anything where being wrong has legal or financial consequences, the calculus changes completely. The Air Canada ruling made this clear — you are liable for what your AI says, full stop.
And for teams already deep in their AI journey: start with an audit. Catalog every model in use, including the ones nobody approved. Implement cryptographic model signing so your inference engine refuses to load anything without a verified signature. Generate an AI Bill of Materials — a complete inventory of every dataset, library, and framework version in your pipeline — so when the next vulnerability drops, you know exactly what's exposed.
The Supply Chain Problem Is a Trust Problem
The AI industry built itself on implicit trust — trust that open-source models are safe, that fine-tuning preserves alignment, that API providers protect your data, that scanning tools catch threats. Every one of those assumptions has been publicly disproven.
What replaces trust is verification. Cryptographic proof that a model hasn't been tampered with. Symbolic logic that checks outputs against known facts. Architectural constraints that make certain failures impossible rather than merely unlikely. Sovereignty over your own models, data, and infrastructure so that your AI's behavior is determined by your policies, not someone else's terms of service.
The incidents of 2024 weren't anomalies. They were the opening chapter of what happens when an industry scales faster than its security practices. The organizations that recognize this now — and invest in verifiable, sovereign AI systems — will be the ones still standing when the next wave of attacks arrives.
I'd genuinely like to hear how your organization is thinking about AI supply chain security. Are you auditing your model sources? Have you encountered Shadow AI internally? The conversation matters more than any single solution.