- The July 19 2024 CrowdStrike outage that crashed millions of Windows machines wasn't a hack or a bad model. It was a schema mismatch: an update declared 21 fields where the kernel interpreter expected 20. An out-of-bounds read that no independent layer was watching for 🧵
- That is the structural gap. Nothing sits between a software vendor's update pipeline and your production fleet. The vendor's own validator is self-policing. SBOM tools cover open-source dependencies, not proprietary channel files. Change boards wave updates through.
- So we built Kestrel, an independent control plane that reads a vendor update before it reaches any endpoint. We replayed the July-19 channel file against a synthetic 8,500-endpoint fleet. It blocked the rollout before a single production machine rebooted.
- The verdict is set by code. An LLM crew advises, then a deterministic verifier and policy gate written in plain Python decide, re-derivable arithmetic a regulator can re-run. An advisory agent leaning "allow" can't clear a critical finding it flagged.
- Four deterministic checks fire on the replay. Schema diff: 20 fields expected, 21 provided, the literal July-19 root cause. Sandbox model: boot failure on 5 of 6 fleet profiles. Blast radius: 100% of the fleet in one wave, no canary plan declared.
- The fourth check is the cruel one. The crashing agent runs pre-boot and is itself the rollback receiver, so it can never re-initialize to accept a rollback. That is why the real recovery meant fixing machines by hand in Safe Mode. Kestrel flags that loop before it can happen.
- The sandbox signal is derived independently of the schema check, so the 5-of-6 boot failure corroborates the field-count finding by a fully separate route. Two independent proofs from two different signals, arriving at one verdict.
- It stays out of the way of good traffic. The same vendor's benign update, declared schema 20 to 20 with a staged canary, clears every check and ships to a 1.2% canary ring in seconds. 0 false-blocks across the 6 benign fixtures in the set.
- Every decision exports a signed evidence record, HTML plus JSON with a sha256 hash, the proofs, the rules that fired, and a per-step trace. On a 12-item labeled fixture set: 12/12 correct, 0/6 false-blocks, about $13.3M estimated prevented downtime, shown with its formula.
- Question for security and SRE folks: what inspects a vendor's channel file on its way into ring 0, before it hits production? If the honest answer is "the vendor's own validator," who is checking the checker? #EndpointSecurity #SupplyChainSecurity #DevSecOps #SRE
- Built as a demo on a synthetic fleet, not a deployment. Real: the deterministic gate, the four checks, the signed record. The vendor feed and ITSM hooks are stubs. See how it works: https://veriprajna.com/demos/software-update-integrity
Published on X · September 11, 2026
On social media
See this post on its original platform
In our archive