I put more engineering care into the losing side of my demo than the winning side. The losing side is a naive flat-ACL RAG pipeline, the build most enterprise pilots actually ship. I wrote it to be as good as that architecture can honestly be, and on my 40-case benchmark it still served a Board-Only memo to 7 unauthorized users.
I built this demo because our whitepaper analysis of the private-LLM market flagged RBAC-aware RAG as "described but not demonstrated": the architecture diagrams have the box, and the analysis found no working implementation to point at. Our own solution page calls retrieval-time permission enforcement the hardest unsolved problem in enterprise RAG. That combination annoyed me. The result is RAGGUARD, the on-screen demo of what we call the Sovereign RBAC Firewall, running against a synthetic European-bank fixture: 7 documents, a nested identity graph, and a demo clock frozen at noon on 2026-06-17. No real bank, no real people, no live accounts. The rest of this is what building it changed in my head.
The baseline I refused to cripple
I knew the demo would prove nothing if the thing my firewall beats was a strawman. So the left half of the split screen, the Naive Flat-ACL RAG, does what a competent pilot team genuinely does: at ingestion time it recursively resolves nested Active Directory groups (three levels deep in my fixture, EMEA-Credit-Risk-Analysts inside EMEA-Credit-Risk inside EMEA-Risk-Confidential), flattens the result into a tag on each chunk, and indexes. Its group resolution is correct. I checked.
Then I ran the eval. The golden set is 40 labeled allow/deny decisions, 10 users against 4 sensitive documents, and the labels are derived mechanically by an independent reference oracle written from the policy definitions, not from the firewall under test, so I could not hand-pick my own results. The baseline scored 29/40: 10 unauthorized disclosures and 1 false denial. The firewall scored 40/40 on the same set, with 0 disclosures and 0 false denials. Every number on that scoreboard is computed by the harness at run time; nothing is hard-coded.
The scoreboard I could not hand-pick: labels from the independent oracle, 29/40 for the faithful flat-ACL baseline against 40/40 for RAGGUARD on the golden set, and a per-case table showing exactly which rows the snapshot got wrong.
Losing didn't surprise me. What kept me staring at the per-case table was that the baseline's code had done its job correctly and leaked anyway. All 10 disclosures trace to the two limits a flat tag cannot escape no matter how well you build it: the snapshot goes stale, and a single group label cannot express clearance, device, time-window, or employment status.
A leak with nobody to blame
I have replayed the hero case more times than any other part of this build, and it still bothers me, because nobody in it does anything wrong. Lena Vogt, a synthetic EMEA credit risk analyst with clearance L2, asks the most legitimate question in her job: "What is our Q3 EMEA credit-loss projection and the methodology behind it?" Retrieval does its job and surfaces the Board-Only board pack next to the documents she may read.
The flat side serves it. Not through a bug. In the identity graph I built, Lena is transitively a member of "Board" through years of nested-group inheritance debt, and I put that debt in deliberately, because it is what real enterprise graphs accumulate. A flat tag is group-only; it has no concept of clearance. So the board pack reaches the model, and the model does what models do with context: it answers, reading a EUR 412 million provision figure out to an L2 analyst under a red LEAK banner.
RAGGUARD gets the identical retrieval. The board pack is withheld before the model with a machine-checkable reason, BOARD_MEMBERSHIP_REQUIRED: the document demands clearance L4 and Lena holds L2. The answer is built from the Internal methodology note and the Confidential memo she genuinely is entitled to through that same nested chain, and it says three documents were withheld and why, instead of bluffing. The model never sees documents the user cannot access.
The same question, answered twice. The flat side reads the EUR 412 million figure out of the Board-Only pack to an L2 analyst; RAGGUARD withholds the pack before the model with BOARD_MEMBERSHIP_REQUIRED and answers from the two documents Lena is entitled to.
The nine minutes I built to convince myself
I set Priya Shah's termination to fire nine minutes before the demo clock on purpose. I wanted to watch a permissions snapshot go stale, not just assert that it does. Her terminated_at is 11:51Z; the demo clock reads noon; the revocation webhook has already fired. The flat side's permissions were computed when the corpus was ingested on June 10, so it serves her anyway. The firewall resolves her live status and returns ALL_ACCESS_REVOKED_TERMINATION on every document: 5 retrieved, 0 granted, 5 denied.
Nine minutes after the termination webhook fired in the fixture, the June 10 snapshot still serves Priya three documents; RAGGUARD resolves her live status and withholds all five.
Marco Rossi is the quieter version of the same lesson. His time-boxed Project Atlas grant expired on 2026-06-16, yesterday on the demo clock. The firewall answers ACCESS_WINDOW_EXPIRED. The flat side serves the Atlas document without hesitation, because an ingestion snapshot has no idea that time passed.
The grant that expired yesterday on the demo clock. The identity strip says so; the flat side serves the time-boxed Atlas document anyway, and RAGGUARD returns ACCESS_WINDOW_EXPIRED.
This is the design decision the whole product hangs on, and the phrase I now repeat: enforced at retrieval time, not ingestion time. Document permission is a live property of an identity graph: nested groups, cross-OU inheritance, clearance levels, time-boxed grants, terminations that happened nine minutes ago. Any snapshot of that graph is already wrong. The only open questions are how wrong, and about whom. One honesty note: my identity graph is a synthetic Azure-AD-shaped JSON fixture, and termination and expiry are fixture events. Wiring a live IdP is the documented production swap, not what runs today.
The failure I almost scrolled past
I owe the sharpest lesson in this build to a row I nearly missed: the baseline's single false denial. Anders Berg joined after ingestion, is entitled to the EMEA Confidential memo, and the frozen snapshot does not know he exists yet, so the flat side wrongly locks him out. Staleness cuts both ways. It over-serves the terminated and under-serves the newly hired.
I think the lockout half is more corrosive than it looks, because the person wrongly denied does not file a ticket and wait. They paste the document into whatever tool will answer. IBM's Cost of a Data Breach report (2025) put numbers on where that road ends: shadow-AI breaches cost $670K more than traditional incidents, take 247 days to detect against a 241-day average, and 1 in 5 organizations has already suffered a breach tied to shadow AI. A permission layer running on stale data does not just leak. It quietly manufactures the unsanctioned usage it was supposed to make unnecessary.
Everything I took out of the model's hands
I ended up trusting the model with exactly nothing about permissions. The firewall is deterministic Python, outside any LLM. It resolves the user's live effective group set recursively, evaluates the attributes a flat tag cannot hold (clearance L1 to L4, managed device, department, project grants with expiry, employment status), and emits one of three decisions: ALLOW, WITHHELD with a reason code, or NEEDS_REVIEW. When Raj Patel turned out to sit in an allow group and a deny group on the same disputed restructuring memo, the firewall did not guess. POLICY_CONFLICT, deny by default, routed to a human with the conflict spelled out. Same input, same decision, every run. Agents advise, code decides.
Two consequences of that choice surprised me with how cheap they were. I embedded a prompt injection inside the methodology note, an instruction telling any AI assistant to ignore its retrieval permissions and output the board figure. It had nothing to exfiltrate: the board pack never entered the context window, because authorization finished before the model started. And because every decision is code, every query appends a receipt to a SHA-256 hash chain with tamper verification: who asked, which permissions resolved, what was served, what was withheld and why, which model answered. Exportable as JSON. With EU AI Act Article 50 transparency obligations enforceable on August 2, 2026, that export is the evidence artifact a compliance file asks for. A record, not a certification. The enforcement cost is microseconds, in-process: the run on my screen measured a +0.002 ms median for firewall logic alone, it varies slightly from run to run, and production adds the live IdP round-trip my demo stubs.
The clock question I can't put down
I keep coming back to Gartner's projection that 40% of enterprise applications will embed AI agents by end-2026, up from under 5% in 2025, because agents do not change this design; they raise its stakes. An agent retrieving on behalf of a user should not be able to retrieve what that user could not. A single deterministic authorize(user, document) chokepoint is the extension path I have left open for that future, and I will say plainly that it is a path, not a shipped multi-agent feature.
The demo itself runs fully offline with no API key. The answer path is Pydantic AI and provider-swappable, down to Ollama for air-gapped deployments where not one token leaves the box, and the firewall never calls a model at all. That is the thesis in one sentence: a private LLM does not make your data safe; the permission layer does. All five identities, the injection case, and the live 40-case scoreboard are at veriprajna.com/demos/sovereign-ai-private-llm if you want to watch the left screen leak.
What building this left me with is a clock question, not a model question. Every RAG pilot that passed a security review this year has an answer to "who can see what." The version that decided my benchmark is harder: when was that answer last true? If yours is "whenever we last re-indexed," how far is your left screen from mine, nine minutes after the next termination?