I've traced most of what we built on eligibility matching back to one protocol flag in a Medidata Rave audit log — an eligible patient flagged ineligible because the matching system saw "catheter" in both the criterion and the chart note and called it a match. That word-not-concept conflation is the error class that LLM-based trial matching can't fix with better prompts. It's what pushed me to build a different architecture.
The protocol was a Phase III anticoagulant trial. The patient's chart contained central venous catheter placement, performed in the ICU for IV access. The protocol excluded cardiac catheterization. The system saw catheter-heart-venous proximity and returned a match. Two procedures on different branches of the SNOMED-CT hierarchy — cardiac catheterization (SCTID 41976001, under "Procedure on heart") and central venous catheterization (SCTID 392230005, under "Catheterization of vein") — treated as equivalent because they share vocabulary. The patient enrolled two weeks late.
That flag is what led me to build Clinical Trial Recruitment AI — not as an improvement to existing matching, but as a fundamentally different approach to the problem.
The Algorithm Reads Words, Not Concepts
My first instinct, working through the catheter error, was to improve the prompt. Add clarifying language around which catheter types the protocol excluded. A few variations in, I hit the underlying constraint: any clarification I added made the system more accurate on that specific criterion and less reliable on adjacent ones. LLMs optimize toward the examples you give them. The error wasn't a gap in prompt specificity. It was architectural — the model has no representation of the SNOMED-CT hierarchy, so it can't traverse it.
That realization sent me to the concept graph itself. What I found in working through SNOMED-CT is that the catheter distinction is a clean example of a general principle: two terms that share surface vocabulary can sit on entirely different ontological branches. "Coronary angiography" and "peripheral angiography" share the word "angiography." SNOMED-CT puts them under different parent concepts. Published evaluations of clinical AI matching tools confirm this class of error appears reproducibly across multiple systems and therapeutic areas (Fierce Biotech, 2025).
What the SNOMED-CT architecture gives you that prompt engineering can't is that the hierarchy is the answer, not an input to the model. I'm not asking the system to reason about which catheter is which. I'm asking it to look up both procedures in a pre-existing concept graph and return whether they share a parent. No ambiguity, no probability estimate — a traversal result.
The Accuracy Benchmark I Stopped Trusting
I spent real time on the Tempus Patient Query accuracy result — 94.39% across 196 evaluation queries after the Deep 6 AI acquisition — trying to understand what it was measuring. The benchmark is real and the improvement over keyword search is genuine. What I couldn't find in the published data was a breakdown of where the 5.6% failure rate concentrated.
If failures are uniformly distributed across criteria types, 94% is a strong result. My concern, built from working through the kind of errors the catheter case represents, is that the failure rate isn't uniform — that it clusters around criteria involving vocabulary overlap across concept branches. That class of criteria isn't evenly distributed across all protocols; it's concentrated in cardiovascular, vascular, CNS, and biomarker-driven trials where procedure naming conventions create exactly the ambiguities LLMs conflate.
Complex protocols now average 27 eligibility criteria, up from 17 in the 1986-1995 era, with many containing nested conditionals and temporal constraints (IQVIA, 2026). The more criteria a protocol has that touch anatomy-adjacent procedure naming, the more the failure-rate distribution matters. I think about IQVIA.ai — 250 million patient records combined with NVIDIA's NeMo stack in a unified agentic platform launched March 2026 — the same way. That scale addresses the data availability problem. It doesn't address the concept hierarchy problem, which lives in unstructured clinical notes regardless of how many records you have.
Benchmark accuracy is a population statistic. The error class I'm describing is a distribution question: where does the failure rate concentrate, and is your protocol in that concentration?
The Coordinator Who'd Given Up on Matching
I remember the site coordinator who showed me the matching tool her team had stopped using. She was managing five studies simultaneously, spending 40-60% of her time on pre-screening work. The tool was supposed to reduce that load.
She walked me through the match list it had generated. One in three flagged patients turned out ineligible on manual review. She'd done the math herself: the tool was adding roughly as much work as it was saving, because every false positive required manual verification. She'd gone back to doing pre-screening manually.
The pattern she described is documented broadly — false positive rates above 30% drive tool abandonment within three months, and 37% of trial sites under-enroll in the studies they're assigned (Antidote.me). The enrollment math behind those statistics is where I started to understand the full cost of precision failure: $800K per day in lost prescription sales per day of trial delay (Tufts CSDD, 2024). The coordinator's experience was the human version of that number, playing out in one site's workload.
What she showed me wasn't a training problem. A tool that generates enough false positives that manual verification takes longer than not using it isn't a change-management challenge. The precision failure is the product failure.
What I took from that conversation was that the catheter-class errors weren't showing up in benchmark reports. They were showing up in coordinators' workdays, in the time spent verifying flags that shouldn't have been flags.
Why the Audit Trail Became the Product
I spent more time on the FDA's January 2026 CDS guidance than I expected to when we were finalizing the architecture. Specifically on the question of what makes a matching function non-device clinical decision support rather than a regulated device — the classification that keeps the system outside 510(k) territory.
What I found is that the key variable is traceability. A system producing a confidence score for eligibility doesn't satisfy the non-device CDS standard; the output isn't specific enough for a clinician to follow the reasoning. A system producing a concept-path audit — which SNOMED-CT concept IDs were evaluated, which branch comparisons were made, which criterion each exclusion traces back to — does. My regulatory team's framing that stayed with me was that the audit trail isn't something the FDA added to the guidance — it's the evidence that a clinician is in the loop.
When we decided to deploy on-premise, inside the institution's firewall, via FHIR R4, I was thinking about two constraints simultaneously. The 6-12 month Epic App Orchard security review is real — cloud-based clinical AI tools get stuck there routinely. Running natively within the institution's own infrastructure bypasses it entirely. The second constraint was HIPAA: the Security Rule updates proposed in January 2025 increased scrutiny around cloud transfer of patient data. On-premise deployment keeps the computation inside the institution's environment. No cloud transfer required.
What I realized partway through the architecture design is that the audit trail and the on-premise choice were actually solving the same problem from different angles — the need for the institution to control and trace every step of the eligibility decision. They converged on the same principle.
The Objection I Keep Coming Back To
I keep getting a version of the same objection from pharma sponsors who've already looked at Tempus, IQVIA, or ConcertAI: if the major platforms are investing at this scale — and they are, ConcertAI launched ACT at SCOPE in February claiming 10-20 month timeline reductions — why build custom?
My honest answer: for straightforward protocols in therapeutic areas the platforms have optimized for, the case for buying in is real. The data networks, the regulatory relationships, the integration depth — a custom build takes years to replicate.
The calculation shifts for protocols that fall outside the typical case. Rare disease, CNS, biomarker-driven eligibility criteria that require ontology coverage the platforms don't publicly carry. It also shifts when you're managing a protocol library with specific amendment patterns. The average amendment now takes 260 days to implement across the industry (Applied Clinical Trials, 2025); on an ontology-based logic layer, an amendment is a versioned update to the concept graph, not a retraining cycle.
The catheter error I traced at the beginning of this piece was already in the matching logs before anyone noticed the enrollment delay. The logs were technically available. No one was reading them through an ontology lens. I think there are trials running that same error pattern right now — at varying rates, in different criteria classes — in logs that aren't being audited for it. That's what I'd want any team to check before they assume their current matching architecture has the coverage problem handled.