Healthcare AI That Cannot Harm Patients by Design

The AI Your Clinicians Are Already Using Has No Safety Layer

Two-thirds of Epic's provider base now has access to GPT-4-generated clinical content. Ambient scribes draft notes. In Basket replies go to patients. MyChart messages get auto-generated responses. And in a peer-reviewed study of real clinical deployments, 2.9% of AI-generated encounter summaries contained hallucinations or factual errors, including incorrect medication statuses and wrong procedure dates. That number sounds small until you consider that 22.7% of providers reported skipping full-length notes entirely in favor of the AI summary. The safety layer between the LLM and the patient does not exist in most EHR integrations today. Epic's CDS Hooks architecture was not designed to validate LLM-generated text against a patient's active problem list, medication list, and allergy list in real time. That is the gap we build into.

This is not a theoretical concern. When researchers tested six leading LLMs with clinical vignettes containing a single planted fake lab value, the models repeated or elaborated on the false information in up to 83% of cases. A simple mitigation prompt halved the rate but did not eliminate it. In clinical workflows where an AI-generated note feeds directly into coding, ordering, and billing, a hallucinated lab value or fabricated medication history can propagate through the entire care chain before a human intervenes.

The $2.3 Billion Coding Problem Nobody Planned For

Blue Cross Blue Shield's March 2026 analysis of commercial claims data found that AI-enabled coding practices are associated with a $2.3 billion increase in healthcare spending: $663 million inpatient, $1.67 billion outpatient. The debate over whether this represents upcoding or improved documentation accuracy misses the point. If your ambient scribe captures more clinical detail from the encounter but nobody validates whether that detail is accurate, you have built an expensive machine for generating higher-acuity claims from the same clinical reality. Providers and payers both agree ambient scribes are raising costs. Neither side has a solution because the problem sits between the scribe and the coder, in a verification layer that does not exist yet.

We build that verification layer. Not by replacing ambient documentation tools or competing with coding engines, but by inserting deterministic clinical validation between the AI-generated note and the downstream systems that act on it. The check is structural: does this documented finding appear in the patient's record? Does this assessment match the documented evidence? Is this level of complexity supported by the encounter data? These are not probabilistic judgments. They are rule-based validations against known clinical facts.

1,451 FDA-Authorized AI Devices and a Recall Problem

The FDA authorized 295 AI-enabled medical devices in 2025 alone, the most in its history, bringing the cumulative total to 1,451. But a review of 60 FDA-authorized AI devices found 182 recalls, with nearly half occurring within a year of approval. The TruDI Navigation System logged 100+ malfunctions and adverse events after AI integration, compared to seven before. Only 5.2% of authorized AI devices have reported any adverse event data at all, which means we are flying blind on the postmarket safety of the other 95%.

The December 2024 PCCP final guidance opened a path for adaptive AI devices to evolve post-clearance without new submissions, but most manufacturers are not equipped to implement the monitoring, validation, and documentation infrastructure that PCCPs require. We build that infrastructure: the continuous monitoring systems, the performance validation pipelines, and the regulatory documentation frameworks that turn a PCCP from a filing strategy into an operational reality.

Governance Gap: 23% Have Frameworks, 78% Are Deploying

Only 23% of health systems have established formal AI governance structures, yet 78% plan to deploy clinical AI within the next 24 months. The AMA released its AI Governance Toolkit in August 2025. The Coalition for Health AI co-branded its Blueprint with the Joint Commission in September 2025, signaling governance expectations to 22,000+ accredited organizations. ONC's HTI-1 rule requires 31 source attributes for predictive decision support interventions and mandates public disclosure of intervention risk management practices. Section 1557's nondiscrimination rule extends to AI-based clinical decision tools with a May 2025 compliance deadline. OCR's AI enforcement actions rose 340% in 2025.

The regulatory surface area is expanding faster than most health systems can staff governance committees. We stand up AI governance programs that satisfy ONC, OCR, CMS, and Joint Commission requirements simultaneously, because a health system deploying Epic's ambient scribe, a predictive sepsis model, and a patient-facing chatbot has three AI risk profiles sharing patient data. Governing them in silos, by vendor, by department, or by compliance requirement, is how organizations end up with overlapping policies, conflicting risk assessments, and gaps that regulators find before the governance committee does.

Pharma AI: 173 Clinical Programs, Zero Approvals, Real Validation Gaps

More than 173 AI-discovered drug programs are in clinical development. Insilico Medicine published the industry's first clinical proof-of-concept for an AI-discovered drug in Nature Medicine in June 2025. Recursion discontinued its lead candidate REC-994 in May 2025 after long-term data did not confirm earlier efficacy trends. Major pharma companies have committed billions to AI partnerships: Novo Nordisk-Valo Health ($2.76B), Eli Lilly-Isomorphic Labs ($1.75B), Bayer-Recursion ($1.5B). No AI-discovered drug has received FDA approval as of early 2026.

The validation gap in pharma AI is not model performance. It is regulatory documentation. When AI touches GxP-regulated processes, every algorithmic decision needs a 21 CFR Part 11 compliant audit trail. The FDA's January 2025 draft guidance introduced a risk-based credibility assessment framework, and the January 2026 joint FDA-EMA principles emphasize fitness for purpose and robust data governance. We build the validation infrastructure that connects AI model outputs to the documentation frameworks regulators actually review: credibility assessment plans, context-of-use definitions, and audit trails that trace every algorithmic decision from training data through clinical endpoint.

The Payer AI Reckoning

CMS launched the WISeR pilot in January 2026, testing AI-powered prior authorization for Medicare beneficiaries in six states. Within months, physicians reported increased denials, longer wait times for peer-to-peer reviews, and patient access delays. The program's financial structure pays vendors based on a share of "savings," creating a direct incentive to deny. The EFF filed suit seeking transparency on vendor financial incentives. Democrats introduced legislation to repeal the pilot. Meanwhile, state legislatures are moving independently: Texas prohibits automated adverse determinations without human oversight, California's SB 243 regulates AI companion chatbots, and Illinois and Nevada have banned AI for behavioral health entirely.

For payers deploying AI in utilization management, the question is no longer whether regulation will arrive but whether your system can demonstrate the audit trail, the clinical rationale documentation, and the human oversight integration that every pending regulation requires. We build algorithmic compliance frameworks for payer AI that document decision rationale at the individual claim level, integrate mandatory human review triggers, and generate the transparency reports that CMS, state AGs, and plaintiff attorneys will eventually demand.

Why Not a Big Consulting Firm?

Deloitte, Accenture, and McKinsey all have healthcare AI practices. They produce governance frameworks, maturity assessments, and strategy roadmaps. They are good at organizational change management and executive alignment. They do not build clinical safety systems. When your Epic integration needs a real-time validation layer that checks AI-generated medication recommendations against a patient's allergy list within CDS Hooks, that is an engineering problem that requires both clinical informaticist depth and systems integration capability. The big firms subcontract that work. We do it directly. When your pharma team needs 21 CFR Part 11 compliant audit trails for an AI model in a GxP process, we build the pipeline, not the PowerPoint describing what the pipeline should look like.