Clinical AI Safety & Governance
Ambient scribes drafting clinical notes. Patient portal AI sending messages on your physicians' behalf. Sepsis models firing alerts. Triage algorithms routing patients. Each tool has its own accuracy claims, its own safety profile, and its own blind spots. The question is not whether your AI works. The question is whether you can prove it, across every patient demographic, when a regulator, a plaintiff's attorney, or a journalist asks.
7.1%
AI-drafted messages posed severe patient harm risk
Lancet Digital Health, April 2024
66.6%
Of harmful errors missed by reviewing physicians
Lancet Digital Health, April 2024
14%
Increase in AI-related malpractice claims since 2022
Medical Economics, 2025
Veriprajna builds the safety infrastructure that sits between your clinical AI tools and your patients. Independent assessments, bias monitoring, governance architecture, and regulatory compliance engineering. Vendor-neutral. Evidence-based. Built for the CMIO who needs answers, not marketing decks.
Clinical AI fails in specific, documentable ways. Each failure mode has its own evidence base, its own regulatory response, and its own technical mitigation. Understanding the distinction matters because the governance controls for each are different.
The AI generates plausible but wrong clinical content, and the physician trusts it.
A hospitalist reviews an AI-drafted MyChart response to a patient asking about a new medication. The draft recommends continuing metformin and notes that the patient's last HbA1c was 6.8%. The physician scans it in 12 seconds and clicks send. The problem: the patient's creatinine has been rising over three visits, and the AI did not flag the renal function decline that makes metformin contraindicated. The physician, trusting the AI's contextual awareness, did not independently check the labs. The draft was linguistically perfect, empathetic, and wrong.
This is not a hypothetical. The Lancet study documented that when AI drafts are well-written and empathetic, physicians enter a cognitive state where the quality of the prose substitutes for independent clinical verification. Ninety percent of physicians in the study reported trusting the AI's performance. The error catch rate was 33.4%.
In a Q1 2025 pilot at three hospitals, an AI discharge assistant recommended a medication for a patient explicitly listed as allergic to that drug class. The error was caught by a nurse, not the reviewing physician. The system's actual clinically actionable misstatement rate was 0.98%, twelve times higher than the vendor's claimed 0.08%.
The vendor says 99.999%. The Texas AG says prove it.
In September 2024, the Texas Attorney General settled with Pieces Technologies over its claim of a <0.001% "critical hallucination rate" for clinical documentation software deployed at Houston Methodist, Children's Health, Texas Health Resources, and Parkland. The AG did not need AI-specific legislation. Existing consumer protection law was sufficient to challenge unsubstantiated accuracy claims.
The five-year Assurance of Voluntary Compliance now requires Pieces to disclose metric definitions, calculation methodologies, training data, and known harmful uses to every customer. This precedent applies to every clinical AI vendor operating in the US. If your vendor claims a specific error rate, you should be asking: calculated on what dataset? Validated by whom? Over what time period? On which patient demographics?
Texas followed the settlement with the Responsible AI Governance Act (June 2025), establishing civil penalties of $80,000-$200,000 per uncurable violation. Colorado's AI Act takes effect June 30, 2026. The EU AI Act's high-risk classification for clinical AI takes effect August 2, 2026, with penalties up to EUR 15 million or 3% of global turnover.
Your model performs differently depending on who the patient is. You may not know.
Pulse oximeters overestimate blood oxygen saturation by 0.6-1.5 percentage points in patients with darker skin tones. Black patients are nearly three times more likely to experience occult hypoxemia that the device does not detect. When your AI triage system uses SpO2 as an input feature, it inherits this bias. A patient with true arterial oxygen at 88% whose pulse oximeter reads 93% will not trigger a high-priority alert set at 92%. The algorithm did not discriminate. The data it ingested was already wrong.
The problem compounds in predictive models. The Epic Sepsis Model claimed an AUC of 0.76-0.83 internally. External validation at Michigan Medicine showed an AUC of 0.63, with sensitivity of just 33% (missing two-thirds of sepsis cases) and a positive predictive value of 12% (88% false alarm rate). It alerted before clinicians in only 6% of cases. Black and Hispanic patients, who experience nearly double the sepsis incidence, face the worst performance from models trained predominantly on data from white patient populations.
In maternal health, AI early warning systems missed 40% of severe morbidity cases in Black patients (California Maternal Data Center). Black women face a pregnancy-related mortality rate of 49.5 per 100,000 live births, 3.4 times higher than white women. When these patients are also 1.79 times more likely to die once a complication occurs ("failure to rescue"), the gap between what the algorithm detects and what the patient needs is measured in lives.
This table is designed to be pulled up in your next AI governance meeting. It covers the categories of tools you are likely already running or evaluating, with honest assessments of where each category falls short. Some gaps point to Veriprajna's capabilities. Others point to organizational challenges that no vendor can solve for you.
| Category | Key Players | What They Do Well | Where They Fall Short |
|---|---|---|---|
| Ambient Documentation | Nuance DAX (Microsoft), Abridge, Ambience Healthcare | Reduce documentation burden by 50-79%. Abridge and Nuance offer linked-evidence traceability. Deep EHR integration (Abridge is Epic's first Pal). | None publish independent, peer-reviewed hallucination rates stratified by clinical specialty. Accuracy is self-reported. No vendor provides demographic performance breakdowns. |
| Clinical Decision Support | Epic (built-in), Viz.ai, Aidoc, Pieces Technologies | Viz.ai has multiple FDA clearances across 1,400+ hospitals. Aidoc cleared for 14-condition abdominal CT triage with 97% sensitivity. | Epic's built-in models (e.g., ESM) showed poor external generalization. Proprietary models often lack independent validation. Subgroup performance data rarely disclosed. |
| AI Governance Platforms | Censinet, Credo AI, Holistic AI, IBM watsonx.governance | Censinet offers healthcare-specific risk management. Credo AI maps regulatory requirements. IBM provides enterprise-scale lifecycle governance. | Governance platforms manage process. They do not test clinical AI for hallucinations, run adversarial probes, or measure demographic performance on your patient data. |
| Hallucination Detection | Vectara (HHEM-2.1), Arthur AI, Galileo | Vectara's HHEM model benchmarks faithfulness. Arthur AI provides full-lifecycle ML monitoring. | General-purpose tools not calibrated for clinical text. "Consider metformin" may be correct for Type 2 diabetes but dangerous for renal impairment. Context-dependent detection requires clinical grounding. |
| Big 4 / Large SIs | Deloitte, Accenture, McKinsey, EY | Enterprise change management. Board-level credibility. Large teams for multi-year implementations. | They implement platforms, not build clinical AI safety infrastructure from the ground up. Engagements start at $500K-$5M+. Generalist teams rotate; domain depth stays shallow. They recommend governance frameworks. They rarely test models against your data. |
| Internal Teams | Your informatics, compliance, and IT teams | Know your workflows, your data, your politics. Essential for sustained governance. | Most health system informatics teams lack adversarial AI testing capability, fairness metric computation infrastructure, and bandwidth for cross-vendor bias monitoring. This is a resourcing gap no external vendor fully solves. Veriprajna can build the infrastructure and train the team, but sustained monitoring requires internal capacity. |
Every engagement starts with your deployed AI tools and your patient population. We do not sell a platform. We build the safety infrastructure that your governance committee and clinical teams need to make defensible decisions about clinical AI.
We test your clinical AI tools against your patient population, not generic benchmarks. For each tool, we measure hallucination rates across clinical specialties, compute sensitivity/specificity/PPV stratified by race, sex, and age, probe for prompt injection and data leakage vulnerabilities, and benchmark vendor claims against independently observed performance.
We reach for Med-HALT-derived testing protocols adapted for clinical documentation, not generic faithfulness metrics. For ambient scribes, we compare AI-generated notes against physician-verified encounter records to compute factual concordance rates by note section (HPI, assessment, plan). For CDS tools, we run retrospective analyses on your historical data to measure alert accuracy by demographic subgroup.
We design and operationalize the governance infrastructure your committee needs to move beyond a charter into enforceable oversight. This includes vendor evaluation scorecards with weighted criteria (clinical validation, demographic performance, regulatory certifications, interoperability), risk-tiered approval workflows calibrated to clinical proximity, model card templates, and post-deployment monitoring dashboards.
We align governance controls to NIST AI RMF and ISO 42001 because these frameworks create the rebuttable presumption of compliance under Colorado's AI Act. We also build shadow AI detection protocols to identify and govern clinician-adopted tools outside institutional oversight.
We build continuous monitoring systems that track equalized odds, PPV/NPV stratification, and Population Stability Index across demographic groups for every clinical AI tool you deploy. When your sepsis model's sensitivity drops for Hispanic patients or your triage algorithm inherits pulse oximetry bias in darker-skinned patients, you know within days.
We account for the upstream data problem. Pulse oximeters overestimate SpO2 in darker-skinned patients. The FDA's January 2025 draft guidance now recommends testing on 150+ diverse participants using the Monk Skin Tone scale, up from 10. We build monitoring that flags SpO2-to-vital-sign discrepancies and tracks whether your AI models' performance correlates with known sensor bias patterns.
We translate AB 3030 (California), Colorado AI Act (SB 24-205), EU AI Act Annex III, and the Texas AG settlement precedent into technical controls and operational workflows. Disclosure templates with per-medium specifications. Meaningful review interfaces that combat automation bias. Audit trail architectures that satisfy AG investigations and Joint Commission accreditation. Vendor contract language reflecting post-Pieces transparency requirements.
For the Colorado AI Act specifically, we map each of your deployed AI tools against the "consequential decision" definition, determine which qualify for the HIPAA provider-recommendation exemption, and build the annual review and impact assessment documentation the law requires.
We simulate adversarial scenarios against your clinical AI systems before a bad actor or an edge case does it for you. Hallucination probing with domain-specific clinical edge cases (drug interactions in polypharmacy patients, rare presentations that mimic common conditions, pediatric dosing in weight-edge patients). Prompt injection testing against patient-facing chatbots and portal interfaces. Data extraction attempts to test whether PHI can be elicited through indirect questioning. Jailbreak patterns that attempt to bypass clinical guardrails and generate unsafe medical advice.
Deliverable: a severity-tiered findings report with specific remediation recommendations, mapped to your risk management framework, suitable for governance committee review and regulatory documentation.
Every engagement follows a four-phase structure. Timelines vary by the number of AI tools deployed and the complexity of your regulatory environment. A single-tool safety assessment can complete in 4-6 weeks. A full governance architecture build for a multi-hospital system with 10+ AI tools typically runs 12-16 weeks.
Phase 1
We catalog every AI tool in clinical use, including shadow AI adopted by individual clinicians or departments outside governance. For each tool, we document the vendor, the clinical workflow it touches, the data it ingests, the decisions it influences, and the current oversight controls (or lack thereof). We review your existing governance committee structure, vendor contracts, and compliance posture against AB 3030, Colorado AI Act, and relevant state/federal requirements. Typical duration: 2-3 weeks.
Phase 2
We run safety assessments on your highest-risk AI tools. This includes hallucination testing with clinical edge cases, demographic performance stratification using your patient population data, adversarial red-teaming, and vendor claim verification. For bias monitoring, we compute baseline equalized odds and PSI metrics that will serve as the reference point for ongoing monitoring. Deliverable: a per-tool safety report with severity-tiered findings. Typical duration: 3-6 weeks depending on tool count.
Phase 3
We design and build the governance infrastructure: vendor evaluation scorecards, risk-tiered approval workflows, monitoring dashboards, incident reporting pathways, model card templates, and regulatory compliance documentation. For meaningful review interfaces (AB 3030), we design the clinical workflow that highlights AI uncertainty, surfaces patient context, and logs review actions. We align all controls to NIST AI RMF and ISO 42001 for Colorado AI Act compliance. Typical duration: 4-8 weeks.
Phase 4
We train your informatics and compliance teams to operate the monitoring infrastructure independently. We conduct tabletop exercises simulating AI safety incidents (hallucination reaching a patient, demographic performance degradation, regulatory inquiry). We establish quarterly review cadences and define the metrics, thresholds, and escalation pathways that trigger governance action. Caveat: sustained monitoring requires internal capacity. We build the system and train the team, but we are honest that external consultancies cannot replace in-house clinical informatics leadership. Typical duration: 2-4 weeks.
Answer 8 questions about your health system's current AI governance and safety infrastructure. The assessment produces a readiness score with specific, actionable next steps you can take independently, whether or not you engage Veriprajna.
Start with three non-negotiable requirements before any demo: subgroup performance data stratified by race, sex, and age for the patient population the tool will serve; an independent external validation study (not vendor-funded); and a completed model card documenting training data provenance, known failure modes, and the specific clinical contexts where the tool has not been tested.
Most vendors will provide overall accuracy numbers. Push past these. Ask for sensitivity and positive predictive value broken out by demographic group. A sepsis model with 80% sensitivity for white patients and 40% for Black patients is not an 80% accurate model. It is two different tools delivering two tiers of care.
Require the vendor to sign contractual language committing to ongoing performance disclosure, not just pre-sale benchmarks. The Pieces Technologies settlement established that marketing accuracy claims without substantiation is a deceptive trade practice. Your vendor contracts should reflect this precedent: tie accuracy representations to independently verifiable metrics, and include remediation clauses triggered by performance degradation.
For ambient documentation tools specifically, request linked-evidence capabilities where every AI-generated statement in a clinical note traces back to a specific moment in the patient encounter audio. Abridge and Nuance both offer versions of this. If your vendor cannot provide source attribution for generated text, that is a hallucination risk you cannot monitor.
The September 2024 Texas AG settlement with Pieces Technologies established that existing consumer protection law, not new AI-specific legislation, is sufficient to pursue healthcare AI vendors for deceptive accuracy claims. The five-year Assurance of Voluntary Compliance requires Pieces to disclose metric definitions, calculation methodologies, training data details, and known harmful uses to all current and future customers.
For your contracts, this creates three immediate action items. First, audit every accuracy claim in your existing vendor agreements and marketing materials. If a vendor claims a specific hallucination rate, error rate, or accuracy percentage, your contract should require disclosure of how that number was calculated, on what dataset, and whether it has been independently validated. Second, add performance transparency clauses to new contracts. Require vendors to provide subgroup performance metrics, disclose model updates that could affect accuracy, and agree to independent third-party auditing at your option. Third, review your liability allocation. Most EHR vendor contracts, including Epic's Master Software License Agreement, contain broad limitation-of-liability clauses. When Epic's built-in sepsis model misfires, the contractual liability typically stays with the health system.
The Pieces precedent suggests that deceptive accuracy marketing may override these limitations, but that theory has not been tested in court. Do not wait for litigation to clarify this. Build independent verification into your governance process now.
AB 3030 requires California health facilities to notify patients when generative AI is used to communicate patient clinical information, with specific notification standards for written, online chat, audio, and video communications. The critical nuance is the "read and reviewed" exemption: if a licensed provider reads and reviews the AI-generated communication before it reaches the patient, the disclosure requirement does not apply.
Most health systems are relying on this exemption. The problem is that relying on it requires physician review to be meaningful, and the evidence says it is not. The April 2024 Lancet study found physicians missed 66.6% of harmful errors in AI-drafted patient messages, with 35-45% of erroneous drafts sent entirely unedited. Median review time at many institutions runs 8-15 seconds per message. If your hospitalist group processes 400+ AI-drafted MyChart messages daily with 12-second median review times, the "read and reviewed" exemption is a legal fiction that will not survive regulatory scrutiny.
Our recommendation: implement both the disclosure infrastructure and meaningful review controls. Add the required disclaimers to all AI-assisted communications as a baseline. Then build a review interface that highlights AI uncertainty, surfaces relevant patient history alongside the draft, requires active confirmation of flagged clinical statements, and logs review duration and specific edits. This protects you regardless of whether the exemption holds, and it addresses the actual patient safety problem.
The $25,000-per-violation penalty for facilities is real, but the malpractice exposure from an AI-drafted message that harms a patient who was never told AI was involved is orders of magnitude larger.
Liability is layered, and the allocation depends on the specific AI tool, how it was deployed, and what the clinician did with its output. In 2025-2026, malpractice claims involving AI tools increased 14% compared to 2022, concentrated in radiology, cardiology, and oncology.
The evolving standard of care creates liability in both directions: a physician who blindly accepts a harmful AI recommendation can be found negligent, and a physician who fails to use a validated AI tool that could have caught an error may also face liability as AI-assisted care becomes the expected standard.
For the health system, three liability vectors matter. First, vendor selection liability: if you chose an AI tool without adequate due diligence on its safety profile, demographic performance, and clinical validation, that procurement decision can be challenged. Second, supervision liability: if your governance structure failed to monitor the tool's ongoing performance or respond to known safety signals, the system bears responsibility. Third, workflow integration liability: if the AI was integrated in a way that made it difficult for clinicians to override or question its recommendations (auto-populated fields, defaulted acceptances, time-pressured workflows), the system design itself becomes a contributing factor.
Malpractice insurers are responding. Some now include AI-specific exclusions. Others require physicians to complete AI safety training to maintain coverage. Your risk management program needs to document your vendor evaluation process, your ongoing monitoring, and your clinician training. The organizations that will be best positioned are those with auditable governance trails showing they identified risks, monitored performance, and acted on signals of degradation.
Bias detection requires continuous monitoring infrastructure, not one-time audits. Start with three concrete steps. First, instrument your clinical AI outputs for demographic stratification. Every prediction, alert, or recommendation your AI tools generate should be loggable with the patient's self-reported race, ethnicity, sex, and age. This does not require changing the AI model itself. It requires building an analytics layer on top of the model's output that computes sensitivity, specificity, and positive predictive value per demographic group on a rolling basis.
Second, establish alert thresholds. If your sepsis model's sensitivity for Black patients drops below 80% of its sensitivity for white patients (a rough analog of the four-fifths rule used in employment discrimination), that triggers a governance review. The specific thresholds depend on your clinical context and risk tolerance, but having no thresholds means you are flying blind.
Third, address the upstream data problem. Pulse oximeters overestimate SpO2 by 0.6-1.5 percentage points in darker-skinned patients. The FDA issued draft guidance in January 2025 recommending testing on 150+ diverse participants using the Monk Skin Tone scale, up from the prior requirement of just 10 subjects. If your AI triage system uses SpO2 as an input feature, it inherits this hardware bias. Black patients are nearly three times more likely to experience occult hypoxemia that pulse oximeters miss. Your clinical protocols should include supplementary assessments when SpO2 readings diverge from other vital signs in patients with darker skin tones.
This is not just an AI problem. It is a data integrity problem that AI amplifies. The Epic Sepsis Model's documented performance gap (AUC 0.63 on external validation vs. 0.76-0.83 claimed) illustrates what happens when site-specific overfitting meets demographic-blind evaluation.
The Colorado AI Act (SB 24-205), now effective June 30, 2026 after an extension from February, is the first comprehensive US state AI law with direct healthcare implications. It defines "high-risk" AI systems as those that are a substantial factor in consequential decisions, including provision, denial, cost, or terms of healthcare services. Healthcare deployers must implement a risk management policy, conduct annual reviews of each high-risk AI system for algorithmic discrimination, complete impact assessments, notify patients when AI makes consequential decisions, and provide appeal opportunities via human review.
A critical exemption exists for HIPAA-covered entities: if the AI provides recommendations that require a healthcare provider to take action to implement them, the system may be exempt. This means your ambient scribe that drafts a note for physician review is likely exempt, but an AI that auto-triages patients or auto-denies prior authorizations is not. The Colorado AG has sole enforcement authority, and compliance with NIST AI RMF or ISO 42001 creates a rebuttable presumption of reasonable care.
For the EU AI Act, clinical decision support is classified as high-risk under Annex III, point 5. By August 2, 2026, any CDS tool serving EU patients must comply with Articles 9-17: risk management systems, technical documentation, data governance, transparency requirements, human oversight, and post-market monitoring. Non-compliance penalties reach EUR 15 million or 3% of global annual turnover.
For both laws, the practical starting point is the same: maintain a centralized inventory of every AI tool deployed in clinical workflows, classify each by risk tier, and document your governance controls for each tier.
As of 2026, 84% of healthcare organizations have established AI governance committees, but most lack operational teeth. CIOs serve on 63% and CMIOs on only 45%, which means nearly half of these committees are making clinical AI decisions without a clinical informatics physician at the table.
The committee needs four operational capabilities, not just a charter. First, a pre-deployment approval workflow with explicit criteria: what evidence is required before an AI tool can be used in clinical settings? At minimum, this includes independent validation data, subgroup performance metrics, a completed model card, HIPAA/BAA/SOC 2 documentation, and a clinical champion who takes responsibility for the tool's safe deployment.
Second, a post-deployment monitoring protocol: who reviews AI tool performance, how often, and what triggers a pause or withdrawal? Define specific metrics (hallucination rate, alert fatigue indicators, demographic performance ratios) and review cadences (quarterly for low-risk tools, monthly for high-risk).
Third, an incident reporting pathway: when a clinician catches an AI error, where does that report go? It should feed into your existing patient safety reporting system, not a separate AI-specific silo.
Fourth, a shadow AI detection and response plan. Clinicians are adopting AI tools outside institutional governance. Your committee needs a process for discovering unauthorized AI use, evaluating its risk, and either sanctioning it within governance or removing it. The committee composition should include the CMIO (clinical safety), CISO (security and privacy), a compliance officer (regulatory), a patient safety officer (incident management), a frontline clinician champion (workflow reality), and a data scientist or informaticist (technical evaluation). Meeting monthly with a standing agenda: new tool requests, monitoring dashboard review, incident reports, regulatory updates.
The interactive whitepapers behind this solution page. Each explores a specific dimension of clinical AI safety in depth.
Forensic analysis of the Lancet patient portal study, automation bias mechanisms, RAG architecture for clinical grounding, and AB 3030 compliance implications.
Technical anatomy of deceptive accuracy claims, the Pieces Technologies settlement, Med-HALT evaluation frameworks, and the AI Safety Level tiering model for clinical workflows.
Pulse oximetry racial bias, Epic Sepsis Model failure analysis, Black maternal health disparities, fairness-aware loss functions, and demographic performance monitoring architecture.
A single AI-related adverse event costs a health system $250,000-$1M+ in investigation, remediation, and legal exposure.
With malpractice claims involving AI tools up 14% since 2022 and state AG enforcement expanding beyond Texas, the cost of independent safety verification is a fraction of the cost of an undetected failure. We start with a focused assessment of your highest-risk AI tool.