AI Accuracy Claims in Healthcare: What 0.001% Really Means

The Problem

In September 2024, the Texas Attorney General forced a healthcare AI company called Pieces Technologies into a landmark settlement. The company had told hospitals its clinical documentation software had a "critical hallucination rate" of less than 0.001% — fewer than one error per 100,000 outputs. The state called that claim both inaccurate and deceptive.

Here is the part that should keep you up at night. Four major Texas hospitals had already deployed this software. Houston Methodist, Children's Health System of Texas, Texas Health Resources, and Parkland Hospital all used it. The AI summarized patient charts, drafted clinical notes, and tracked barriers to discharge. These are not low-stakes tasks. Errors in clinical documentation can directly affect patient safety.

The Texas Attorney General did not need a new AI law to act. Existing consumer protection rules — the Texas Deceptive Trade Practices–Consumer Protection Act — were enough. The state concluded that the accuracy metrics Pieces Technologies marketed were "likely inaccurate" and potentially misleading. Your organization does not need to be in healthcare to feel the impact. If you deploy AI and make claims about its accuracy, your state attorney general already has the tools to investigate you.

This case is not just about one company. It is a warning shot for every enterprise buying or building AI systems that touch regulated decisions.

Why This Matters to Your Business

The financial and legal exposure from unverified AI claims is no longer theoretical. The Pieces Technologies settlement created a five-year compliance obligation with the state of Texas. That means five years of mandatory disclosures, ongoing monitoring, and the possibility of independent third-party audits. Consider what that kind of scrutiny would do to your operations.

The numbers paint a stark picture across the industry:

• Only 5% of companies are achieving measurable business value from AI at scale. The other 95% are spending without clear returns.
• 65% of developers report that AI "loses relevant context" during complex tasks, introducing subtle errors or inconsistencies.
• Companies that buy specialized AI tools from vendors have a 67% success rate, while those building internal tools from scratch succeed only 33% of the time.

The settlement now requires Pieces Technologies to disclose the specific data and models used to train its products. It also demands disclosure of the methodology behind any performance metrics. If your AI vendor cannot show you how they calculated their accuracy claims, you share their risk.

For your board and your compliance team, the lesson is direct. The regulatory standard is not "did the AI work most of the time." The standard is: can you prove it, and did you tell your customers the truth about how you measured it? If your organization deploys AI in any regulated function — clinical, financial, legal — this settlement is your new baseline for due diligence.

What's Actually Happening Under the Hood

To understand why a 0.001% hallucination claim is so suspicious, you need to know how these AI systems actually work.

Large Language Models — the technology behind tools like GPT-4 — are prediction engines. They generate text by guessing the most likely next word based on patterns learned during training. Think of it like autocomplete on your phone, except scaled up to write entire paragraphs. The system does not "understand" facts. It predicts what sounds right.

A hallucination happens when the AI assigns a high probability to a word or phrase that sounds correct but is factually wrong. In a clinical setting, this might mean generating a medication name that fits the sentence structure but does not match the patient's actual chart.

Measuring these errors with 0.001% precision would require an enormous, perfectly labeled dataset of clinical summaries. That dataset does not exist. Clinical documentation is too fragmented and too specific to individual patients and physicians for any universal gold standard.

This is the core problem with what the whitepaper calls the "wrapper" model. A wrapper sends your data to a general-purpose AI through an API and displays whatever comes back. It adds minimal domain-specific checks. This approach gets products to market fast, but it lacks the technical safeguards needed to catch hallucinations, prevent data leakage, or resist prompt injection attacks. A simple API call to a general-purpose model cannot account for a patient's full medical history or a physician's specific documentation style. Speed to market is not the same as safety in practice.

What Works (And What Doesn't)

First, let us clear out the approaches that give you a false sense of security:

• Relying on vendor-reported accuracy metrics alone. The Texas case proved these benchmarks can lack the rigor and independence required for enterprise-grade validation. Your procurement team needs independent proof.
• Trusting a single AI model to police itself. A general-purpose model applying its own generic safeguards is not sufficient for high-risk clinical or financial decisions.
• Scaling AI pilots without a data quality strategy. Research shows that leading companies follow a "70:20:10" rule: 70% of implementation effort goes to organizational transformation, 20% to the technology stack, and only 10% to the algorithm. Most laggards scale models on messy or siloed data.

Here is what does work — a three-step approach built around verifiable outputs:

1. Ground every AI response in your actual data. Retrieval-Augmented Generation (RAG) — a method where the AI retrieves and references your real source documents before generating an answer — keeps outputs tied to facts you control. Instead of letting the AI guess from its training data, you feed it the actual patient record, contract, or policy document. This approach enhances context retention through real-time data retrieval and long-term vector storage.

2. Add an adversarial detection layer. This means deploying a second AI system whose only job is to check the first one's work. In the Pieces Technologies architecture, their detection module was 7.5 times more effective at catching clinically significant hallucinations than random sampling. The principle matters even if the overall rate was disputed: automated checking catches errors humans miss.

3. Keep humans in the loop for high-risk decisions. AI should draft, not decide. For Pieces, summaries flagged by their detection system went to board-certified physicians for review. The median time to correct a flagged error was 3.7 hours. In an acute care setting, that delay works for progress notes but could be dangerous for real-time decision support. You need to tier your AI use cases by risk and required response speed.

The audit trail advantage ties this together for your compliance and legal teams. Frameworks like FAIR-AI recommend creating an "AI Label" for every deployed tool. This label discloses the training data, model version, and known failure modes to the end user. When a regulator or auditor asks how your AI made a specific decision, you can show them exactly which source documents it referenced, which checks it passed, and which human approved the final output. That transparency is what separates a defensible deployment from a liability.

For organizations in healthcare and life sciences, this kind of grounding and citation verification is not optional — it is the emerging standard. A GraphRAG architecture that connects your AI to structured knowledge graphs and validated source records is the technical foundation that makes this possible.

You can read the full technical analysis or explore the interactive version for the detailed engineering behind these recommendations.