A healthcare AI company told hospitals its software hallucinated less than once in every 100,000 outputs. That claim helped it land contracts at four major Texas hospital systems, where the tool drafted clinical notes and summarized patient charts. Then the Texas Attorney General investigated — and concluded the accuracy metrics were "likely inaccurate" and potentially deceptive.
The resulting settlement with Pieces Technologies in September 2024 wasn't just a marketing scandal. It was the first enforcement action against a healthcare AI vendor for misleading accuracy claims, and it rewrites the rules for every enterprise deploying generative AI in high-stakes environments.
The question isn't whether your AI makes mistakes. It's whether anyone can prove how often.
We've spent considerable time analyzing this case — the technical claims, the regulatory response, and what it reveals about how most enterprise AI is actually built. What we found has implications far beyond healthcare.
Why a 0.001% Hallucination Rate Should Have Raised Eyebrows
Large language models don't look things up. They predict the next word based on patterns learned during training. Every output is a probability calculation — the model picks the most likely next token, then the next, then the next. Sometimes that chain of predictions produces something that reads perfectly but is factually wrong. That's a hallucination.
Claiming your system hallucinates less than once per 100,000 outputs means you'd need an enormous, perfectly labeled dataset to even measure that rate. For clinical summaries — which vary by hospital, specialty, physician style, and patient history — no such gold-standard dataset exists. The claim wasn't just bold. It was, practically speaking, unverifiable.
The Texas Attorney General didn't need new AI legislation to act. Existing consumer protection law — the Texas Deceptive Trade Practices Act — was enough. The message to vendors: if you advertise a number, you'd better be able to show exactly how you calculated it.
What the Settlement Actually Requires
The resulting agreement — formally called an Assurance of Voluntary Compliance — puts Pieces Technologies under a five-year transparency mandate. The specific obligations read like a checklist every enterprise buyer should be demanding from their AI vendors right now:
Disclose how accuracy metrics are defined and calculated. No more proprietary benchmarks that customers can't interrogate.
Notify customers of known or foreseeable harmful uses. The vendor must proactively tell you where its tool might fail — not wait for you to discover it.
Document training data and model types. Customers get to know what's under the hood.
Submit to independent third-party auditing as an alternative to self-reported metrics.
This shifts the burden of proof. Before this settlement, AI vendors could market impressive numbers and leave hospitals to figure out whether they were real. Now, at least in Texas, the vendor has to substantiate its claims or face enforcement.
We explored the full regulatory and technical dimensions of this shift in our interactive analysis, which maps out how existing law is already catching up to AI marketing.
The Real Problem: Most Enterprise AI Is a Thin Shell Over Someone Else's Model
This case isn't really about one company's marketing. It exposes a structural weakness in how most enterprise AI products are built.
The dominant approach is what the industry calls a "wrapper" — a software layer that sends your data to a foundational model (like GPT-4), receives the response, and presents it in a polished interface. The wrapper company adds branding, maybe some prompt engineering, and a user-friendly dashboard. But the core intelligence? That's rented from someone else's API.
This works fine for low-stakes tasks. For drafting clinical notes that influence patient care, it's dangerously inadequate. A generic model doesn't understand your hospital's specific EHR system, your physicians' documentation styles, or the longitudinal history of a particular patient. It just predicts plausible-sounding text.
Wrapping a general-purpose AI in a healthcare interface doesn't make it a healthcare AI.
The alternative — what we'd call deep integration — involves embedding AI within an organization's actual data infrastructure. That means retrieval-augmented generation (pulling answers from your real documents, not the model's training data), fine-tuning on domain-specific information, and layered human oversight. It's harder to build. It's harder to sell. And it's the only approach that can deliver the kind of reliability high-stakes environments demand.
Adversarial AI: Using One Model to Police Another
One defense Pieces Technologies offered was its use of "adversarial AI" — essentially, a second AI model that reviews the first model's outputs and flags potential hallucinations. Their technical documentation showed this Adversarial Detection Module was 7.5 times more effective at catching clinically significant errors than random sampling.
That's a meaningful capability. But it comes with a critical caveat: speed.
The median time to correct a flagged error in the Pieces system was 3.7 hours. For a progress note that a physician will review tomorrow morning, that might be acceptable. For a real-time clinical decision — say, a discharge recommendation or a medication alert — a nearly four-hour delay could be dangerous.
This points to a principle we think every enterprise should adopt: tier your AI use cases by risk and required response time. An AI drafting an administrative email needs different oversight than an AI summarizing a cancer patient's treatment history. The safety architecture should match the stakes.
How to Actually Evaluate AI You're Buying
The gap between AI marketing and AI reality exists partly because most buyers don't have the tools to evaluate what they're purchasing. A vendor says "99.9% accurate" and the procurement team has no way to challenge that number.
Specialized evaluation frameworks are starting to fill this gap. Two are worth knowing about:
Med-HALT (Medical Domain Hallucination Test) goes beyond simple accuracy scores. It tests whether a model can resist being tricked — for instance, by presenting a wrong answer as "suggested" and seeing if the model agrees, or by asking questions where the correct response is "none of the above." These tests reveal whether a model actually reasons or just pattern-matches confidently.
FAIR-AI (Framework for Appropriate Implementation and Review of AI) takes a broader view, evaluating not just accuracy but equity, usefulness, and transparency. Its most practical contribution is the concept of an "AI Label" — a standardized disclosure, like a nutrition label, that tells the end user what the model was trained on, what it's designed to do, and where it's known to fail.
For the detailed technical methodology behind these evaluation approaches and our full analysis, see our research deep-dive.
Only 5% of Companies Are Getting Real Value from AI
Here's the uncomfortable backdrop to all of this: research indicates that just 5% of enterprises are achieving measurable business value from AI at scale. The other 95% are running pilots, accumulating costs, and struggling to show impact.
The differentiator isn't better algorithms. The companies succeeding follow what researchers describe as a 70:20:10 rule — 10% of the effort goes to the model itself, 20% to the technology stack, and 70% to organizational transformation. They redesign workflows. They invest in data quality before they invest in AI capabilities. They measure business outcomes, not technical benchmarks.
The companies winning at AI aren't building better models. They're building better organizations around their models.
This also shapes the build-versus-buy decision. Companies that purchase specialized AI tools from focused vendors succeed 67% of the time. Those that try to build equivalent tools internally? Just 33%. The implication is clear: the value of enterprise AI lives in integration and governance, not in model creation.
What Should You Do Differently Starting Monday?
The Texas settlement isn't an isolated incident. It's a signal. State attorneys general have found that existing consumer protection law gives them everything they need to go after misleading AI claims. More actions will follow — in healthcare, financial services, insurance, and anywhere AI touches consequential decisions.
If you're an enterprise leader deploying or purchasing AI, five things matter now:
Demand metric transparency from every vendor. If they can't explain how they calculated an accuracy number — including the dataset, methodology, and sample size — treat the number as marketing fiction.
Create AI disclosure labels for internal deployments. Every tool your team uses should have a one-page card explaining what it does, what data it was trained on, and where it's known to fail.
Tier your use cases by risk. Not every AI application needs the same oversight. But the high-stakes ones need adversarial validation, human review, and fast correction loops — not just a confidence score.
Invest in data quality before model sophistication. The most common reason AI underperforms isn't the model. It's the messy, siloed, inconsistent data feeding it.
Move from isolated pilots to governed platforms. Fragmented AI projects create ungovernable risk. A unified platform approach reduces per-use-case costs and enforces consistent standards.
The era of taking AI accuracy claims at face value is over. The enterprises that thrive will be the ones that learned to ask the uncomfortable question: prove it.
If your organization is navigating these decisions — especially in regulated industries — we'd welcome the conversation about what rigorous AI governance looks like in practice.