A healthcare AI company told hospitals its software hallucinated less than once in every 100,000 outputs. Texas called that claim deceptive and forced a first-of-its-kind settlement. The fallout is reshaping how every enterprise should think about AI accuracy.
In September 2024, the Texas Attorney General settled with Pieces Technologies, a Dallas-based AI firm whose clinical documentation tools were deployed across four major hospital systems — Houston Methodist, Children's Health, Texas Health Resources, and Parkland. The company had marketed a "critical hallucination rate" of less than 0.001%. The state's position was blunt: that number was likely inaccurate, and the way it was presented was misleading. No new AI law was needed. Existing consumer protection statutes were enough.
When I first read the settlement documents, I didn't see a marketing scandal. I saw the future of enterprise AI accountability arriving faster than most companies are prepared for.
Why a 0.001% Hallucination Rate Should Have Raised Alarms
Think about what that number actually means. Fewer than one error in 100,000 AI-generated clinical summaries. To validate a claim like that, you'd need an enormous, perfectly annotated dataset where expert physicians reviewed every single output against the patient's actual medical record. That dataset doesn't exist — not for clinical summaries, which are messy, idiosyncratic, and vary wildly by specialty and physician.
Large language models are, at their core, prediction engines. They generate text by calculating the most probable next word given everything that came before it. They're extraordinarily good at producing language that sounds right. But sounding right and being right are different things, especially when the output is a clinical note that influences a patient's care.
Measuring AI hallucinations with the precision of 0.001% requires a gold standard that nobody in healthcare has built yet.
The Texas AG's investigation didn't just question the number — it questioned whether the company had disclosed how the number was defined and calculated. That distinction matters. A "critical hallucination" might mean something very different from a "hallucination." If you define your metric narrowly enough, you can make almost any system look flawless.
What the Settlement Actually Requires — And Why It's a Blueprint
The resulting agreement — called an Assurance of Voluntary Compliance — imposed a five-year transparency regime on Pieces Technologies. I've gone through the obligations in detail in our interactive analysis, but the key requirements are worth understanding because they're likely to become the baseline for any company selling AI into regulated industries.
The company must now disclose the definitions and calculation methods behind any accuracy claims. It must tell customers about known or reasonably foreseeable harmful uses of its tools. It must reveal the training data and models underlying its products. And if it doesn't want to disclose all of that directly, the alternative is hiring an independent third-party auditor to verify the claims.
That last point stopped me cold. We're used to financial audits. We're used to security audits. But the idea that an AI vendor might need an independent auditor to verify its accuracy claims — that's a new category of accountability, and it's coming whether the industry is ready or not.
The Texas settlement didn't create new AI law. It applied existing consumer protection rules to AI marketing — and that's what makes it so powerful. Every state has similar statutes.
The Real Problem: Wrapper Architecture in High-Stakes Settings
Here's what made this case particularly instructive for my team. Pieces Technologies wasn't building its own foundational AI model. Like most enterprise AI companies, it was building on top of existing large language models — sending patient data to a model, getting a summary back, and presenting it to clinicians. This is what the industry calls a "wrapper" approach: your product is essentially a user interface and a set of prompts layered over someone else's AI.
Wrappers can be built fast. They can demo beautifully. But they inherit every limitation of the underlying model, and they add very few safeguards of their own. When a general-purpose language model encounters a complex patient chart with decades of medical history, contradictory notes from different specialists, and abbreviations that mean different things in different departments — it doesn't flag its own confusion. It generates a confident-sounding summary.
To be fair, Pieces did implement some deeper safeguards. They built what they called an Adversarial Detection Module — essentially a second AI system that reviewed the first AI's output and flagged potential errors. Their technical documentation showed this module was 7.5 times more effective at catching clinically significant hallucinations than random sampling. That's genuinely meaningful engineering.
But the median time to correct a flagged error was 3.7 hours. For a progress note that a doctor reviews the next morning, that might be acceptable. For a real-time clinical decision — a medication dosage, a discharge recommendation — 3.7 hours could be catastrophic.
This is the core tension I keep coming back to: the same AI system can be safe enough for one use case and dangerous for another, and most organizations aren't making that distinction rigorously.
What "Deep AI" Actually Looks Like in Practice
After studying this case, I started asking a different question: what would it take to build AI systems for high-stakes environments where accuracy claims are genuinely defensible?
The answer isn't a single technique. It's a set of architectural choices that compound. Retrieval-Augmented Generation — where the AI pulls from verified source documents before generating a response — is one layer. Fine-tuning models on domain-specific data is another. Multi-stage validation pipelines where different systems check each other's work add more. And human-in-the-loop oversight, where trained professionals review high-risk outputs before they reach the end user, is the final and most important layer.
The 5% of companies achieving measurable AI value at scale aren't spending their energy on algorithms. They're spending 70% of their effort on organizational transformation — redesigning workflows, roles, and governance.
That statistic comes from enterprise AI ROI research, and it matches everything I've seen. The companies that treat AI as a technology problem get wrapper-quality results. The companies that treat it as an operational redesign problem get something durable.
One framework I've found particularly useful is what researchers call AI Safety Levels — a tiered system that matches the degree of human oversight to the stakes of the task. Administrative scheduling? Light-touch audits. Clinical documentation? Mandatory physician review. Autonomous patient-facing interactions? Strict guardrails, escalation protocols, and explainable outputs at every step. The full taxonomy and its implications are covered in our technical deep-dive.
How to Evaluate AI Accuracy Claims Before You Buy
I've started giving enterprise teams a simple checklist when they're evaluating any AI vendor's accuracy claims. It's not exhaustive, but it catches the most common problems:
Ask how the metric is defined. "Accuracy" and "hallucination rate" can mean wildly different things depending on who's measuring and what they're measuring against. If the vendor can't give you a precise, written definition, that's a red flag.
Ask what the benchmark dataset looks like. How large is it? Who annotated it? Is it representative of your specific use case, or is it a generic test set?
Ask whether the evaluation was internal or independent. Vendors grading their own homework is not validation. It's marketing.
Ask about failure modes. Every AI system fails. The question is whether the vendor knows how their system fails, and whether they've told you.
The FAIR-AI framework — developed for healthcare but applicable broadly — recommends that every deployed AI tool come with something like a nutrition label: what data trained it, what it's been validated against, where it's known to struggle. I think every enterprise buyer should demand this.
What About Industries Beyond Healthcare?
The Pieces case happened in healthcare, but the principle applies everywhere AI outputs influence consequential decisions. Financial services firms using AI to generate compliance reports face the same challenge: how do you prove the AI didn't fabricate a regulatory citation? Legal teams using AI for contract review face it too. So does any manufacturer using AI-generated inspection reports.
The Texas AG didn't invoke a healthcare-specific law. The Deceptive Trade Practices Act applies to any commercial claim that's materially misleading. If your AI vendor tells you their system is "99% accurate" and you can't verify that independently, you're exposed — not just to bad outcomes, but to regulatory action.
I've had a few conversations with enterprise leaders who assumed this was a healthcare-specific story. It's not. It's a marketing accountability story, and it applies to every vendor selling AI into any high-stakes workflow.
What I Think Happens Next
The settlement creates a template. Other state attorneys general are watching. The FTC has already signaled interest in AI accuracy claims. And enterprise procurement teams are starting to ask harder questions — not because they've read the settlement, but because they've been burned by AI tools that underdelivered.
My prediction: within two years, independent third-party AI audits will be a standard procurement requirement for any AI system deployed in healthcare, financial services, or legal. The companies that build for that standard now will have an enormous advantage. The ones still relying on self-reported accuracy metrics will find themselves in increasingly uncomfortable conversations.
What I keep telling my team is this: the era of AI marketing is ending. The era of AI accountability is beginning. And the companies that thrive will be the ones who treated accuracy as an engineering discipline, not a slide in a pitch deck.
If you're navigating this transition — trying to figure out how to evaluate AI vendors, how to structure oversight, or how to build systems that can withstand regulatory scrutiny — I'd genuinely like to hear what's working and what isn't. The playbook is still being written.