We test your assistant's answers to real questions for hallucinations. You get one rate with its range, and each failure shown against your sources.
First reply within 1 business day
Embedded YouTube player: the AI hallucination audit, what it measures and what you get, 1 minute 30 seconds.
An AI hallucination audit verifies your own assistant's or agent's output against the sources it is meant to answer from, and measures how often it says something those sources do not support.
The same words are used for AI-assisted professional work, such as reports, filings and client deliverables with citations, and for AI-assisted content that agencies and publishers release. We check both, each scoped separately, against the sources you provide.
We split each sampled answer into its claims and check each claim against your sources. Automation proposes a verdict; a person decides every one. Answers to real questions are scored apart from edge cases we write to probe each failure type.
Before a person marks a claim unsupported, they search your full source set, not only the passages the assistant retrieved.
Your assistant's hallucination rate comes from a random sample of your own conversations, one answer scored per conversation. In this audit, the rate is the share of sampled answers with at least one claim your sources do not support, given with its range.
A published hallucination rate belongs to its own benchmark. Vectara's public leaderboard, for example, measures one task, summarising documents, scored by its HHEM-2.3 model. Its own FAQ agrees that it "does not definitively measure all the ways a model can hallucinate".
Rates by model answer a different question: which model does best on someone else's task. NIST's Generative AI Profile notes that testing restricted to benchmark datasets "may not extrapolate well to" real-world conditions. Rates are also quoted per claim rather than per answer; your report states its unit.
In Moffatt v. Air Canada, a website chatbot's answer about bereavement fares contradicted the airline's own policy page, and a customer relied on it.
The tribunal held that the airline is responsible for all the information on its website, whether it comes from a static page or a chatbot, and ordered it to pay damages.
Source: Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14 Feb 2024
Scores from a model or rules are estimates. Until they are checked against answers a person verified on your own data, you cannot tell how far to trust them.
The chatbot's answer, from the screenshot quoted in the decision: "If you need to travel immediately or have already travelled and would like to submit your ticket for a reduced bereavement rate, kindly do so within 90 days of the date your ticket was issued by completing our Ticket Refund Application form." The linked page said the policy does not apply to requests made after travel is completed.
The words "bereavement fares" in the answer linked to that page, and the tribunal inferred that a screenshot the airline supplied showed part of it.
The airline argued it could not be held liable for information its chatbot provided; the tribunal called this "a remarkable submission". It also said the airline did not explain "why customers should have to double-check information found in one part of its website on another part of its website", found the customer's reliance reasonable, and found negligent misrepresentation made out. It found the airline had not taken reasonable care to ensure its chatbot was accurate.
The decision does not state the chatbot's technology: the airline did not provide any information about its nature.
The chatbot's answer
"If you need to travel immediately or have already travelled and would like to submit your ticket for a reduced bereavement rate, kindly do so within 90 days of the date your ticket was issued by completing our Ticket Refund Application form."
From the screenshot quoted in the decision
The airline's own policy page
The linked page said the policy does not apply to requests made after travel is completed.
Two lines side by side. The chatbot: submit your ticket for a reduced bereavement rate within 90 days of the date your ticket was issued. The linked page: the policy does not apply to requests made after travel is completed.
Your report lays out each failure like this.
Every failure in the sample is written up in this layout, so your engineers can check each verdict against the passage for themselves.
Questions we write to probe each failure type, reported as failures found among the questions written for that type, false commitments to a customer first. They are never part of your rate.
The same frozen questions, re-run on your fixed version, with the change question by question and its range. A new measured rate needs a new random sample, quoted separately.
The frozen test set and each failure case, with the expected source passage.
The sampled question, with the conversation up to that turn.
Your assistant's answer, exactly as logged.
Claims against your sources
An answer split into claims, each checked against a passage in your sources; one claim is marked contradicted.
Each claim with its verdict: supported, with its passage; contradicted, with the passage it contradicts; or not found in your sources after a full search.
For example, a false commitment to a customer about a price, refund, legal or safety point.
Retrieval, generation, source content, or scope and decline rules; or not determinable from the export.
What usually fixes this type at this stage, and the frozen question that will show whether it did.
If you are choosing a company for an AI hallucination audit, here is what we commit to and how your data stays protected.
How client data moves in an audit: you send a redacted or pseudonymised export under our DPA; no client data goes to a third-party model or grading service unless agreed in writing; no production endpoints are used; the data is deleted on the schedule in your scope.
Five steps. The one fixed point: we capture the before run before any fix ships, so before and after compare like with like.
We agree what we audit, the sampling frame and what you provide; the written scope sets the sample size and dates.
What we can audit
What you provide
The rate needs only a redacted log export, plus a record extract for account-specific answers, or those are reported as a coverage gap. Probable stage needs the passages your assistant retrieved for each sampled turn, as logged at the time, or, where the index is unchanged since the turn, from context replay, labelled as replayed; without them it is reported as not determinable from the export.
A test setup is needed only for the before-and-after test, in place before fixes ship; without one, that test is left out of the scope.
The test setup is a non-production endpoint, a scoped key with a spend cap that is revoked at handover, or a script you run yourself, with context replay. We never use production endpoints. For agents: a sandbox or dry run, with its state record. From your side: a few hours of one reviewer's time, plus engineering time for the export, the record extract, the test setup and replay, each estimated in the written scope.
We decide each sampled answer; your reviewer blind-checks a subset.
The duration comes from one thing: a person decides every sampled answer, and searches your full source set before confirming any unsupported verdict. An optional interim read-out of confirmed failures, without a rate, is given after the before run is captured; fixes wait for that capture.
Your report arrives, then one readout call of about 60 minutes to walk through the failures and the fix order.
You ship your fixes.
The same questions re-run on the fixed version.
Two public frameworks name this risk and suggest testing for it.
| ID | Name | What it says | How the audit relates |
|---|---|---|---|
2.2 | Confabulation | Confidently stated but erroneous or false content, known colloquially as hallucinations | What counts as a failure, in the deep dive |
MS-2.5-001 | Avoid extrapolating from anecdotal assessments | Suggested action: avoid extrapolating performance from narrow, non-systematic and anecdotal assessments | A random sample of your own conversations, with its interval |
MS-2.5-003 | Review and verify sources and citations | Suggested action: review and verify sources and citations in outputs, before deployment and in monitoring | Each sampled claim checked against your sources; the cited-documents unit |
A.1.4 | Pre-deployment testing | Benchmark testing may not extrapolate well to real-world conditions | Your rate measured on your own questions, not a benchmark |
No match in NIST AI 600-1, Generative AI Profile.
| ID | Name | What it says | How the audit relates |
|---|---|---|---|
Example 1 | Unsupported or false decision support | Incorrect or unsupported information influences business, legal, healthcare, financial or operational decisions | The rate and each failure on sampled answers |
Example 2 | Incorrect state inference in workflows | An LLM infers that a condition has been met when it has not | Agent unit: stated system state checked against the sandbox state record |
Example 4 | Misleading summaries and critical omissions | Summaries omit key constraints, exceptions, timestamps or risks | Omissions counted separately from the rate |
Mitigation 1 | Ground claims before action | Require outputs to be grounded in authoritative and current sources | Fix guidance by stage |
Mitigation 4 | Use verification signals, not just confidence | Incorporate groundedness and consistency checks | Provisional verdicts from automation, each decided by a person |
Mitigation 8 | Monitor and test for misinformation | Log claims, evidence and outcomes, and test adversarial scenarios | The edge-case set by failure type, and the before-and-after test |
No match in OWASP Top 10 for LLM Applications 2026.
7 questions
Keep the detector or checker: it proposes verdicts at scale. Published work notes detectors' performance varies across models and datasets, and that it is hard to tell which to rely on. The audit gives you a random sample of answers a person verified against your own sources, handed back so you can check your detector against it. Labelling a sample yourselves works too; the audit adds the sampling design, the interval, a separate edge-case set and a comparable before-and-after test.
Yes. You send a redacted or pseudonymised export under our DPA; redaction removes identities, not the facts being checked. No client data goes to a third-party model or grading service unless agreed in writing. Your data is deleted 30 days after the before-and-after test, or 30 days after its window closes if it is not used, or on request.
Through a free 30-minute discovery call: what we audit, where the sample comes from, how large it should be, and whether a test setup exists for the before-and-after test. You get a written scope and a fixed fee within 2 business days of the call. The fee is agreed in writing before any work.
The earliest start date is in the written scope. The fixed fee does not change if the schedule slips on our side, and you can halt or end the engagement at any time.
The report still states the coverage: what was sampled, how each answer was checked, and what was out of scope, so a low rate shows exactly what it covers. A failure type that never appears in the sample gets an upper bound rather than a zero, and the edge-case set still shows how each type fails when probed.
Your results and test set are yours; what you tell your customers about your own testing is your decision. The report is prepared for you only; no third party may rely on it, and we issue no letters, and we do not name ourselves to your customers without written consent. Only the real-traffic estimate may be described as a measured rate; edge-case results are failures found on a test set. Attestations are a separate service: Independent AI Assurance & Third-Party Testing.
Fix guidance for each failure is always in the report. Hands-on help with the fixes is an optional follow-on, quoted separately. If Veriprajna authored any remediation, the report says so, and fixes we wrote are labelled in the before-and-after results.
Sampling. We draw conversations at random from a named log frame, for example all support conversations in a stated date range, and score one randomly chosen answer per conversation. Answers in one conversation can be correlated, so if more turns are scored, the interval is clustered by conversation. The rule: the interval is computed at the level that was sampled independently. Breakdowns by intent can be agreed in the scope; rare failure types are covered by the edge-case set.
Verdicts. Automation splits each answer into claims, retrieves candidate passages and proposes a provisional verdict for each claim. A person confirms or overturns every provisional verdict, and before confirming "unsupported" searches the full source set, not only the retrieved passages.
Rate. Answers with at least one claim unsupported by or contradicting your sources, over sampled answers, with a two-sided 95% interval. Omissions, and answers the assistant declined although your sources held the answer, are counted separately.
Type and stage. Failures are grouped by type, false commitments to a customer first. A probable stage, retrieval or generation, is assigned only where the passages the assistant actually retrieved for that turn are available: logged at the time, or replayed where the index is unchanged since the turn, and labelled as replayed.
Otherwise the stage is "not determinable from the export", and the fix order uses failure type alone. Problems in the source content itself, or in the scope and decline rules, need no retrieved passages.
Account-specific answers. Answers about one customer's account are checked against a record extract for the sampled conversations; without one, they are left out of the rate and counted as a coverage gap. Redaction removes identities, not the facts being checked.
Edge cases. Questions written to probe each failure type, reported as failures found among the questions written for that type. Inside the random sample, a type that never appears gets an upper bound, not a zero: for 0 failures in n answers, the one-sided upper bound at the same confidence is about 3 in n once n is 30 or more, and we compute the exact limit.
Before-and-after test. Two runs on the frozen question set, both generated in test. The before run is captured during sampling and verification, before any fix ships; the after run uses the fixed version.
Both use the same question text with the conversation so far, the same settings and the same pinned instrument, with three answers per question in each run unless the scope sets otherwise. We report the change question by question, with its interval. The logged production rate is never paired with a test run.
Pinned instrument. How claims are split changes the score, so the splitter, the models and their versions, the prompts and the labelling rules are recorded and reused; any change is stated, and the before run is re-scored.
Who labels. We decide every sampled answer. Your reviewer blind-labels a subset and may dispute; where labels differ, the settled label is applied and the report shows both rates. Labels judge support by your own source text, not professional correctness.
Method sources: Miller, "Adding Error Bars to Evals" for clustered errors, paired differences and repeated answers; Wanner and others, "A Closer Look at Claim Decomposition" on how claim splitting moves scores; Hanley and Lippman-Hand, "Interpreting zero numerators" on upper bounds when nothing fails.
NIST's Generative AI Profile calls it confabulation: "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications')". This audit uses a narrower test that can be checked: does your own source text support what the answer says? An answer can be true in general and still count as unsupported here, because your assistant's job is to answer from your sources.
The OWASP Top 10 for LLM Applications 2026 files this risk under LLM07:2026 Misinformation. One of its example scenarios is a customer-service agent that misreads a policy and approves a refund that violates the terms.
For an AI agent, a false answer is often about state: the agent reports something as done or true, such as "the refund was issued" or "the account is verified", that the system's own record does not show. We check those reports against the record, in a sandbox or dry run only.
The stage tells you where to look first.
Keeping the frozen questions as a check on every release, and calibrating automated judges, is LLM & Agent Evaluation.
Whether you run an assistant or agent, draft documents with AI, or publish AI-assisted content as an agency or publisher, start with a conversation about your sources.
