Sample report
Sample drill report: Alert drills and an AI agent audit log, checked
Faults planted by us, the audit trail counted both ways, decisions rebuilt from the trail and planted edits, on our own demo support agent with replayed traffic. Tests run by Veriprajna, which built this stack; figures describe this demo only.
https://veriprajna.com/ai-services/ai-observability-audit-trails/sample-drill-report
Results: Every drill, check and outage on our demo agent
Faults, a gap, orphan records and edits were planted by us on a local test environment with replayed traffic. Each figure states its n, unit and traffic.
- Planted faults on replayed traffic, never a production detection rate. Every fault, gap, orphan record and edit counted here was planted by us. No count or share is a rate.
- AI spot-check, not a person's review. A separate AI reviewer checked the test messages' expected end states, the route of each message and the reconstruction tool's verdicts, and re-derived every figure from the stores. It found no wrong label and no figure that differs.
- The shop, its customers and its orders are invented; no client system or data appears here.
Traces carrying version IDs, joined to the change ledger
Counted per request trace and per span over the whole trail run, with a scan of every span for the personal data seeded into the demo.
Each alert rule against the fault planted for it
Fired or not, and the time to alert, per rule and route. Rules and routes with no planted fault are listed with the reason and never counted as passed.
The intermittent fault, per-route and site-wide rules side by side
Both rules on the same runs, read beside the quiet-period counts, because a firing that also happens with no fault proves little.
False alerts over a quiet period, in requests and hours
Counted with the thresholds frozen and the quiet period held out from the baseline that set them.
Coverage in both directions, with the planted gap
Missing records and orphan records by count and route, the planted gap flagged or not, and a random sample from the gateway log looked up in the trail.
Decisions rebuilt from the trail alone, misses by field
A random sample from a stated logged period. Where no miss is found, the report gives the highest miss rate the sample size still allows.
Planted edits against the integrity check, deletions reported apart
How many planted edits were flagged, out of how many, with both limit cases run and stated.
Trace pipeline down and audit store down: What happened to requests
Latency, errors and what happened to in-scope decisions in each condition, on the test environment.
What was not run, with the reason
Anything left out is listed, so a reader never mistakes a gap for a pass.
Source: Our recorded runs of 16 November 2026 on a local test environment. Every counted run replays offline from the recorded model replies, and a test re-derives every figure; the run files stay in our private demo stack.
- System, traffic and rules
The system. Our own demo support agent for an invented home-goods shop, fake data only, behind four routes:
/,refund /,cancel /andaddress /. A separate gateway log records each request as it arrives. For every request the app writes a trace, a decision record in the audit store that holds references rather than content, the message text in a separate content store that can be deleted, and an end-state grade that feeds the quality rule. A change ledger holds every deploy with its model, prompt and configuration version IDs.order-help The traffic, API route. Model replies are recorded API replies of the same demo agent (same model, prompt and settings), recorded on 9 November 2026 by Veriprajna's LLM evaluation demo and replayed here; every request was checked byte for byte against its recording before import. No model was called through the API in this build. Traffic is the agent's test messages, each answered by one of its recorded replies (15 per message; 8 or 9 for the prompt planted by us), drawn at random and replayed at 10 requests per simulated minute on a simulated clock. Recordings repeat. Not live or production traffic.
The traffic, bridge runs. Two runs use the access route "Claude Code headless CLI via local bridge": model replies recorded on 16 November 2026 through that CLI via a local bridge on the build machine (no tools, empty working directory), not through the API. Tools were described in text and the model answered in a fixed JSON form that the app turned back into tool calls; sampling, token limit, tool choice and thinking settings cannot be set on this route, and the CLI adds a short environment section and an account reminder to every call. Replies were recorded once and replayed. These figures are kept apart from the API-route figures and are never added to them.
The stack, as run. The OpenTelemetry Python SDK, with the generative AI attribute names (
gen_), exports spans over OTLP/HTTP to a receiver written for this demo, which listens on the local machine only and writes a SQLite trace store. The audit store, content store, gateway log, change ledger and evaluation store are local SQLite stores too. Nothing was deployed.ai.* The alert rules. Each rule is checked every 60 simulated seconds. The change rules fire when a request in the last 60 seconds was served a model, prompt or configuration version other than the route's last known good deploy in the ledger. The other rules read a 15-minute window, need at least 10 requests in it (a rate rule also needs at least 3 bad events), and fire above a threshold. Thresholds come from a separate baseline run of 360 simulated minutes, as the mean plus 3 standard deviations of each rule's window value, and were fixed before the quiet period and any drill ran.
- Traces and change ledger
Trail run, 3,600 requests over 360 simulated minutes.
What was counted Result Request traces carrying the model, prompt and configuration version IDs 3,600 of 3,600 Request traces whose version IDs resolve to a change-ledger entry 3,600 of 3,600 Model-call spans with the requested model, the model that answered, and the prompt version 8,706 of 8,706 Tool-call spans with the source the tool read 8,234 of 8,234 Spans holding an email address, a seeded customer name or a seeded street address 0 of 20,540 The last row is a scan of the whole trace store for the personal data seeded into the demo shop.
- Alert drills
API route: ten scenarios, each 150 simulated minutes (1,500 requests), with its fault planted by us from minute 60. Bridge route: one model-change run, set out below the table and counted apart. Time to alert is counted in simulated seconds from the fault start.
Fault, planted by us Rule and scope Time to alert Prompt change: the 30-day refund window removed, deployed on every route Prompt change, each of the four routes 60 s on each The same prompt change Quality (end-state check), /refund 180 s Configuration change on /address Configuration change, /address 60 s HTTP 500 before the agent runs, on 25% of requests on every route Failures, per route /180 s,refund /300 s,order-help /andcancel /420 saddress The same app errors Failures, site-wide average 120 s Order lookups on /take 4 s longercancel Latency, /cancel 120 s Retry bug: every model request on /sent twicerefund Cost per request, /refund 120 s A refusal simulated in the model client on 30% of /requests (no model call made)address Refusals, /address 240 s Intermittent HTTP 502 on a share of /only, four runsrefund Failures on /, and site-widerefund Next section Bridge run: a second, larger model deployed on every route Model change, each of the four routes 60 s on each Every target fired: 22 of 22 rule and scope pairs on the API route, counting both rules in each intermittent run, and 4 of 4 on the bridge run, counted apart.
Model change, bridge run. Model change planted by us: a second, larger model deployed on every route at minute 30 of 60 simulated minutes of replayed bridge traffic (10 requests per minute, one recorded reply per message and model). The model-change rule, which compares the model requested and the model that answered with the last known good deploy in the change ledger, fired on all 4 routes 60 s after the change; no change alert fired before it (criterion fixed before the run: fire on each route within 120 s, with no change rule firing before the change; met). A change rule fires on the first request served with the new version, so at this traffic 60 s is what the design gives. The answering model always matched the requested one, so the answering-model check was not exercised.
- Quality rule: Counted as drilled on
/only, where the planted prompt moved the quality signal. The refusal fault planted by us onrefund /also moved the signal there, and the quality rule onaddress /fired 240 s after that fault started; it was not a target fixed in the drill plan, so it is listed with the other alerts and not counted.address - Configuration change: The setting changed, planted by us, was a tool timeout; the demo's tools do not use it, so this drill shows only that the change rule fires on the version mismatch in the ledger.
- Shares: The planted shares are per-request probabilities, so the share of requests actually hit differs from the planted share.
- Other alerts: After several faults, rules other than the target fired as well, most often the latency rule on
/, which also fires with no fault (see false alerts).refund
- Quality rule: Counted as drilled on
- Intermittent fault
Intermittent fault, planted by us: an HTTP 502 error returned by the app before the agent runs, on a set share of
/requests (about 40% of traffic). Requests actually hit after the fault started: 9 of 372, 13 of 371, 30 of 387 and 69 of 356refund /requests, in the order of the table. The failure rule sees this error directly. Both rules fired at every planted share.refund Share of /planted by usrefund Per-route rule on /refund Site-wide average rule 2% 4,380 s 1,380 s 5% 1,800 s 2,340 s 10% 240 s 240 s 20% 240 s 480 s With no fault, over 8 simulated hours, the site-wide rule fired 9 times and the per-route
/rule 0 times, so at small shares a site-wide alert cannot be told apart from normal noise. After the fault, therefund /failure share rose from 0 to about 3%, 4%, 7% and 20%, and the site-wide average by about 1, 1.5, 3 and 8 points, so at 2% and 5% the site-wide average stayed within the range it reaches with no fault. The normal noise comes from one test message whose expected "no customer with that email" lookup the failure rule counts as an error;refund /has no such message, which is why its threshold is zero.refund Supplementary, not fixed before the run: Of the 90 rule checks after each fault started, the per-route rule fired on 12, 35, 80 and 87, and the site-wide rule on 2, 12, 52 and 81, in the order of the table.
The smallest share planted (the first row) is the smallest at which both rules fired: a limit at this traffic, not a rate, and not the smallest share either rule can catch. No smaller share was planted. An intermittent fall in answer quality on some requests, the kind the postmortem on the service page describes, was not planted.
- Rules not drilled
Nine rules ran in this demo, and all nine were drilled on at least one route; the model-change rule on the bridge run only. The five below are listed because this demo has nothing for them to watch.
Rule What it watches Why not drilled Retrieval index change The index version differs from the last known good The demo agent has no retrieval index Provider or platform change The provider or platform differs from the last known good One provider ran, each run used one platform, and no platform change was planted Guardrail and approval rates How often guardrails block or wait for approval The demo agent has no guardrail or approval step Input drift and outcome rates Drift for a scoring model, outcomes as labels arrive The demo has no scoring model User feedback The rate of user feedback The demo has no user feedback signal Routes where no fault was planted for that rule: Configuration change on
/,refund /andcancel /; latency onorder-help /,refund /andaddress /; cost onorder-help /,cancel /andaddress /; refusals onorder-help /,refund /andcancel /; quality onorder-help /,cancel /andaddress /. None of these counts as passed.order-help - False alerts
Quiet period: 4,800 requests over 8 simulated hours, no planted fault and no deploy, thresholds frozen from the baseline run and held out from it. 32 alerts. A run of consecutive firing checks of one rule and scope counts as one alert.
Rule and scope Alerts Latency, /refund 14 Failures, site-wide average 9 Failures, /cancel 7 Cost per request, /cancel 1 Cost per request, /order-help 1 Warm-up of the ten drill scenarios, before each fault started (6,000 requests): 24 more.
- Latency alerts rest on a model, not a measurement. Simulated latency: each model call counted as 0.5 s plus 8 ms per output token (a stated model; the recordings carry no timings), plus 0.03 s per tool call and any planted delay. The latency rule's thresholds, its 14 quiet-period alerts on
/and the 120 s alert in the latency drill rest on this model and are not measurements; with no network variation in the model, the thresholds are tight. Only the wall-clock app times (trace pipeline down, audit store down) are measured.refund - Failure alerts come from one expected lookup. The site-wide and
/failure alerts are driven by the test message described under the intermittent fault, a rule-design choice rather than an agent fault.cancel - Not retuned. The thresholds stayed as fixed before the quiet period, so these counts are what the fixed method gave.
- Latency alerts rest on a model, not a measurement. Simulated latency: each model call counted as 0.5 s plus 8 ms per output token (a stated model; the recordings carry no timings), plus 0.03 s per tool call and any planted delay. The latency rule's thresholds, its 14 quiet-period alerts on
- Coverage, both directions
Trail run: 3,600 in-scope decisions (gateway requests completed with status 200) over 6 simulated hours, reconciled in both directions against the gateway log.
Route In scope Missing from the trail /refund 1,413 0 /cancel 912 91, all inside the gap planted by us /address 551 0 /order-help 724 0 - Planted gap, flagged: Yes. For one hour on
/, planted by us, the decision record was skipped when the agent made no successful write. The 91 missing are exactly the records the app skipped, and none is missing outside the gap.cancel - Orphan records: 6. Trail records with no matching gateway request; all 6 came from a synthetic probe planted by us.
- Gateway sample: 49 of 50 found. 50 gateway records drawn at random and looked up in the trail; the one not found lies inside the planted gap.
- The source counted against: The gateway log here is a separate store, written at the demo app's request entry in the same process. On a client system the count comes from an independent source, such as the gateway's own log; completeness is checked only against the sources in scope.
- Planted gap, flagged: Yes. For one hour on
- Reconstruction sample
40 decisions drawn at random from a stated logged period (simulated hours 3 to 6 of the trail run, after the planted gap), each rebuilt from the trail alone: the audit record, the content store, the trace and the ledger references. A field is a miss when it is absent, not linked, or not version-matched to the ledger and the trace.
Field group Fields checked Misses flagged by the tool Who and with what permission Principal, agent, permission scope, trace and session, delegation 0 What it saw Request content 0 What produced it Model, prompt and configuration versions; platform and route 0 What it did Tool calls, refused actions 0 Output and time Output content, time 0 - Confirmed misses: 0. The tool flagged none, so there was nothing to confirm or count. Its verdicts on all 40 decisions were re-derived field by field in an AI spot-check, not by a person.
- Highest miss rate the sample still allows: 7.2% (exact one-sided 95% bound, since no miss was found in 40).
- Content not checked: 4 fields. Request and output content erased on request, reported apart and never counted as a miss.
- What the 0 can and cannot show: The trail run had one deploy, so the version match never met a version change in the sample. No record in the trail has a refused action, and there is no delegation, one agent, one permission scope and one platform, so those fields each took a single value. The record and the trace are written by the same app code in the same request, so they agree unless that code has a fault; the check's ability to catch a mismatch shows only in integrity limit case L2.
- Integrity check
Records are hash-chained, and checkpoints of the chain go to a separate append-only anchor log in its own folder, standing in for a place the application cannot rewrite. Each edit, planted by us, was made on a fresh copy of the trail's audit store and anchor.
Edit, planted by us Flagged Flags raised E1: One field of a record changed, hash not updated Yes 1 E2: One field changed and that record's hash recomputed Yes 2 E3: One field changed and every later hash recomputed, anchor untouched Yes 36 E4: One record deleted from the middle Yes 2 E5: The last record deleted Yes 1 E6: A forged record inserted in the middle, with its own hash Yes 82 E7: Two records swapped Yes 3 API route: 7 of 7 planted edits flagged.
- E8, on the API route, not run: Removing a refused action from a record. No decision record in the trail run had a refused action to remove.
- E8 and refused actions, bridge run, counted apart: A 20-minute run of 200 requests. For this run, 4 test messages that each ask for a write were sent on a route whose permission scope is read only (the model was still offered every tool; the app denied the writes): 21 write requests, 26 denied write calls in the traces, and the same 26 recorded as refused actions in 21 decision records. A refused action removed from one record on a copy, planted by us, was flagged: 1 of 1, reported apart from the 7 of 7 planted edits on the API route. The live bridge trail raised 0 flags. E8 is the same kind of edit as E1; what it adds is that refused actions sit inside the record the integrity check covers.
- E5 needs its caveat: It was flagged because the final checkpoint anchored the last record; a deletion after the last anchor would not be flagged.
- Limit cases, not flagged, as expected: L1, a consistent rewrite of both the store and the anchor log; L2, a wrong value written at the time (a wrong prompt version, hashed as written). The check finds later changes to stored records, never a wrong value written at the time or a rewrite by someone who controls both the store and the anchor. Supplementary: The L2 record's prompt version differs from its trace, so the reconstruction version match would flag it.
- Scheduled deletions, reported apart: 1,210 content items erased on request (3 principals) and 994 expired by retention. The live trail after these deletions: 0 flags across 4,928 records. A deletion is never counted as a flag or a miss.
- Trace pipeline down
Wall clock on our machine (a local workstation, Python 3.13.11). Model replies were replayed, so model time is excluded. 300 requests per condition, served one after another, with spans exported in batches (
BatchSpanProcessor).Trace receiver Errors; decisions in the trail App time p95 (median) Traces reaching the store Up 0; 300 of 300 0.91 ms (0.56 ms) 300 Refusing connections 0; 300 of 300 0.83 ms (0.54 ms) 75 Accepting, never answering 0; 300 of 300 0.88 ms (0.61 ms) 300 With the receiver refusing connections, 225 traces were lost; every decision still reached the audit trail.
Contrast, export inside the request (
SimpleSpanProcessor): a median of 5.5 ms per request with the receiver up (30 requests), and 10.0 s with the receiver accepting and never answering (3 requests)."Requests never wait on telemetry": Holds on this demo for the batch-export design, under the criterion fixed before the run: in each down condition, 0 request errors, every decision in the trail, and p95 app time within 5 ms of the up condition's. It does not hold for export inside the request. It held over the short run above; a longer outage that fills the export queue was not tested.
- Audit store down
90 simulated minutes; the audit store was taken offline for the middle 30, then restored and the spool flushed. Each route's failure mode was set before the run.
Route and mode Requests while down What happened /, set to fail closedrefund 135 All 135 declined with HTTP 503 before the agent ran /, spoolcancel 73 73 spooled, all in the trail after recovery /, spooladdress 40 40 spooled, all in the trail after recovery /, spoolorder-help 52 52 spooled, all in the trail after recovery - Spooled records: 165 written to the trail at recovery, 0 lost. No other errors. The integrity check after the flush raised 0 flags.
- App time p95, wall clock: 0.87 ms before, 0.40 ms while the store was down, 0.84 ms after.
- On
/: With the audit store unreachable,refund /(set to fail closed) declined each request with HTTP 503 before the agent ran: 135 of 135 in 30 simulated minutes. Holding a single action until its record is written was not built or tested.refund
- Not run, and why
Item Status Reason Production check Not run Not simulated: It exists only on client systems Your own faults Not run Every planted fault was known to those who built the alerts Answering-model check Not exercised On all 600 requests of the model-change run the model that answered was the model requested Integrity edit E8, API route Not run No record had a refused action to remove; run on the bridge route instead Every figure in this report was seen by us; none was confirmed by a client or taken from a client export.
Limits, method and what this sample does not show
Where the drills are easier or narrower than a live system
- The faults were shaped so the recorded model inputs stay unchanged. The intermittent fault is an app error returned before the agent runs, the configuration change touched a setting no demo tool uses, and refusals were simulated in the model client. Each makes the drill easier or narrower than a fault in a live system.
- The audit store outage is a folder taken offline. The app's next write finds the store unreachable.
- The failure rules count one expected result as an error. The test message with an unknown email gets the correct answer "no customer with that email", which the rules count as a tool error. That choice alone produces the site-wide and
/failure alerts in the quiet period.cancel - The zero threshold on
/is a property of this message set, not a general result: norefund /message gives an error in normal traffic.refund - The fixed threshold method gives tight thresholds on near-constant signals, which is why the latency and site-wide failure rules raise the most false alerts.
- The refused-action run used a route assignment made for that run only, by the AI build agent, so that write requests met a read-only route. The 21 write requests fail the end-state check by design, because their writes were denied; they are not agent failures.
- One deploy and one agent in the trail run leave several reconstruction fields with a single value, as set out under the reconstruction sample.
Who wrote the rules and labels, and what was checked
- Labels: The quality signal's reference is the expected end state of each test message, written by Veriprajna's AI build agent on 9 November 2026 for the LLM evaluation demo and copied unchanged. The route of each message was assigned by the AI build agent before any run.
- Fixed before the runs that test them, with UTC timestamps on 16 November 2026: the messages, routes and labels at 14:17:34; the drill plan, with every fault, seed, detection rule and limit case, at 14:26:59; the thresholds from the baseline run at 14:27:27; the bridge plan, with the model-change criterion, at 15:27:27, before any counted bridge call; its second version, with the guard line and new run IDs, at 15:36:33, before any of its calls.
- No model judged anything. The quality signal is a code check of the shop's end state after each request.
- The AI spot-check was done by an AI reviewer that did not build the demo. It checked every expected end state against the shop's data and prompt, every route, and all 40 reconstruction rows field by field, re-derived every figure with its own scripts reading the stores directly, and ran the offline replay. For the bridge runs it parsed all 561 recorded exchanges and re-derived the model-change, refused-action and E8 figures from the stores. No person has checked the labels, the routes or the tool's verdicts.
Run configuration
A local workstation with Python 3.13.11, the OpenTelemetry Python SDK 1.45.2 and its OTLP/HTTP exporter 1.45.2. Nothing was deployed, and the only network listeners were the trace receiver and, for the bridge runs, the bridge, both on the local machine.
Agent. The demo support agent shared with our evaluation and guardrails sample reports: one pinned model through a commercial model API, medium effort, up to 10 model turns per request, no sampling settings sent.
Bridge runs. The same agent prompt and tools through the Claude Code headless CLI via local bridge, with an adapter that describes the tools in text and reads one JSON tool call or reply back; two models from the same provider, medium effort. Sampling settings are not exposed on this route.
| Run | Simulated length | Requests | Used for | |
|---|---|---|---|---|
| Baseline | 360 minutes | 3,600 | Thresholds only | |
| Quiet period | 480 minutes | 4,800 | False alerts, held out | |
| Ten drill scenarios | 150 minutes each | 1,500 each | Alert drills, fault from minute 60 | |
| Trail run | 360 minutes | 3,600 | Traces, coverage, reconstruction, integrity | |
| Audit store down | 90 minutes | 900 | Spool and fail-closed behaviour | |
| Trace pipeline down | Wall clock | 300 per condition | Errors, decisions, app time | |
| Bridge model change | 60 minutes | 600 | Model-change drill, change at minute 30 | |
| Bridge refused actions | 20 minutes | 200 | Refused actions and E8 |
Runs excluded, superseded and re-run
- A live recording attempt recorded no reply and is excluded from every figure.
- A 20-minute code check, run before the drill plan was fixed, is excluded from every figure.
- One pass of every recorded baseline reply through the app, with no fault, was run before the drill plan was fixed, to see the normal error and quality rates. The intermittent shares and the rules' minimum-count guards were chosen after seeing it, and no figure uses it.
- A bridge smoke test of the plain model and the adapter is excluded from every figure.
- The first bridge recording is superseded. In the first bridge recording, the CLI's account context reached the agent once: one of 280 replies looked up a customer with an email that was not in the message (no order was touched); 9 of 600 requests in one run used that reply. We fixed in advance a guard line telling the model to take customer details only from the message, re-recorded every reply (281) and re-ran both drills under new run IDs; the scan found no such reply. Figures come from the second recording; the first is kept and reported as superseded, and its drill results were the same, so no counted figure changed. The CLI still adds its context to every call, which stays a limit of this route.
- One recorded conversation was left out at import, because one of its turns was not in the source recordings.
- The trail run ran twice. The first attempt stopped on a code error in planted edit E7 before writing any result; the whole step was run again with the same plan and seeds. The run is deterministic, and the offline replay reproduces it byte for byte.
What this sample does not show
- A detection rate. No real traffic was sampled, so no figure says how often a live system would raise or miss an alert.
- Live model latency. The recordings carry no timings, so request latency is simulated; only app times are measured.
- Your stack. Every store here is a local SQLite store. In a build we work with the tracing tool you already run where it fits, or a stack you host, as confirmed in the written scope.
- The agent as deployed, on the bridge runs. The bridge call adds a tool protocol and the CLI's own context, and its settings cannot be set, so bridge replies are not comparable with the API-route replies and say nothing about the agent as deployed.
- A model swapped by the provider. The model-change drill fired on the model requested; no swap by the provider was planted or seen.
Know your alerts fire and your audit trail holds, before someone asks
This sample is our own demo agent. A build adds tested alerts and an audit trail you can check to yours, starting with a short discovery call.
- Email or WhatsApp us
- Reply within 1 business day
- Free 30-minute discovery call
- Written scope and fixed fee within 2 business days of the call



