Sample report: how often a policy assistant's answers go beyond its sources

Questions are quoted from Stack Overflow under CC BY-SA 3.0 or 4.0, by post date, each linked to its post. Usernames are removed.

The frame, named before any answer was generated. Stack Overflow questions tagged [node.js], asked from 2015-09-08 to 2026-10-31, whose title mentions LTS or end of life, or whose title or body mentions LTS, end of life, EOL, long term support, release schedule, maintenance or active LTS, odd-numbered or security release: 1,056 questions. This is a stratum of real public questions, not live traffic to any assistant.

The screen. One stated rule, applied to every question in the frame before sampling: can the core of the question be answered from Node.js's own release, support, security or licence documentation? Install, configure and debug questions, third-party package vulnerabilities and other products' lifecycles are out. 20 questions passed.

The sample. A seeded random draw of up to 30 from the eligible questions returned all 20, so this is every eligible question in the frame, not a subsample.

Disclosed change. A first, narrower frame (from May 2018, without the bare "LTS" term) gave too few eligible questions. It was widened to the frame above before the assistant answered any question.

Sources. 10 current Node.js documents, snapshotted on 2026-11-04: the Release Working Group README, its release schedule data and codename list; the nodejs.org Releases, End-Of-Life and Security Reporting pages; two nodejs.org blog announcements on the release schedule and the Node.js 16 end-of-life; and the Node.js security policy and a licence excerpt. Each is pinned to a commit (nodejs/Release 6065a83, nodejs/nodejs.org a2db8a5, nodejs/node 3b209b2).

Retrieval. The documents are split at headings into 126 passages, and keyword search (BM25) gives the top 5 to the model with each question.

Model. A current general-purpose model (claude-opus-5-5) with its default settings, instructed to answer from the passages, cite them, and say so when they do not contain the answer. The same model, with separate pinned prompts, splits answers into claims and proposes verdicts.

Planted document (planted arm only). The Release Working Group README as of 3 May 2022. It gives Node.js 16 an end-of-life of 30 April 2024, since moved to 11 September 2023, and the older release-plan wording. It has the same title as the current README, so neither the assistant nor the verdict step can tell it is planted. Nothing else was planted.

The harness split each answer into claims and proposed a verdict for each one, searching the whole snapshot before calling a claim not found. In the real arm, a separate AI review then decided every provisional failure, every claim where the sources disagree, every decline and every answer's verdict, reading each against the full source snapshot rather than only the passages the assistant retrieved.

No person confirmed these verdicts. In a client engagement, a person decides every verdict.

A statement about what the assistant found in its passages, a restatement of the user's own question, and advice that states no checkable fact are not claims. Where Node.js's own documents disagree with each other, the claim is left uncounted rather than called a failure.

This sample shows the report and the method on a small, fixed set. It does not include:

• -Omissions: what an answer should have said but did not.
• -Questions written to probe each failure type.
• -A second labelling pass after a gap, to show how stable the labels are.
• -A fix, and a before-and-after test on the same questions.
• -Agent actions or stated system state: this assistant only answers questions.