The first time I ran our AML recertification demo on the learner I had built to game it, I almost shipped a bug I would have called a feature. On paper the learner was flawless. Near-perfect answers across all 18 compliance concepts, finished faster than anyone else in the cohort. Most systems that call themselves "adaptive" would read that as a star pupil and stamp a full certificate. Mine was about to do the same. Then I looked at the response times.
The learner who looked perfect is the one that reshaped the product
I keep coming back to that learner precisely because he is not real. The Operations Associate in the demo is a synthetic profile I wrote on purpose to model the thing that keeps compliance officers up at night: an employee who breezes through a mandatory module by letting an AI assistant auto-answer it. His interaction stream is fast-and-perfect across 18 unrelated concepts, a response-time distribution that no genuine recall produces. The corporate LMS underneath him does exactly what corporate LMSs do. It marks the course "Completed" and moves on. That green checkmark is the whole problem I set out to attack. It records that a video was watched, not that anything was learned, and in regulated work that gap is a liability an auditor can price.
The synthetic AI-gaming learner: apparent mastery near-perfect, credential withheld, 0/18 certified. The completion checkmark said "trained." The behavior said otherwise.
Why I stopped trusting a mastery score on its own
I designed the model to be smart, and then I refused to let it be the one that decides. The core of Attest is a real self-attentive knowledge-tracing transformer (SAKT, about 119,809 parameters, trained on CPU), not a quiz score and not the collaborative filtering most "adaptive" tools ship. It reads a learner's raw interaction history and infers a per-concept mastery probability. For the gaming learner it did what a pure mastery score does: it reported high numbers, 0.93 to 0.99 across the board. If I had let that number certify people, I would have certified a fraud. So the decision does not live in the model. It lives in a competence gate that is plain Python the model cannot override, and it certifies a concept only when three things hold at once: mastery at or above 0.747, at least three real interactions of evidence, and a response pattern that is not anomalous. The gaming learner cleared the first bar and failed the third. Result: 0 of 18 certified, all 18 routed to "needs proof," evidence flagged, seat-time withheld pending verification. I had already given up on asking a model to police its own evidence; agents advise, code decides, and that is the one architectural line I will not move.
The gate's own record: 18 concepts, mastery 0.93 to 0.99, every one withheld. High confidence is not the same as sufficient evidence, and the gate is where I encode the difference.
The certificate I actually want to hand a regulator
I only understood what I was building when I generated the first honest one. The Senior BSA Analyst, another synthetic profile but this one modeling someone who genuinely knows the material after 8 years in the role, came through very differently. The model found real gaps on Trade-Based Money Laundering (over and under invoicing, mastery 0.59) and one SAR red-flag concept (0.73), both below the 0.747 line. So the gate certified 16 of 18 and routed those two to "needs proof," and it cut this learner's seat time from 240 flat minutes to 47.3 adaptive minutes by skipping what was already proven, an 80.3% reduction. What comes out is not a checkmark. It is a signed Competence Certificate: per-concept mastery, the evidence trail behind each estimate, the regulatory mapping (31 CFR, FATF, FinCEN), the model version and its held-out AUC, and an HMAC-SHA256 signature. Those two amber "needs proof" rows sitting next to 16 green ones are my favorite thing in the demo, because a system willing to say "I cannot prove this one yet" is the only kind I would trust to say "certified" about the rest.
The artifact I want an auditor to receive instead of a green check: every mastery claim, its evidence count, the regulation it maps to, and two honest "needs proof" lines the system would not bluff.
A word on that AUC, because it is the one number people over-read. It is 0.8315, held out on a seeded synthetic corpus I generated with Performance Factors Analysis, a different model family from the transformer I train, which is why the score means the model recovered real learning structure rather than grading its own assumptions. It is not a public-dataset result. Published SAKT on the public ASSISTments data sits around 0.80 (Pandey and Karypis, 2019), and I cite that only as outside context. The AUC proves this is a genuine knowledge-tracing model. It is not the headline, and it measures next-item prediction, not long-term retention, which the certificate's own footer says plainly.
Why refusing to certify is the feature, not the bug
I get asked whether withholding a certificate is bad for the pitch, and I think it is the entire pitch. Over a simulated 500-person annual-recertification cohort at a $75/hour fully loaded wage, the model cuts roughly 51.9% of cohort seat time by skipping proven concepts, which works out to about $77,872 recovered on this one module and a labeled projection near $389K annualized across a typical compliance program. Those numbers are computed from the model's outputs, not borrowed from a customer I do not have. And they are not even the durable part. U.S. companies spend about $102.8B a year on corporate training, roughly $874 per learner (Training Magazine, 2025), largely to confirm that a video was watched. Meanwhile EU AI Act Article 4 role-based AI-literacy enforcement begins August 2, 2026, with penalties up to €35M or 7% of global revenue, AML non-compliance averages $9.4M an incident, and TD Bank absorbed a $3.1B AML penalty tied partly to an inadequate training program. A better base model does not touch any of that. Cornerstone shipped an "Adaptive Learning Agent" in March 2026 that still leans on collaborative filtering rather than concept-level knowledge tracing, and a smarter recommendation engine still cannot prove to an auditor which interactions backed which mastery claim. Provenance can. That is the part that holds no matter how good the models get.
The question I keep asking L&D leaders
I have started ending every conversation with the same question, so I will end this one with it too. If an auditor pointed at a single "Completed" checkmark from your last mandatory recertification and asked you to prove which specific interactions justified it, and whether the person actually knew the material or just clicked through with an assistant, could you answer? Most L&D leaders I talk to go quiet, because the honest answer is that completion was never evidence of competence, it was evidence of attendance. You can run the whole thing yourself, the gaming learner and the signed certificate included, at veriprajna.com/demos/adaptive-learning-ai. I am less interested in whether you like the demo than in this: what would your organization have to change before a checkmark could survive that one question?