My knowledge-tracing AI wanted to certify a learner who gamed the course. The code I wrote refused.

Two employees, two identical green checkmarks, and one of them is a lie

The first time I put the two records side by side, the checkmarks looked identical, and that was the whole problem. A senior BSA analyst with eight years in the role and a branch manager six months post-transfer had both just "completed" the same four-hour anti-money-laundering recertification. Their LMS reported the same thing for each of them: a green check, "SCORM Completed," done. One of them could run a suspicious-activity investigation in her sleep. The other had real gaps in exactly the concepts a regulator cares about. The system that was supposed to certify them for compliance work could not tell them apart, because it was never measuring what they knew. It was measuring whether the video finished playing.

That gap is not academic, and building this demo forced me to keep the stakes in front of me. U.S. companies spend about $102.8 billion a year on corporate training, roughly $874 per learner (Training Magazine, 2025), and a full quarter of L&D leaders say they cannot measure whether any of it worked. In regulated work the gap has a price on the public record. TD Bank absorbed a $3.1 billion AML penalty tied in part to an inadequate training program. Starting August 2, 2026, the EU AI Act's Article 4 begins enforcing role-based AI-literacy, with penalties up to €35 million or 7% of global revenue. In every one of those cases the "Completed" record is worthless as evidence, because completion was never competence.

The system was not measuring whether they learned. It was measuring whether the video finished playing.

I built a demo called Attest to see whether I could close that gap honestly. You can run it at https://veriprajna.com/demos/adaptive-learning-ai. This essay is about the part of building it that genuinely surprised me, and it was not the model.

What I actually set out to build was a model that knew what each person knew

I started where every "adaptive learning" pitch starts, and then I threw most of it away. Most products that call themselves adaptive are running collaborative filtering underneath: "people like you took Course X next." That is a recommendation engine wearing a diploma gown. It knows what similar learners did. It has no idea what you, specifically, understand. Cornerstone shipped an "Adaptive Learning Agent" in March 2026 that still works this way. I did not want a better recommender. I wanted a model that reads one person's raw interaction stream and infers, concept by concept, what they have actually mastered.

That is a real and old research problem called knowledge tracing, and the honest version of it is a transformer, not a quiz average. I trained a self-attentive knowledge-tracing model, SAKT, about 119,809 parameters, small enough that it trains on a CPU in ninety seconds and caches after. It replays each learner's sequence of interactions and outputs a mastery probability for each of the eighteen concepts in the course. On our seeded synthetic knowledge-tracing benchmark it reached a held-out AUC of 0.8315, with next-step accuracy of 0.771. The benchmark data is generated by a completely different model family (Performance Factors Analysis, a classical logistic model), so the transformer is not grading its own assumptions. For external context, published SAKT results on the public ASSISTments dataset land around 0.80 (Pandey and Karypis, 2019). Mine was in the same neighborhood.

I was proud of that number. I thought the number was the product. It took exactly one learner to show me it was not.

Then I fed it someone who had gamed the entire course

The learner who broke my confidence was the third one in the cohort, an operations associate who had used an AI assistant to auto-answer the whole module. Watch what my model did with him. Concept after concept, the SAKT inference came back high: 0.927 on SAR red-flag identification, 0.95 on trade-based money laundering, 0.992 on correspondent banking risk, 0.994 on enhanced-due-diligence triggers. Near-perfect everywhere. If I had trusted the model, I would have certified all eighteen concepts and handed this person a clean, signed compliance record.

The trouble is the mastery was fake. The answers were fast and perfect across eighteen unrelated concepts, a response-time distribution that is inconsistent with genuine recall. A real expert is quick on some things and slow on others. This pattern was flat and impossibly fast everywhere, the signature of an answer key, not a memory. My careful 0.83-AUC model looked at an answer key and saw a star pupil.

My model did not merely miss the gaming. It was most confident exactly where it was most wrong.

That was the moment this essay became true for me. A better model would not have rescued me here. A more accurate transformer trained on cleaner data would have looked at the same fast-perfect answers and been even more certain this person had mastered everything. The failure was not a shortage of accuracy. It was that I had let a probabilistic model make a decision that a model should never be allowed to make on its own.

Why not just train the model to catch it?

My first instinct was the wrong one, and I want to be honest about the day I lost to it. I tried to make the model itself robust to gaming. Add response-time features, train it to distrust implausibly fast sequences, teach the transformer to be suspicious. It is a seductive plan because it keeps everything inside one elegant system, and I like elegant systems. It also does not work, for a reason that took me longer to accept than it should have. Anything you teach a model to detect, you have also taught it to be confidently wrong about in the cases you did not anticipate. A model's whole job is to generalize and smooth. A governance decision needs the opposite: it has to be brittle in exactly the right place, and refuse.

So I stopped, and I moved the decision out of the model entirely. The model advises. Plain deterministic code decides. After the SAKT inference runs, a piece of pure-Python logic I call the competence gate makes the certify-or-not call, and the model cannot override it. The gate certifies a concept only if three things are all true at once: mastery is at or above 0.747, there are at least three real interactions of evidence behind it, and the response pattern is not anomalous. Fail any one of the three and the concept is not certified. It is marked "needs proof," and the learner gets a verification challenge instead of a checkmark.

When the gate looked at the operations associate, it did the thing my model could not. It flagged the evidence as anomalous, refused every concept, and certified zero of eighteen. Seat time withheld, pending verification.

I want to be careful about how I frame that, because the brief I wrote for myself is careful about it. This is not a cheater-detector with a hit rate, and I never present it as one. It is a governance outcome on one illustrative learner: concepts withheld because the evidence was not good enough to certify. The point is not "gotcha, you cheated." The point is that Attest certifies only what it can prove, and stays honest about the rest.

Agents advise, code decides, and that line turned out to be load-bearing

I keep returning to that division of labor because, by the time I finished, it was the whole architecture rather than a slogan. There are exactly two places in Attest where something intelligent and probabilistic runs, and both of them only advise. An LLM agent (built on Pydantic AI, defaulting to claude-opus-4-8) does the concept tagging, decomposing one flat "AML Training" course into an eighteen-concept taxonomy: CDD, SAR narrative, structured-transaction detection, OFAC screening, and the rest. That tagging step is the thing the field will tell you is the single biggest reason adaptive-learning projects die, and it is genuinely hard, but it is advisory. The SAKT model infers mastery, also advisory. Every decision that actually matters, what path a learner walks and what gets certified, is made by deterministic code no model can talk its way past.

The sequencer is the other half of that code, and it is where the money hides. Per concept: if mastery is at or above 0.67 the learner skips or just verifies; between 0.38 and 0.67 they stay in the flow zone; below 0.38 they get scaffolding. Those cut-points are not numbers I picked by feel. They are cross-validated against the labeled cohort, and the repo ships a five-fold A/B test showing they beat the hand-set fallback while leaving the decision meaning unchanged. Run that logic over the senior analyst and she skips nearly everything she already knows: sixteen of eighteen concepts certified, seat time falling from 240 minutes to 47.3, an 80.3% reduction. Run the identical code over the branch manager who has real gaps, and it refuses to hand him the same discount: ten of eighteen certified, only 38.9% saved, because he genuinely needs the training.

The same code hands the expert a 47-minute course and the novice a real one. Neither of them can argue with it.

That asymmetry is the entire point. A system that saves everyone the same time is just a shorter course with better branding. A system that saves time in strict proportion to what each person can prove they know is doing knowledge tracing, and it is doing it in code I can read line by line.

What you hand a regulator instead of a checkmark

The artifact I am proudest of is the one a green checkmark can never produce. When the gate finishes, Attest exports a signed Competence Certificate, and I keep two real samples on disk. For the senior analyst it lists all eighteen concepts, each with its mastery probability, the count of interactions behind that estimate, the certify-or-needs-proof status, and the specific regulation it maps to (31 CFR 1020.220, FATF Rec. 12, 31 USC 5318, and the rest). It records the model name, its version, and the benchmark AUC. It carries an honest limitation footer noting that the AUC measures next-item prediction rather than long-term retention, because a mastery probability is evidence, not a guarantee. And the whole document is signed with an HMAC-SHA256 signature so it cannot be quietly edited after the fact.

This is the durable answer to the auditor's real question, which is never "did they finish?" It is "prove which interactions backed which mastery claim." A checkmark cannot answer that. Neither can a better base model, however good it gets, because accuracy is not provenance. A more accurate model still cannot show an auditor the evidence trail behind a decision it made. That trail has to be built as an artifact, on purpose, outside the model. That is what the certificate is, and it is the thing that made me stop thinking of the model as the product.

Accuracy is not provenance. A better model still cannot prove which interactions backed which claim.

You can inspect all of this in the running demo at https://veriprajna.com/demos/adaptive-learning-ai, including the two concepts the certificate refuses to certify for the analyst, where her mastery honestly sits below the 0.747 line.

The part that survives a better model

I built this expecting the model to be the hard part, and being wrong about that changed how I think about the whole category. The model was the tractable part: ninety seconds on a CPU and a respectable AUC. The hard part, the part that actually makes the output trustworthy, was everything I put around the model to keep it from making decisions it had no business making: the deterministic gate, the evidence-sufficiency rule, the anomaly refusal, the signed trail.

That is also the part that does not age out. When I show the cohort numbers (about 51.9% seat-time reduction across a simulated 500-person recertification cohort, roughly $77,872 recovered on this one module, an annualized figure near $389,362 that the app labels a projection on screen), the headline is deliberately not the AUC. The AUC only proves it is a real knowledge-tracing model and not a gradebook. The headline is the percentage of seat time saved and the percentage of concepts auto-certified versus routed to proof, and both of those hold no matter how good the underlying model becomes. A smarter transformer next year does not change the fact that the decision, and the receipt, live in code you can audit.

And if you would rather see it than read me describe it, here is the whole thing running end to end.

So here is the question I have been sitting with, and the one I would put to anyone building in this space. If your system's most important output is a decision a regulator will lean on, do you want that decision made by the model, or made by something the model is only allowed to advise? I had it backwards for a day. Then a fake star pupil, fast and perfect on eighteen things he did not know, walked in and set me straight.