The afternoon I built a toggle to make my own tool look worse
I spent an afternoon building a feature whose only job was to make CodeGraph look worse, and it turned out to be the most honest thing in the demo. It sits in the top corner of CodeGraph, the COBOL modernization demo I built, labeled "Naive AI context view." Flip it on and the dependency graph collapses to the single source file a text-based tool can actually see, that file plus a few lines around it. I ran it on the wire-transfer module the demo lands on, a variable called TRN-LIMIT, and watched six of the nine facts about that one change go grey and fall out of the panel. The score dropped from graph retrieval 9 of 9 to a naive single-file 3 of 9, and a red banner spelled out the consequence in the app's own words: handed only the three visible facts, a model emits long TRN_LIMIT and corrupts the database. I built the thing that hides the evidence because I wanted to see exactly which evidence disappears. These numbers are measured on the shipped synthetic wire-transfer fixture I authored, not a claim about any real bank.
The naive single-file view on the TRN-LIMIT wire-transfer module. Six facts go grey, retrieval falls from 9 of 9 to 3 of 9, and the banner names the outcome: a model emits long TRN_LIMIT and corrupts the DB.
Why I stopped believing translation was the hard part
I used to assume the risky part of modernization was the translation step, and building this is what talked me out of it. Roughly 70 to 80 percent of mainframe modernization projects fail to meet their objectives (industry meta-analysis, 2025), and the easy story is that turning COBOL into Java is just hard. The six facts that vanish are why I no longer buy that story. They are not trivia. They are the ones that decide whether the port is even correct. TRN-LIMIT is declared PIC S9(9)V99 COMP-3 in a copybook three files away (CBACCT.cpy:11), which means it is packed decimal and has to become a BigDecimal, not a long. The line right under it, TRN-LIMIT-ALPHA REDEFINES TRN-LIMIT (CBACCT.cpy:12), overlays the same six bytes as text. A flag called LIMIT-TYPE-FLAG (CBACCT.cpy:13) decides which of those two interpretations is live at runtime, and that flag is not set in the wire program at all. LIMITSET writes it, BATCHUPD writes it again in the nightly run, and a JCL job named NIGHTLY at 02:00 runs before WIREJOB, so the flag is set before the wire ever executes. At the call site the COMPUTE on TRN-LIMIT looks trivial. The fact that breaks it is a packed-decimal type chosen by a flag set in a different program and sequenced by a 2 a.m. batch job, and none of it is visible in the one file a context window gets.
The hidden COMP-3 type. Each fact carries its file:line source: the packed-decimal declaration, the REDEFINES overlay, and the controlling flag all live in a copybook the call site never shows.
The number I actually trust, and the receipts under it
I trust the 9 of 9 because I can open every one of the nine. This is the part I keep pointing people to. CodeGraph does not read the estate as text, it parses it into a typed knowledge graph, and every edge carries its own file:line source. On the shipped fixture the graph holds 47 nodes and 70 edges across seven node types: programs, copybooks, variables, DB2 tables, JCL jobs, datasets, and one unresolved placeholder. When I ask for the impact of a change to TRN-LIMIT, it returns the transitive closure, the full set of dependencies that change actually touches, and each fact points back at the line that proves it. Against the fixture's known ground-truth set the graph recovers 9 of 9; the simulated single-file window recovers 3 of 9. Those are measurements on one labeled wire-transfer scenario I built, where the true dependency set is known by construction, and I say that scope out loud on purpose rather than dress it up as a guarantee over arbitrary COBOL. The analyses underneath are plain graph algorithms, deterministic Python with no model in the loop, so the same fixture returns the same closure every run. This is the whole thesis for me: the map is the product, and translation is a downstream use case that needs the map first.
The one reference I let the tool refuse to resolve
I am proudest of the reference the tool refuses to resolve. On the same fixture CodeGraph resolves 33 of 34 references, which it reports as 97.1 percent coverage. The one it cannot resolve is a dynamic CALL WS-PROGNAME inside a program named DISPATCH, where the target is computed at runtime (DISPATCH.cbl:15), so there is no honest way to know statically which program it invokes. The tool flags it for review and refuses to guess. That was a decision I made early: every PERFORM, CALL, COPY, and DB2 reference has to resolve or be marked "needs review," never silently dropped. A tool that quietly drops the one dependency it could not chase is worse than useless inside a bank, because the gap stays invisible right up until it surfaces as a referential-integrity failure in UAT. I would rather ship a real 97.1 percent with a named exception than a comfortable 100 that lies.
The completeness gate. 33 of 34 references resolve to 97.1 percent, and the single unresolvable one, DISPATCH's runtime-computed dynamic CALL, is flagged for review rather than dropped.
The order I would actually touch the code in
I also wanted the tool to answer the question every engineering lead asks me first: where do I even start. So the demo scores all 14 programs by coupling and blast radius and hands back a strangler-fig extraction order. AUDITLOG comes out first, rank 1 at risk score 0, because it has zero coupling and nothing cares when you touch it. WIRETXN lands at rank 11 for its one COMP-3 trap plus JCL criticality, and DISPATCH at rank 12 for that unresolved dynamic call. The god-program ACCTMGR extracts last, rank 14, coupling 5, risk 15, because everything leans on it and it should be the final thing you dare to move. It is the reverse of how these projects usually begin, which is with whatever module is politically loudest that quarter. The ranking is deterministic too, the same fixture yields the same order every run, so it is an argument you can rerun in front of a skeptic rather than a gut call.
The safe extraction order. AUDITLOG first at zero risk, the god-program ACCTMGR last at coupling 5 and risk 15, so you move the safe modules before the one everything depends on.
The question I keep coming back to
I keep coming back to one belief this build hardened for me. A real mainframe estate runs to one, five, ten million lines and beyond, and it does not fit in any context window, present or future, so the hard part was never the model's reasoning. It is retrieving the exact slice a change touches and proving you found all of it. A better model drafts cleaner Java, and I am glad of it, but a perfect model still cannot show a regulator which dependencies it retrieved, still owes a safe extraction order, still cannot tell you the flag was set by a 2 a.m. job it never saw. Agents advise, code decides. If you want to flip the same toggle and watch the six facts vanish for yourself, the Legacy COBOL Modernization demo runs the whole analysis end to end with no API key, no database, no Docker. So the question I would put to anyone about to greenlight a COBOL migration is this: before you translate a single line, can your tool prove which dependencies that change touches, and can it name the one it could not? Mine spent an afternoon learning to admit the one it can't, and I think that admission is the product.