A student loan company just paid $2.5 million because its AI penalized borrowers for where they went to college. Not their credit score. Not their income. Their school's average default rate — a number that correlates heavily with race and generational wealth. Earnest Operations didn't set out to discriminate. But its algorithm did, systematically, and Massachusetts' Attorney General proved it.
This isn't an isolated incident. It's the opening salvo in what I believe will be the defining regulatory battle of the next three years: algorithmic accountability in lending. And most financial institutions are nowhere near ready.
I've spent the last year studying how AI systems fail in financial services — not in theory, but in the actual enforcement actions, settlements, and lawsuits now piling up. What I've found is that the problem isn't AI itself. The problem is how we're building it: thin layers of code wrapped around general-purpose language models, with no understanding of the legal minefield underneath.
The School You Attended Shouldn't Determine Your Creditworthiness
The Earnest case is worth understanding in detail because it reveals a failure pattern I see everywhere.
Earnest used something called the Cohort Default Rate — essentially, the average loan default rate at a specific college — as a factor in its student loan refinancing decisions. On paper, it seems reasonable. Schools with higher default rates might produce riskier borrowers, right?
The problem is what that number actually measures. Historically Black Colleges and Universities and schools serving low-income communities have higher default rates not because their graduates are less capable, but because of decades of systemic underfunding and intergenerational wealth gaps. By using CDR as a scoring factor, Earnest's model applied a collective penalty to individual applicants based on demographics they couldn't control.
An algorithm that judges you by your school's average instead of your actual finances isn't assessing risk. It's laundering historical discrimination through math.
That wasn't the only issue. The company also had hard-coded "knockout rules" that automatically rejected anyone without at least a green card. And when investigators looked at the human oversight layer — the part that was supposed to catch these problems — they found underwriters routinely bypassing the model with no documentation, no consistency, and no accountability.
When I first read the settlement details, what struck me wasn't the bias itself. It was the complete absence of architectural safeguards that should have caught it years earlier. No independent model validation. No proxy variable testing. No systematic monitoring of outcomes across demographic groups. I wrote about the full taxonomy of these failures in our interactive analysis of the algorithmic accountability crisis.
Navy Federal: When the Numbers Are Damning and the Excuses Don't Hold
If Earnest shows how a specific variable can poison a model, Navy Federal Credit Union shows what happens when an entire system drifts toward inequity and nobody notices — or nobody acts.
In 2022, the nation's largest credit union approved roughly 77% of white mortgage applicants but only 48.5% of Black applicants. That 29-percentage-point gap was the widest of any top 50 mortgage lender in the country.
Navy Federal's defense was predictable: public data doesn't include credit scores and cash reserves, so the comparison is unfair. But when independent researchers controlled for more than a dozen variables — income, debt-to-income ratio, property value, neighborhood characteristics — Black applicants were still more than twice as likely to be denied as white applicants with identical financial profiles.
That residual gap is the part that should keep every lending executive up at night. It means something inside the underwriting process is producing discriminatory outcomes that can't be explained by legitimate credit factors. And in May 2024, a federal judge agreed that was enough to let disparate impact claims proceed to discovery — meaning the institution now has to open up its "secret" underwriting logic.
Statistical disparity alone can survive a motion to dismiss. The burden then shifts to you to prove your process is necessary and the least discriminatory option available.
I had a conversation with a compliance officer at a mid-size lender last year who told me, "We don't have a bias problem — we use the same model for everyone." That's exactly the mindset that created Navy Federal's situation. Treating everyone the same with a biased instrument doesn't produce fairness. It produces uniform discrimination.
Why Most AI Lending Systems Are Built on Sand
Here's the uncomfortable truth about how most financial institutions are deploying AI today: they're running thin "wrapper" architectures. Take a general-purpose large language model — GPT-4, Gemini, whatever's newest — pipe some borrower data into it, and use the output for decisions or document processing.
This approach has three fundamental problems that no amount of prompt engineering can fix.
The hallucination problem. LLMs work by predicting the most likely next word in a sequence. They don't retrieve facts or perform calculations. In a credit underwriting context, that means an LLM can generate a perfectly plausible-sounding justification for a loan denial that has no basis in the applicant's actual financial file. If your automated system misstates eligibility criteria — even once — you're facing direct liability. Ask Air Canada, which was held legally responsible for its chatbot's fabricated refund policy.
The context vacuum. Generic AI platforms don't understand mortgage documents, tax returns, or the difference between a one-time bonus and recurring income. Without industry-specific training, these models produce what I call "confident incompetence" — wrong answers delivered with perfect grammar and zero hesitation. Creditworthy borrowers get rejected because the model can't recognize alternative income patterns or non-traditional credit histories.
Training data contamination. LLMs learn from the internet, which is saturated with historical biases. When one of these models evaluates a borrower's employment history or background, it may apply stereotypes baked into its training data — associating certain professions, nationalities, or neighborhoods with lower creditworthiness, even when the individual's actual data tells a completely different story.
The question isn't whether your AI wrapper has bias. It's whether you've built the infrastructure to detect it before a regulator does.
"The Algorithm Decided" Is No Longer a Legal Defense
The regulatory landscape has shifted from vague guidance to explicit mandates, and the timeline is now, not someday.
The Consumer Financial Protection Bureau has made it clear: lenders must provide "accurate and specific reasons" for adverse actions. You can't hide behind broad categories like "insufficient credit history" if the real reason was an algorithmic flag triggered by a shopping pattern or a non-traditional data point. "The algorithm decided" is not a defensible statement — the CFPB has said so directly.
Meanwhile, banking regulators are applying SR 11-7 — the definitive standard for model governance — with new intensity to AI systems. Three requirements now carry real teeth. First, you must document the actual economic logic of your model and prove it isn't relying on spurious correlations. Second, the team validating the model must be completely independent of the team that built it. Third, you need regular back-testing against real-world outcomes, not just training data performance.
And NIST's updated AI Risk Management Framework introduces something called an "AI Bill of Materials" — essentially a complete inventory of where your data comes from, what models you're using (including third-party APIs), and how all the pieces interact. Think of it as a nutritional label for your AI system. If you can't produce one, you can't defend your system.
For the detailed regulatory analysis and the technical framework we've developed to address these requirements, see our full technical deep-dive on fair lending and Deep AI architecture.
What Actually Works: Building AI That Can Defend Itself
After studying these enforcement actions and working through the technical requirements, my team arrived at a conviction: the wrapper model is fundamentally incompatible with regulated financial services. What's needed is what we call "Deep AI" — systems where fairness, explainability, and compliance aren't features bolted on at the end, but structural properties of the architecture itself.
The approach we've built uses multiple specialized layers instead of a single general-purpose model. Hard compliance checks — like residency requirements or age verification — run through deterministic rule engines that produce the same answer every time, with no probabilistic wiggle room. Structured credit scoring uses gradient-boosted models like XGBoost, which are inherently more interpretable and stable than neural networks. LLMs only enter the picture for what they're actually good at: reading unstructured documents and extracting specific entities, grounded by the applicant's actual files through retrieval-augmented generation.
Before any data touches a model, it passes through validation across six dimensions: accuracy, completeness, consistency, timeliness, relevance, and representativeness. Dirty data is the single most common source of biased outcomes, and most wrapper architectures do zero data validation.
On the fairness engineering side, we run continuous monitoring of approval rates across demographic groups, tracking metrics like the Disparate Impact Ratio — where the approval rate for a protected group should stay above 80% of the rate for the control group. When that ratio starts drifting, the system flags it before it becomes a pattern. We also employ adversarial debiasing during model training: a secondary model tries to predict the applicant's race or gender from the primary model's predictions. If it can, the primary model is retrained until it can't.
Fairness isn't a checkbox you tick during an annual audit. It's a continuous measurement that requires infrastructure most lenders haven't built.
And for every adverse action, the system generates counterfactual explanations in plain language: "If your credit utilization were 15% lower, or your income were $5,000 higher, this loan would have been approved." That's not just good compliance — it's the kind of transparency that builds genuine trust with borrowers.
What If Our Approach Is Too Expensive or Too Slow?
I hear two objections constantly. The first: "This sounds like it costs ten times what a wrapper costs." It does cost more upfront. But Earnest just paid $2.5 million for a single settlement in a single state, plus the cost of rebuilding its entire underwriting system under regulatory supervision. Navy Federal is facing consolidated class-action litigation with discovery into its core algorithms. The question isn't whether Deep AI is expensive — it's whether you can afford the alternative.
The second objection: "Won't all this validation and monitoring slow down loan decisions?" It can, if built poorly. But the bottleneck in most lending pipelines isn't model inference time — it's the manual review triggered by low-confidence decisions, the rework caused by inaccurate document extraction, and the legal review required when adverse action notices are vague. A system that gets it right the first time, with defensible documentation, actually moves faster end-to-end.
The Real Choice Facing Lenders Right Now
The first wave of AI in financial services is over. It was defined by speed-to-market, black-box experimentation, and the assumption that regulators would be slow to catch up. They weren't.
The second wave will be defined by something different: whether your AI system can explain itself under oath. Can you show exactly why a specific applicant was denied? Can you prove you searched for less discriminatory alternatives? Can you demonstrate that your human oversight layer actually functions, with documentation?
The institutions that answer yes will have a genuine competitive advantage — not just in avoiding fines, but in earning the trust of borrowers, regulators, and the public. The ones that keep running thin wrappers and hoping for the best are building on borrowed time.
I'd genuinely like to hear from anyone navigating this transition — whether you're a compliance leader trying to audit AI systems you didn't build, a CTO evaluating architecture options, or a founder in the lending space wrestling with these trade-offs. What's the hardest part for you right now?