Amazon built an AI recruiting tool that taught itself to penalize any resume containing the word "women's." Not because someone programmed it to — because that's what the math told it to do. The system trained on a decade of hiring data from a male-dominated industry, found that "being male" correlated with "being hired," and optimized accordingly. Graduates of all-women's colleges were downgraded. "Women's Chess Club Captain" became a penalty. Amazon's engineers tried to fix it, failed, and scrapped the entire project.
I've spent the last two years thinking about why this wasn't a bug — it was an inevitability. And I've become convinced that the entire approach most HR tech companies use to evaluate candidates is fundamentally broken. Not because the engineers are careless, but because the architecture itself guarantees failure.
We built something different. We call it the Glass Box.
The Machine That Couldn't Stop Discriminating
What happened at Amazon wasn't a one-off coding mistake. It was a mathematical certainty baked into how deep learning works.
Statistical models learn by minimizing the gap between their predictions and historical outcomes. Feed a model ten years of resumes where 70% of successful hires were men, and the model learns that maleness predicts success. It doesn't understand why someone is qualified. It doesn't know that Python is a programming language. It only knows that the string "Python" appeared on resumes that got hired — and so did patterns associated with being male.
Amazon's engineers tried the obvious fix: strip out gendered terms. But deep learning models are relentless pattern-finders. They discovered proxy variables — linguistic fingerprints that correlate with gender even when gender itself is hidden. Research shows that male resumes tend to use verbs like "executed" and "captured," while female resumes lean toward more communal language. The model didn't need the word "woman" to reconstruct the bias. It found another path.
The model didn't discriminate out of malice. It discriminated out of math.
This is the core problem with what I call "Black Box" AI in recruitment. The decision logic is distributed across millions of parameters. No one — not even the engineers who built it — can point to the specific reason a candidate was rejected. When Amazon's team tried to surgically remove the bias, they couldn't do it without destroying the model's ability to function at all.
Why LLM Wrappers Don't Fix This
After Amazon's public failure, you'd think the industry would have learned. Instead, the next wave of HR tech startups took Large Language Models — GPT-4, Claude, Gemini — and wrapped them around the same broken process.
LLMs are remarkable at reading and understanding language. But using them to judge candidates introduces three new problems that most vendors quietly ignore.
First, hallucination. An LLM might infer a candidate holds a certification simply because the resume's tone sounds authoritative. It invents qualifications that don't exist.
Second, inconsistency. Feed the same resume into an LLM twice and you may get two different scores. I tested this myself early on — same candidate, same job description, five runs, three materially different rankings. In an audit, that's indefensible.
Third, frozen knowledge. LLMs are trained on data with a cutoff date. A new programming framework that emerged six months ago? The model doesn't know it exists.
Using an LLM to judge candidates is like asking a brilliant speed-reader to perform surgery. The skill set doesn't transfer.
We use LLMs in our system — but only as readers, never as judges. That distinction changed everything.
The Regulatory Walls Are Already Up
This isn't just an ethical debate anymore. Governments have decided.
NYC Local Law 144, effective since July 2023, requires any employer using an automated tool to screen candidates in New York City to undergo an annual independent bias audit. The law demands specific calculations: selection rates and impact ratios broken down by race, ethnicity, and sex. If the selection rate for any protected group falls below 80% of the most-selected group — a threshold known as the four-fifths rule — that's a legal red flag for discriminatory impact.
The EU AI Act goes further. It classifies AI systems used for recruitment as high-risk, requiring that these systems be transparent enough for users to interpret their outputs, and that humans can meaningfully override decisions. Article 13 demands interpretability. Article 14 demands human oversight. A recruiter can't exercise informed oversight over a score that arrives without explanation.
Under the GDPR, candidates have the right to "meaningful information about the logic involved" in automated decisions. A generic rejection email doesn't satisfy that. "Neuron 4,502 fired at intensity 0.8" doesn't satisfy it either.
I explored the full regulatory landscape in our interactive analysis of the Glass Box paradigm. What struck me most wasn't the complexity of the laws — it was how structurally impossible compliance becomes when your underlying architecture is opaque.
What We Actually Built
The insight that changed our direction was simple, and I'm almost embarrassed it took me as long as it did to see it: you don't need to predict who will succeed. You need to measure how close someone's skills are to what the job requires.
Prediction is where bias enters. Measurement is where it exits.
We built our system around what we call an Explainable Knowledge Graph — a structured map of skills, roles, certifications, and their relationships. Think of it as a giant, interconnected web where "PyTorch" connects to "Deep Learning," which connects to "Artificial Intelligence." If a job requires "AI experience" and a candidate lists "PyTorch," the graph traces the connection and finds the match — even though the keyword "AI" never appears on the resume. A traditional keyword-matching system would miss this candidate entirely.
The architecture enforces a strict separation between reading and reasoning. The LLM reads the resume and extracts structured facts: skills, roles, project contexts. It maps "ReactJS" and "React.js" to the same node. It converts "Women's Chess Club" into a neutralized entry — "Chess Club, Leadership Role" — before the data ever reaches the reasoning engine.
Then the graph takes over. All matching, scoring, and ranking happens through graph traversal — tracing paths between nodes, calculating distances, measuring overlap. Given the same inputs, it produces the same output every time. No hallucination. No stochasticity. Every decision is traceable.
We don't offer a magic box that guesses. We offer a precision instrument that measures.
How Demographic Masking Actually Works
This is the part that convinced me we were on the right track.
When our matching algorithm runs, it operates on a restricted view of the graph — what we call the Inference Graph. This subgraph contains skills, roles, experience levels, and certifications. It structurally excludes nodes for name, gender, ethnicity, address, and graduation dates (which are often age proxies).
The bias isn't filtered out after the fact. It's architecturally impossible for the system to use it, because the nodes simply don't exist in the space where decisions happen. There is no path from "Candidate" to "Gender" to "Job Role." The connection has been severed at the structural level.
Compare this to a deep learning model, which ingests the full raw text of a resume. Even if you remove the "Gender" field, the model reads "Women's Chess Club" and infers gender. You can't stop it — that's what neural networks do. They find patterns you didn't ask them to find.
In our system, by the time the reasoning engine sees the data, the gendered modifier has already been stripped during extraction. The graph sees a strategy-oriented leadership activity. Nothing more.
Measuring Skill Distance Instead of Predicting Success
Traditional applicant tracking systems use Boolean logic — does the resume contain the keyword "Java"? Yes or no. This is brittle and eliminates qualified candidates who use different terminology.
We replaced this with something more nuanced: Skill Distance measurement using graph embeddings. Every skill in our ontology gets a vector representation — a mathematical position in multidimensional space. Skills that frequently co-occur or share relationships in the graph (like "Python" and "Pandas") end up close together. Unrelated skills (like "Python" and "Phlebotomy") end up far apart.
When we score a candidate against a job, we calculate the geometric similarity between their skill vectors and the job's requirement vectors. A candidate who knows PowerBI but not Tableau still gets a high score, because those skills are semantic neighbors in the "Business Intelligence" region of the graph. A keyword search would give them zero credit.
We also calculate what we call the Shortest Path distance for gap analysis. If a job requires Kubernetes and a candidate has Docker experience, the graph traces the connection — Docker → Container Orchestration → Kubernetes — and reports a distance of two hops. That's a trainable gap, not a disqualification.
For the full technical methodology behind our scoring algorithms, including our use of cosine similarity, Jaccard coefficients, and geodesic distance calculations, see our detailed technical research.
The Candidate a Black Box Would Have Rejected
One scenario crystallized this for me. We were testing with a candidate profile that a conventional AI system had rejected — no explanation given, just a form email.
When we ran the same profile through our graph, the system produced something entirely different: "Candidate lacks explicit SQL experience. However, graph analysis shows extensive experience with Pandas DataFrames and R Dplyr. Graph distance between DataFrames and SQL is short — both map to the concept of Data Manipulation. Recommendation: Interview. High transferability."
That's a real person who would have been filtered out by a system optimizing for keyword matches or historical patterns. Someone who probably attended a smaller school, used slightly different tools, but possessed the exact underlying competencies the role needed.
When your AI measures skills instead of matching keywords, your talent pool doesn't just get fairer — it gets larger.
This is the business case that I think gets underappreciated. Explainable AI isn't just about compliance or ethics, though those matter enormously. It's about finding people that opaque systems miss. Every false negative — every qualified candidate rejected by a Black Box — is a cost. To the candidate, obviously. But also to the company that never met them.
What About Keeping the Graph Current?
One objection I hear often: "Doesn't a knowledge graph require constant maintenance? New skills emerge all the time." It's a fair question. The answer is that adding a new skill to a graph is trivially easy compared to retraining a deep learning model. When a new framework appears — say, a JavaScript library that didn't exist six months ago — we add a node, connect it to its parent concepts, and the system immediately understands its relationships. An LLM with a knowledge cutoff can't do that without retraining.
Another common concern: "Doesn't stripping demographic data mean you can't monitor for bias?" Actually, the opposite. We maintain a separate Audit Graph where anonymized scores are rejoined with demographic data specifically to calculate real-time impact ratios. If a specific job requirement is disproportionately filtering out a protected group, the system flags it — and because the graph is transparent, we can trace exactly which requirement is causing the disparity. A Black Box that fails a bias audit gives you no path to fix it. Our system shows you the specific node to adjust.
What This Means If You're Evaluating HR AI
If you're a CHRO, a VP of Talent, or anyone evaluating AI tools for hiring, I'd push you to ask three questions of any vendor:
Can you reproduce the exact same decision twice? If the answer involves hedging about probabilistic outputs, that's a compliance risk under NYC LL 144 and the EU AI Act.
Can you show me why a specific candidate was rejected? Not a confidence score. Not a summary generated after the fact. The actual reasoning chain. GDPR Article 15 requires "meaningful information about the logic involved."
Where does demographic data live relative to your decision engine? If the answer is "we remove it during preprocessing" rather than "it's structurally excluded from the reasoning architecture," the system can still find proxy paths to the same bias.
The era of trusting Black Box AI with high-stakes human decisions is ending — not because the technology is bad, but because the architecture was never designed for accountability.
Data is a mirror. If you train a model on a biased past, you automate that bias into the future. The alternative isn't to abandon AI — it's to build AI that measures what actually matters.
I'd genuinely like to hear from anyone wrestling with this in practice. How is your organization approaching the gap between AI efficiency and AI accountability in hiring?