Chicago built an algorithm to predict who would be involved in gun violence. By the time they shut it down, the system had flagged over 400,000 people — including 56% of Black men in the city between ages 20 and 29. Most had never been convicted of a violent crime. The algorithm didn't predict the future. It automated the past.
I've spent the last two years studying how AI systems fail in high-stakes environments, and the pattern that keeps emerging isn't a technical bug. It's an architectural one. The same structural flaws that destroyed predictive policing — opaque logic, biased training data, zero accountability — are now being shipped into enterprise AI products at breathtaking speed. And most companies deploying these tools have no idea they're sitting on the same kind of risk.
The Algorithm That Ate Itself
The Los Angeles Police Department used a system called Geolitica (formerly PredPol) that borrowed its math from earthquake prediction. Seismologists model aftershock patterns — where one tremor hits, another is likely to follow. Geolitica applied that same logic to crime, generating tiny 500-by-500-foot "hotspot boxes" where officers should patrol.
It sounds reasonable until you look at what actually happened.
A 2019 audit by the LAPD Inspector General found "significant inconsistencies" in how officers entered data. Hotspot predictions were contaminated by patrol time logged at police stations, not actual crime scenes. In comparable jurisdictions like Plainfield, New Jersey, the system's prediction accuracy was documented at less than 1%.
The LAPD didn't quietly sunset the program. They ran it for nearly a decade before abandoning it in early 2024.
The system didn't uncover criminal patterns. It validated existing patterns of policing.
What struck me when I first dug into this wasn't the technology's failure — it was how long the failure persisted without anyone being able to prove it. The system was a black box. The vendor treated its methodology as proprietary. And the police department using it couldn't even measure whether it worked.
Feedback Loops: When Bad Data Breeds Worse Decisions
The deeper problem is something researchers call a runaway feedback loop — and once you see it, you start noticing it everywhere.
It works like this: an algorithm predicts high crime in a neighborhood based on historical arrest data. More officers get deployed there. More officers means more stops, more arrests for minor offenses — jaywalking, small-quantity drug possession — things that go unnoticed in wealthier areas. Those new arrests feed back into the system as fresh "evidence" that the neighborhood is dangerous, which triggers even more police presence.
The data from California's Racial and Identity Profiling Act makes this painfully concrete. Officers conducted 4.7 million stops in 2023. Black individuals were stopped 126% more frequently than their population share would predict. But when officers actually searched people, they were less likely to find contraband on Black and Latino individuals than on white individuals.
The algorithm wasn't finding crime. It was manufacturing the appearance of crime by concentrating enforcement where enforcement had always been concentrated.
When the output of your model influences the collection of your next training set, you don't have a prediction engine. You have a bias amplifier.
I remember the moment this clicked for my team. We were mapping the architecture of a client's risk-scoring system — completely different domain, financial services — and someone said, "Wait, this has the same circularity problem." The model's flags were triggering reviews, and the review outcomes were being fed back as training data. Nobody had designed it that way intentionally. It had just... emerged.
We explored this dynamic in depth in our interactive analysis of algorithmic integrity, where we trace how these feedback loops form across different industries.
Over 40 Cities Said "Enough" — Your Industry Is Next
The regulatory response to predictive policing has been swift and decisive. San Francisco banned police use of facial recognition in 2019. Boston and Portland followed. Santa Cruz explicitly banned predictive policing technology. Chicago let its ShotSpotter contract expire. The White House issued a landmark policy in March 2024 requiring federal agencies to conduct independent testing and mandatory impact assessments for any AI system that affects people's rights.
Over 40 cities have now moved to ban or strictly restrict these technologies.
This isn't just a law enforcement story. Every piece of legislation being written about AI accountability — the EU AI Act, California's expanding transparency requirements, the NIST AI Risk Management Framework — draws directly from the lessons of these failures. If your enterprise deploys AI in hiring, lending, insurance underwriting, or medical triage, the same scrutiny is coming for you. In some sectors, it's already arrived.
The question isn't whether your AI will face an audit. It's whether it can survive one.
Why Most Enterprise AI Can't Survive Scrutiny
This brings me to what I see as the most urgent problem in enterprise AI right now: the vast majority of companies are deploying what the industry calls "LLM wrappers" — essentially a thin interface layer on top of a foundation model like GPT-4 or Claude, with some prompt engineering bolted on.
These wrappers deliver quick wins. They're fast to build, impressive in demos, and cheap to deploy. But they inherit every structural risk that destroyed predictive policing: they can't explain their reasoning, they reflect whatever biases exist in their training data, and they have no mechanism for domain-specific validation.
When we tested a naive LLM wrapper on security vulnerability classification — a task where getting the answer wrong has real consequences — it achieved roughly 51% accuracy. Barely better than a coin flip. The model couldn't distinguish between a minor bug and a critical exploit because it lacked the specialized reasoning layers needed for that domain.
Worse, many foundation models have safety alignments that make them hedge on difficult questions. In enterprise contexts like legal review or financial underwriting, an AI that won't take a firm, evidence-based position isn't just unhelpful — it's a liability.
A coin-flip AI with a confident interface is more dangerous than no AI at all.
By contrast, when we built multi-layered systems with composable agents — specialized modules that can perform deep research, pull context from proprietary knowledge bases, and apply both inductive and deductive reasoning — classification accuracy jumped to approximately 89%. The architecture matters more than the model.
What "Deep AI" Actually Means (And Why It's Hard)
I use the term "deep AI" not as marketing language but as a technical distinction. A deep AI solution has four properties that a wrapper does not.
It audits its own data before it learns from it. The first thing we do with any client engagement is examine the data for quality, accessibility, and historical bias. If predictive policing taught us anything, it's that the most sophisticated algorithm in the world will produce garbage if you feed it garbage. Seventy-eight percent of enterprise AI users in a 2024 Microsoft study were using unauthorized external AI tools — "shadow AI" — meaning most organizations don't even know what data their models are training on.
It can explain why it reached a conclusion. Explainable AI isn't a nice-to-have. It's the difference between a system you can defend in front of a regulator and one you can't. We use validation frameworks that test whether an AI's explanation actually corresponds to the features driving its decision — not just whether the answer happens to be correct.
It measures fairness mathematically, not aspirationally. Think of it like this: if your model approves loans, you need to verify that the approval rate doesn't depend on the applicant's race or gender. That sounds obvious, but achieving it requires monitoring specific metrics — like whether your true positive rates and false positive rates are equal across demographic groups — throughout the entire lifecycle: before training, during training, and after deployment.
It gets harder to break over time, not easier. AI-driven cyberattacks increased by 300% between 2020 and 2023. A production AI system needs continuous monitoring for adversarial inputs, prompt injections, and model drift — the gradual degradation that happens when real-world data shifts away from what the model was trained on. What was fair yesterday might not be fair tomorrow.
For the full technical methodology behind these approaches, including our fairness metrics and bias mitigation strategies, see our detailed research on architectures of trust.
"But Our Use Case Is Low Risk"
I hear this constantly, and it's almost never true.
The teams that built Chicago's Strategic Subject List didn't think they were building a tool of mass surveillance. They thought they were building a violence prevention system. The algorithm over-relied on arrest records, including low-level misdemeanors with no statistical connection to future gun violence. It created a "suspect" status for people who had done nothing wrong, triggering unannounced police visits that shattered community trust. By the time the Inspector General's audit documented the failures, the system had been running for nearly a decade.
Every enterprise AI system that touches people — customers, employees, patients, applicants — carries this risk. The feedback loop doesn't announce itself. The bias doesn't show up in the demo. It shows up three years later, in an audit, in a lawsuit, in a pattern of outcomes that nobody designed but everybody enabled.
If your organization is deploying AI in any decision-making path that affects human outcomes, there are three things I'd recommend doing this quarter:
Trace your training data backward. Where did it come from? What human decisions does it encode? Would those decisions survive public scrutiny?
Test for circularity. Does your model's output influence the data it will be retrained on? If yes, you have a feedback loop. Map it.
Demand explanations. If your AI vendor can't tell you why their system reached a specific conclusion — not just that it did — you don't have an AI partner. You have a black box with a billing address.
The Real Question
The collapse of predictive policing isn't a cautionary tale about law enforcement technology. It's a preview of what happens when any organization deploys AI without architectural integrity — without explainability, without fairness measurement, without continuous oversight.
The enterprise world is roughly where policing was in 2014: early in adoption, high on potential, and largely unaware of the structural risks accumulating beneath the surface. The organizations that build governance into their AI architecture now won't just avoid the failures. They'll be the ones still standing when the regulatory wave hits.
I'd genuinely like to hear from anyone wrestling with this in their own organization. What does your AI audit process look like today? And if the answer is "we don't have one" — what's stopping you?