Every carrier writing flood already knows the FEMA flood zone is a blunt instrument. That argument was won years ago. The part almost no one has solved is what comes after the model: proving a new rating factor is genuinely better without letting it grade its own homework, and proving it does not carry demographic signal that a Department of Insurance examiner will reject. AI-powered flood risk underwriting lives or dies in that second half, and a better model does not fix it. A better model that self-grades is not evidence, and a better model that fails the 80% rule is a filing your examiner sends back.
We built FloodProof (veriprajna.com/demos/flood-risk-underwriting) to make that second half concrete on a real book of claims. It runs on public OpenFEMA NFIP records for Harris County, Texas, screens every priced variable for disparate impact against real census demographics, and then, on this book, refuses to mark its own filing ready. That refusal is the product.
The number that needs no model at all
Start with a fact no algorithm touched. Of the real building-claim dollars paid in this Harris County book, 3.1 billion, or 45.9 percent, was paid on properties FEMA rated outside its high-risk Special Flood Hazard Area zones. That is straight from 135,381 scored claims (170,803 raw records ingested), with no scoring model involved. The zone map is not slightly off at the margins. Nearly half the loss dollars sit outside the lines it draws.
The FEMA zone to AI composite toggle. Gold rings are FEMA misses: properties rated outside the high-risk zone that the composite scores as high risk. 3.1B in real claim dollars (45.9 percent) fell outside FEMA high-risk zones in this book.
Beating the zone is the easy claim to make and the hard one to prove
Anyone can report a lift number. The question a chief actuary asks next is whether the backtest was rigged. Ours is built to answer that in the negative. The composite is fit on a 70 percent train split and scored on the held-out 30 percent it never saw. The baseline it is measured against is the real FEMA ratedFloodZone, and the label is the real amountPaidOnBuildingClaim. Both are independent of the model, so "composite beats FEMA zone" is a genuine out-of-sample result rather than a curve fit. We deliberately exclude insured coverage from the score, because payment is mechanically bounded by coverage and including it would let the model cheat off the answer.
On 40,615 held-out real claims, the composite captures 1.69x the FEMA-zone baseline's top-decile loss dollars, with a Gini of 0.31 against the zone's 0.08. Rerun with a different random seed and the lift lands between roughly 1.69x and 1.85x, so it is not a lucky split. Those numbers are for this one county book and this held-out split, not an open-world guarantee, and they recompute live on every run.
The backtest is non-circular by construction: fit on a 70 percent train split, scored on the held-out 30 percent, measured against the real FEMA zone with the real claim dollars as the label, and with the payment-bound feature deliberately excluded so the score cannot cheat.
The held-out backtest. Top-decile claim-dollar capture and Gini, composite versus FEMA zone only (roughly 0.31 against 0.08), computed on the 40,615 claims the model never trained on.
The audit that grades our own score, and FEMA's
A rating factor that ranks loss well can still rank demographics well, which is how a good model becomes a rejected filing. So FloodProof runs a disparate-impact audit across 738 real census tracts (CDC/ATSDR SVI 2022), and it holds every priced variable to the same EEOC 80 percent rule, band 0.80 to 1.25, including our own composite and FEMA's own zone tier.
The honest part is this. The composite score itself passes at an adverse-impact ratio of 0.938. We do not get to tell a tidy story where the AI is the villain. But the audit screens every variable, and 5 of them fail. The worst offender is number of floors, at an AIR of 0.363, well outside the band, with a correlation to tract minority share of -0.406. Floors carries a real causal flood-loss signal and is also the strongest demographic proxy in the book. Insured value fails at 0.526. And FEMA's own zone tier fails at 1.467, which means the baseline everyone files against today would not pass the fairness test the AI factor has to pass.
The disparate-impact table across 738 real CDC-SVI tracts. Composite passes at 0.938; floors (0.363), insured value (0.526) and FEMA's own zone tier (1.467) fail the 80 percent rule. Five variables fail in total.
The gate that produces the filing, or refuses to
None of that audit matters if a human can wave it through under deadline. So the decision is not left to a person or to an LLM. A deterministic policy gate marks the filing ready only when three conditions all hold: the held-out sample is large enough (n at least 2,000), the composite lift over FEMA zone clears the required minimum (at least 1.1x), and every priced variable's adverse-impact ratio sits inside the 0.80 to 1.25 band. It layers a per-state filing checklist on top, so Colorado's requirement for per-variable justification is checked, not assumed.
On this book, floors fails the band, so the gate returns NOT FILING-READY, names floors as the worst offender, and routes it to human actuarial justification instead of shipping the rate. The rule is one line and it is the whole philosophy: filing is ready if and only if there are zero blocking findings. Deny by default.
Agents advise, code decides. The optional Pydantic-AI crew can draft the memo and argue both sides of a fairness call, but it cannot override the gate. Every trust-critical number is plain numpy outside the agent framework.
The generated DOI filing package. The gate assembles the examiner-ready memorandum, backtest, and feature-importance tables, and stamps the verdict at the top: on this book, NOT FILING-READY.
Why this holds as the models get better
The tempting read is that this is a flood-modeling story. It is not. Vendors already sell strong flood models, and the ones stubbed behind FloodProof's schema (ZestyAI Z-FLOOD, ICEYE SAR-depth) plug in through a config swap, not a rewrite, alongside a stubbed Guidewire and Duck Creek connector. Swap in a better model tomorrow and every piece of infrastructure around it still has to run: the non-circular backtest, the fairness audit on every variable, and the gate that can refuse to file. That infrastructure does not age out as the models improve. It is the part regulators are now asking about directly, as the NAIC AI Model Bulletin reaches 24-plus states, New York DFS Circular Letter 2024-7 requires proxy-discrimination testing, and Colorado requires per-variable justification for AI-driven rates.
The durable value is the integration and the proof, not the claim that a model is wrong. FloodProof (veriprajna.com/demos/flood-risk-underwriting) is our attempt to build exactly that, on real public data, with the disclosure that public OpenFEMA claims censor geocoordinates to roughly tract centroid, so property-level precision belongs to a carrier's own exact geocodes.
Here is the question we keep coming back to, and it is a fair one to put to your own stack. When your AI-augmented flood factor is ready to file, what actually decides whether it ships: a person signing off under deadline, or a rule that checks every priced variable against the 80 percent band and blocks the filing before the examiner does?