The number I was proud of, and the number that stopped me an hour later
I had the lift number I wanted, and for about an hour I thought the hard part was behind me. I was building FloodProof on a real book of 170,803 NFIP claim records from Harris County, Texas, pulled from OpenFEMA, with 135,381 of them carrying a paid building claim. (OpenFEMA censors each claim's coordinates to roughly its tract centroid, which the demo says out loud, so the backtest runs at the honest resolution public data allows.) I fit a ridge composite of real building attributes (age, number of floors, elevated indicator, post-FIRM code, occupancy, obstruction) on a 70% training split and scored it on the 30% I held out. On 40,615 claims the model had never seen, the composite captured 1.69x the FEMA flood zone's top-decile loss dollars, Gini 0.31 against the zone's 0.08. It was a genuinely non-circular result: the baseline was the real ratedFloodZone, the label was the real amountPaidOnBuildingClaim, and I had deliberately left insured coverage out of the score so it couldn't cheat off the payment bound.
Then I ran the fairness audit, and the number that came back was 0.363.
Why I aimed the audit at my own model, not just the raw data
I had assumed, going in, that fairness screening was something you point at the incumbent. FEMA's zone is a blunt instrument, everyone in flood pricing knows it. In this one book, $3.1B of building-claim dollars, 45.9% of the total, was paid on properties FEMA rated outside its high-risk zones. That is the real, unmodeled fact I keep coming back to, and it is exactly the mispricing a smarter factor is supposed to fix.
FloodProof toggles the same real claims between the FEMA zone and the composite. The gold rings are properties FEMA rated safe that the composite scores high-risk.
But a factor that ranks loss better can also carry demographic signal, and if it does, it is not a rate improvement, it is a discriminatory rate the examiner will reject. So I made the audit screen every priced variable against real tract-level minority share from CDC/ATSDR SVI 2022, using the EEOC 80% rule, the adverse-impact ratio has to sit inside the band [0.80, 1.25]. I made it screen my own composite score too. The composite passed, AIR 0.938. I was relieved for about a second, until I read the rest of the table. Five variables failed. Number of floors came in at 0.363. Insured value at 0.526. And, with a kind of poetry I did not expect, FEMA's own zone tier failed the other direction at 1.467.
The afternoon I decided code, not the crew, would say no
I spent an afternoon arguing with myself about who gets to say "do not file this." FloodProof has an optional crew of Pydantic-AI agents, a factor-explainer, a fairness justifier, an adversarial challenger, a memo-drafter. It would have been easy, and demo-friendly, to let them deliberate and reach a verdict. I tried leaning on a model to arbitrate its own output once, and stopped, because the failure mode of "the model grades its own homework" is not a story I want under a rate filing. The decision that actually mattered was smaller and more boring: the thing allowed to say no is a rule in plain Python, not the crew.
So the gate is deterministic. Filing is marked ready only if the held-out sample is large enough, the lift over the FEMA zone clears 1.10x, and the adverse-impact ratio for every priced variable sits inside [0.80, 1.25], plus the per-state checklist (Colorado, for one, requires per-variable justification). Deny-by-default, filing_ready if and only if there are zero blocking findings. The agents advise. The code decides. On this book, with floors at 0.363, there was nothing to deliberate.
The audit screens all nine variables against real CDC-SVI tracts. The composite passes at 0.938, five variables fail, and the gate names floors as the worst offender.
What routing a variable to a human actually looks like
The first time I watched the gate refuse its own filing, I realized I had built the opposite of what I set out to build, and that this was the point. I set out to prove a better factor. What I ended up trusting was the part that catches the better factor when it misbehaves. The gate does not silently drop floors or auto-correct it. It emits NOT FILING-READY, names floors as the worst offender at 0.363, and routes that variable to human actuarial justification. A real actuary has to decide whether floors carries a defensible causal flood-loss mechanism or is standing in for something it should not. The value is catching that before a Department of Insurance examiner does, not pretending the model is clean.
The examiner-ready filing package, generated deterministically, carries the backtest, the Shapley attribution, and the fairness verdict, and it prints the NOT FILING-READY stamp when the gate blocks.
The lift chart is still the part people lean in for, and I understand why. Beating the incumbent 1.69x on data it never saw, robust between roughly 1.69x and 1.85x across random seeds, is a real result. But that is the easy half.
Fit on 70% of real claims, scored on the 30% it never saw. The composite concentrates loss dollars far more sharply than the FEMA zone alone.
The question I keep asking myself now
I used to think the flood model was the product. I do not think that anymore. Vendors like ZestyAI and ICEYE already sell strong models, and in FloodProof those feeds, along with the Guidewire connector, are honest stubs behind a real schema, the config swap is the production path, not a rewrite. What does not commoditize is the proof: a backtest that cannot cheat, an audit that judges your own factor and FEMA's zone tier by the same rule, and a gate that would rather block your filing than let you ship a rate you cannot defend. As 24-plus states adopt the NAIC AI Model Bulletin and New York DFS Circular Letter 2024-7 asks carriers to test for proxy discrimination, that gate stops being paperwork and starts being the point. You can run the whole thing yourself, on the same real public data, at veriprajna.com/demos/flood-risk-underwriting.
Here is the question I still sit with. If your best rating factor came back with a variable at 0.363, would your pricing stack be built to catch it, or to file it? I would rather find out on my own book than in front of an examiner.