My first instinct, when a growth lead showed me their retention dashboard, was that we were looking at a healthy system. Twenty-eight percent save rate. Save offers converting at double the baseline. Churn trending flat.
My second instinct, a week later when I'd run the numbers differently, was that we were looking at a system optimizing itself in the wrong direction.
The issue wasn't the save rate. The issue was what save rate was counting. Three of the four populations inside that metric shouldn't have been in the cancel flow at all.
The Holdout That Changed How I Think About Cancel Flows
I came into this work with the assumption that most subscription businesses had run proper holdout experiments on their retention flows. The holdout setup isn't exotic: assign a random subset of cancel-intent users to a control condition, no save offer, immediate clean exit. After 90 days, compare churn rates. The counterfactual tells you what the save flow is actually causing versus what would have happened anyway.
When I ran that experiment for the first time on a production cancel flow—a well-resourced SaaS platform with a dedicated retention team and a mature ProsperStack integration—the control group retained at a comparable rate to the treatment group for one segment of users, and at a higher rate for another.
That second finding is the one that takes a moment to process. The cancel flow was triggering churn that would not have existed. Users who were currently paying and would have continued paying were being surfaced a save offer—and the offer reminded them they were paying $49 a month for something they hadn't used since January. Your retention system manufactured its own churn.
The research literature has a name for this: Sleeping Dogs. Telenor, the Norwegian telecom, discovered the same pattern when they found that retention campaigns were producing 2% higher churn in the treatment group versus holdout. They only found out because they ran the experiment. Most subscription businesses never have.
What the CATE Distribution Actually Showed
The framing I use when building these models is Conditional Average Treatment Effect—CATE—which estimates not "is this user likely to leave?" but "will this specific user leave because of our intervention, or regardless of it?"
The distinction matters because standard churn prediction answers the wrong question. A high churn-risk score identifies a user who's likely to cancel; it says nothing about whether contacting them changes the outcome or accelerates it.
When I showed the CATE distribution from that first production deployment to the growth lead who'd been celebrating the 28% save rate, the conversation was uncomfortable in a specific way. Roughly a third of their "saved" users were Sure Things—people who clicked cancel by accident or to probe for a discount, who were going to stay regardless of what offer appeared. About 12% of the cancel-intent users the flow was touching showed negative treatment effects: the save flow contact was associated with higher eventual churn than the holdout. Those were the Sleeping Dogs.
Save rate was adding the Sure Things to the numerator as wins and hiding the Sleeping Dogs as ordinary churn. The metric was structurally incapable of showing the problem it was creating.
The hardest conversation in that engagement wasn't with the growth lead. It was with the board. The question "so we should contact fewer users?" is genuinely counterintuitive when your retention instinct has always been to run more interventions, more offers, more touchpoints. The answer is yes, for specific segments, and no, for others—and knowing which segment each user belongs to is the entire job.
The Compliance Piece Nobody Was Watching
I started this work focused on the causal segmentation problem. The FTC enforcement landscape pushed me to take the compliance side as seriously.
The sequence of enforcement actions since 2022 is documented publicly: Amazon's $2.5 billion settlement in September 2025 for the "Iliad Flow" (a 4-page, 6-click, 15-option cancellation maze); Epic Games' $245 million for one-button purchases and locked accounts; Vonage's $100 million for continued charging after cancellation requests. Chegg and HelloFresh each settled for $7.5 million in September 2025.
What caught my attention most was JustAnswer in January 2026—the first major FTC enforcement action against an AI save agent. The chatbot was called Pearl. The FTC described it as facilitating "rampant consumer deception" by locking consumers into recurring charges through a conversational flow. The legal basis was Section 5 of the FTC Act, the same authority used against Amazon. The precedent it establishes is direct: an AI save agent that adds conversational friction before surfacing the cancellation confirmation carries the same exposure as a multi-step manual flow.
Every company I've talked to deploying a GPT-based save agent—and there are more than a few—treats it as a customer experience optimization. Almost none of them have run it through the ROSCA checklist. The JustAnswer action means that gap is now a documented enforcement risk, not a theoretical one.
The state-level patchwork is its own problem. California's "One Save" rule limits retention offers to one per cancellation attempt. New York requires online-only cancellation for online signups. Maryland and Connecticut have their own disclosure and pre-renewal notification requirements. When your subscriber base spans multiple states, the strictest applicable law governs, but most compliance reviews I've seen apply the federal ROSCA standard across the board and miss the state-level exposure.
The Vendor Gap That Pushed Me to Build This
I spent time with the main cancel-flow tools before deciding the causal segmentation problem was worth building for. ProsperStack and Chargebee Retention (formerly Brightback) optimize the save offer experience—which offer converts at what rate, which segment responds to a discount versus a feature walkthrough. They're well-built tools for the question they're answering.
The question they can't answer is whether contacting the user was the right call in the first place. The A/B testing framework is structurally misaligned with the causal question: it measures which offer converts, not which users convert because of the offer. A holdout isn't built into the platform. Sleeping Dog segmentation isn't a feature.
ChurnZero and Custify do churn prediction from health scores and usage signals. I respect what those products do for customer success workflows. But churn prediction identifies who might leave; uplift modeling identifies whose leaving is caused by our intervention. Different question, different answer, different system.
Pega Customer Decision Hub can do next-best-action decisioning at telco scale. I've seen the Vodafone references. It costs north of $500,000 to implement, doesn't audit for ROSCA compliance, and doesn't touch the Sleeping Dog segmentation problem in any of its documented case studies.
Nobody combined causal segmentation, compliant flow design, and dark pattern auditing in one system. That's what we built at Veriprajna.
What the Build Actually Looks Like
When I describe what we actually built, I start with the billing event stream—Stripe webhooks, Chargebee events, Recurly signals—and the uplift model running against it that estimates CATE for each cancel-intent user and classifies them into the four segments. Persuadables get relevant intervention. Sure Things get a fast-exit path that doesn't generate friction or FTC exposure. Lost Causes get clean confirmation. Sleeping Dogs don't enter the cancel flow.
The first deployment always includes instrumenting the holdout infrastructure, because most platforms have never run one. The Telenor result—retention campaigns causing churn in the treatment group—isn't an outlier; it's what you find when you measure causally for the first time and the prior system was contact-everyone by default.
The compliance layer maps every subscriber jurisdiction to the applicable automatic renewal law, generates audit-grade documentation for each cancellation interaction, and checks AI save agent flows against the JustAnswer pattern: does the conversational design add friction before surfacing the confirmation, in a way the FTC's Section 5 authority would treat as "not simple"?
The math a finance team can hold: a B2B SaaS platform with 200,000 subscribers and 3% monthly voluntary churn has roughly 6,000 cancel-intent users per month. Research benchmarks put the Sleeping Dog population at 10–20% of that. If the cancel flow contacts all 6,000 by default, you're activating somewhere between 600 and 1,200 Sleeping Dog cancellations per month. At $50 average revenue per user, that's $360K–$720K in annual revenue your retention system is destroying.
The retention team celebrating the save rate doesn't know about this number. The save rate can't show it. The full architecture is at veriprajna.com/solutions/ethical-subscription-retention-ai for anyone who wants the technical detail.
The question I keep sitting with, after a year of building in this space, is how many subscription businesses are running the same system and have never run the holdout that would surface the Sleeping Dog problem. The answer isn't zero. It's probably most of them.