Analyze
Practices This Methodology Rejects
Thirteen ways a dunning recovery rate gets distorted or stripped of context, each with the rule it breaks and the page that teaches it.
This page is about dunning recovery, the campaign that starts when a subscription payment fails. Cancel flows settle into definitive sessions within a 24 hour window and have their own rules; see Two Measurement Frames.
Complete cohorts. Every outcome counted.
Every practice below breaks one of those two rules, or the rolling analysis principle that follows from them.
These practices show up across the industry, and each one makes a recovery number less trustworthy than the underlying data. If you see one in a vendor's reporting, or in your own, ask what the number would look like without it. The gap is rarely small.
1. Excluding cancellations from the denominator
The practice. Reporting a recovery rate that leaves cancellations out of the denominator, on the logic that a customer who cancelled didn't fail to recover.
Why it misleads. The recovery experience itself can cause the cancel: billing confusion, friction in the card update flow, a badly timed retry. There's no clean way to know afterward which cancellations the campaign caused and which would have happened anyway. Pulling them out of the math can lift a reported recovery rate by 10-20 points with no change in performance, and it hides spikes in active churn.
The rule. Every outcome counts. The denominator doesn't move.
Where it's taught. The Four Outcomes.
2. Dropping in-progress campaigns from a row while keeping the resolved ones
The practice. Calculating a day's rate as recovered divided by (started minus still in progress), so the row publishes a number before the day has finished.
Why it misleads. The denominator shrinks by the campaigns whose outcome you don't know yet, while the numerator keeps every recovery so far. The inflation is worst when a large share of campaigns are still running. Recent data calculated this way doesn't compare to historical data, where every campaign has resolved.
The rule. If a date has any active campaigns, the entire date is set aside until everything resolves.
Where it's taught. Daily Cohorts and Complete Cohorts.
3. Publishing a final rate for a cohort that hasn't resolved
The practice. Reporting a recovery rate for a period that still has active campaigns as if it were final, including the current month figure on a dashboard that groups campaigns into calendar month cohorts.
Why it misleads. Checked mid-period, the figure comes in low by construction, and it changes every day until the last campaign finishes. The problem is the label, not the number. A projected rate for an incomplete cohort can be useful when it comes from a weighting method with a track record on that account's history and is marked as an estimate rather than as data. Presenting it as the rate, with no marker, is what this page rejects.
The rule. Until a cohort completes, no final recovery rate is published for that date.
Where it's taught. Daily Cohorts and Complete Cohorts.
4. Comparing fixed calendar periods
The practice. Putting January next to February and calling the difference a change in performance.
Why it misleads. A calendar month is one possible window out of many. January 1 to January 31 and January 15 to February 14 can tell opposite stories about the same business, because where the boundary falls decides which swings each period catches.
The rule. Rolling analysis sums every possible window of a given length and plots them in sequence.
Where it's taught. Rolling Analysis.
5. Claiming incremental lift without a baseline
The practice. Stating that a tool or a change lifted recovery by some amount when there's no before number, or the before number came from a different method.
Why it misleads. Lift is a difference between two measurements, so it's only as good as the weaker one. A baseline that excludes cancellations, drops in-progress campaigns, or overlaps the transition produces a lift figure that measures the methodology gap rather than the change. With no baseline, there's nothing to check the claim against.
The rule. A baseline is only comparable when it uses the same methodology as the after.
Where it's taught. Comparing Periods, Tools, and Migrations and Proving a Lift.
6. Quoting a single recovery rate as one number
The practice. Citing one recovery rate, ours or any vendor's, as though it were a fixed property of the tool.
Why it misleads. Recovery rates vary widely across accounts for reasons that have nothing to do with the recovery system: sample size, seasonality, price points, billing cycles, customer acquisition, service quality. A single figure is one account's position on that distribution, or the top of it, presented as the whole.
The rule. There is no single Churn Buster recovery rate. Each account sits somewhere on a range.
Where it's taught. Realistic Recovery Rate Range.
7. Attributing recoveries by which system got there first
The practice. Counting only the recoveries one tool's retry or email triggered, and reporting that as the recovery rate.
Why it misleads. Attribution rewards retrying early and aggressively to beat the platform to the payment, rather than recovering the customer. Two tools running on the same account can each claim credit for most of the recoveries.
The rule. A successful payment is a successful payment, whether it came from a card update, a retry we ran, or a retry the platform ran.
Where it's taught. The Four Outcomes.
8. Comparing a resolved cohort against a frozen or mid-period number
The practice. Judging a migration by comparing the new tool's complete cohorts against a baseline that stopped updating when the old flow stopped, or against the old tool's current period rate.
Why it misleads. Native recovery analytics usually exist only while the native flow is running, so the old number froze at whatever the dashboard showed on the day of the switch, often mid-cohort. The new tool's number is a finished cohort. The comparison makes a healthy migration look like a regression.
The rule. Take the pre-switch number deliberately, through an export or a read-only connection, or it's gone, and you end up comparing an in-progress cohort on the new tool against a frozen number on the old one.
Where it's taught. Comparing Periods, Tools, and Migrations.
9. Comparing nested windows
The practice. Comparing the last 3 months to the last 6 months, where the 6-month window contains the 3-month one.
Why it misleads. The two numbers share most of their data, so any difference between them comes out muted and looks like flat or declining performance. Any dashboard with a period picker invites this.
The rule. Compare non-overlapping periods.
Where it's taught. Rolling Analysis and Comparing Periods, Tools, and Migrations.
10. Comparing two retention figures without checking the denominator
The practice. Putting one tool's retention rate next to another's without confirming what each counts as active.
Why it misleads. A retention figure that keeps long inactive subscribers in the active bucket flatters itself. Subscribers who passively churned years ago, with no payment since, still count as retained.
The rule. Check what the denominator counts before comparing two retention numbers.
Where it's taught. Comparing Periods, Tools, and Migrations.
11. Measuring a vendor's contribution against a baseline fixed at the worst period
The practice. Setting the baseline at one low period, often the first month, and crediting every point above it to the vendor, month after month.
Why it misleads. Recovery rates swing 20-30 points between their best and worst rolling window with no change to the recovery process. A baseline pinned to a low point credits that swing, and any improvement in the business itself, to the tool. The same failure appears whenever a vendor bills or reports on each upward swing without first establishing the range the account moves in anyway.
The rule. Only a move outside your range, sustained across several rolling windows, is evidence that something changed.
Where it's taught. Natural Variance.
12. Claiming a specific retry time outperformed another
The practice. Reporting that a retry at one hour or one day recovered more than it would have at another.
Why it misleads. Every other number in this methodology traces back to a row in a table; this one has no row to trace to, and answering as if it did implies a precision the data can't support. The same holds for adjacent questions in the same shape, such as which retry number in a campaign did the work.
The rule. The alternate version of that charge never ran, so there's no counterfactual to measure it against.
Where it's taught. What Moves a Recovery Rate.
13. Reporting a standard deviation or confidence interval as natural variance
The practice. Presenting a statistical spread as the account's natural variance.
Why it misleads. A bad week inside the band isn't a crisis. A good week inside the band isn't a victory. Both calls need the ends of the range: the highest and lowest period. A standard deviation or confidence interval summarizes those ends away.
The rule. We measure it as the gap between the highest and lowest period in your complete history, not a standard deviation, not a confidence interval.
Where it's taught. Natural Variance.
Before trusting any recovery number, ask which of these it depends on.
Prerequisite: none required. The Glossary of Recovery Measurement Terms defines the terms.
Previous: Comparing Periods, Tools, and Migrations, the last teaching page in Analysis.