How to Compare Against a Retention Baseline
Ideally, we would A/B test every change to our dunning process. However, when comparing two systems, the best approach is to analyze the before and after results. In the past, before and after testing meant comparing date ranges and seeing if the failed payment recovery rate improved.
Makes sense, right? Well... not quite.
To illustrate the problem with this approach, consider this chart showing the monthly recovery rate while using the exact same dunning process:
[chart showing ups/downs]
Below, we'll dig deeper into the problem and show you how to solve it.
The Problem with Comparing Two Random Date Ranges
- When you compare two recovery rates, before and after, the results are heavily influenced by the start and end dates of your test.
- For example, using the same dunning process, one test might appear successful ("won") while another seems unsuccessful ("lost") due to natural business fluctuations.
- Dunning solutions can’t take credit for these natural variations.
Solutions to This Problem
1. Compare Against Rolling X-Day Ranges:
- Calculate the recovery rate for every X-day interval in your baseline. For instance, if you’re examining 30 days of recent data, compare it against all 30-day periods in your baseline.
- This approach avoids the issue of random or cherry-picked test dates.
2. Compare Against the Normative Range:
- Compare your recent data to the normative range of recovery rates from your baseline. For example, if you're currently recovering 50% of failed payments, knowing that your baseline recovery rate ranged from 48% to 57% is more insightful. This range might indicate a downward trend, as opposed to a misleading comparison against the low end of the range, which might incorrectly suggest the opposite.
3. Use Segmentation:
- Segment your data to minimize the impact of external factors, such as surges in new subscribers or discounting.
- Analyze the four possible outcomes of dunning:
- Card updates
- Retry wins
- Cancellations
- Passive churn
This approach provides a clearer and more accurate understanding of your recovery rates and the effectiveness of your dunning process.
---
Accurate measurement takes time, and it's worth the wait. When it comes to customer retention, decisions made based on incorrect data analysis can have large, compounding effects on your business.
Please feel free to contact the Churn Buster retention team for help interpreting early test results at support@churnbuster.io.