When to Measure Recent Performance
Everyone is eager to see early results when running tests, but it's important to know when these results can be accurately calculated.
You might be surprised to learn that during a 20-day dunning campaign, you still can't calculate a recovery rate accurately after 7 days of testing.
Here's what 7 days of results can look like:
💳 1,000 failed payments entering 20-day recovery campaigns
✅ 400 recovered
❌ 100 canceled
🔄 500 still in progress
So... how do you calculate recovery rate with 500 campaigns still in progress?
You have two options:
A) Calculate a 80% recovery rate by excluding the in-progress campaigns (the subscribers most likely to churn) from the calculation.
❌ This will never be an accurate metric, and will always over-inflate your performance.
B) Calculate a 40% recovery rate by including the in-progress campaigns.
✅ This tells you what has happened so far, and the recovery rate will move higher as in-progress campaigns are recovered.
These options lead to opposite conclusions.
Neither provides an accurate calculation at this stage. However, Option A will always be misleading, whereas Option B is something you can build on over time.
In this example, after 21 days of testing you will only have 1 day of completed results to analyze. With each passing day, you will accumulate one more day of data to analyze.
Depending on your volume of failed payments, it might take at least 30 days to average out the natural variance in your data and find meaningful insights.
---
Accurate measurement takes time, and it's worth the wait. When it comes to customer retention, decisions made based on incorrect data analysis can have large, compounding effects on your business.
Please feel free to contact the Churn Buster retention team for help interpreting early test results at support@churnbuster.io.