Performance Comparison
Early Results

Early Results

Everyone is eager to see early results when running tests, but it's important to know when these results can be accurately calculated.

You might be surprised to learn that during a 20-day dunning campaign, you still can't calculate a recovery rate accurately after 14 days of testing.

Here's an example to illustrate this:

  • 💳 1000 failed payments entering 20-day recovery campaigns
  • ✅ 500 recovered
  • ❌ 200 canceled
  • 🔄 300 still in progress by day 14

So... how do you calculate recovery rate with 300 campaigns still in progress?

You have two options:

A) Calculate a 71.4% recovery rate (500 of 700 payments recovered)  by excluding the in-progress campaigns (the subscribers most likely to churn) from the calculation.

B) Calculate a 50% recovery rate (500 of 1000 payments recovered)  by including the in-progress campaigns, considering them all as effectively churned.

These options would likely lead to opposite conclusions, and neither provides an accurate calculation at this stage.

In this example, after 21 days of testing you will only have one day of completed results to analyze. Comparing this single day to each day in your baseline will yield varying outcomes. For instance, the recovery rate could be 80% one day and 20% the next.

With each passing day, you will accumulate one more day of data to analyze.

After 7 days, you could compare this period against other 7-day periods in your baseline. Depending on your volume of failed payments, it might take at least 30 days to average out the natural variance in your data and find meaningful insights.

Accurate measurement takes time, but it's worth the wait. When it comes to customer retention, decisions made based on incorrect data analysis can have large, compounding effects on your business.

Please feel free to contact the Churn Buster retention team for help interpreting early test results.