An introduction to champion-challenger

In the second episode of How to Lend Money to Strangers I talked to Graham Whitley about why to build a test-and-learn culture in support of any data-driven initiative; and it came up again in this week’s chat with Georg Steiger so it is probably worth a more detailed article – which is this one here.

Now, I am guilty of using the terms test-and-learn and champion-challenger inter-changeably, which can be confusing, but rest assured they do mean the same thing, which is essentially the process of repeatedly pitting the status quo against our next best idea (in a controlled manner).

Test and learn is a very descriptive way of saying this, champion-challenger a little more poetic – taking it spiration from boxing, where the current champion must always stand ready to defend their crown against the top-rated challenger. When the champion wins, they remain champion and a new challenger is found, when the challenger wins, they become the champion and a new challenger is found.

And likewise with a strategy: if it is our current opinion that offering a three-month interest-free balance transfer period is the best way to bring on new customers that’s our champion, and we should be making it compete against some alternatives, perhaps a six-month offer will actually attract sufficiently more customers to compensate for the lost revenue. Or perhaps it won’t. That’s what we’ll learn, by testing.

Now, I once worked for a major international bank that said its strategies included champion-challenger components, but the fact is that if for three years the same ‘champion’ has been competing with the same ‘challenger’ win or lose, then you’re doing it wrong – that’s a money-grabbing exhibition fight with a YouTuber, or in-built, random, inefficiency.

To actually be champion-challenger, it has to drive change, the challenger must be allowed to ascend to the throne. But, and this is perhaps a nuance better captured by the test-and-learn terminology, this has to be done in a controlled, scientific approach.

We don’t actually make a series of complete substitutions of one strategy for another, instead, we’re always running at least two strategies side-by-side. In examples, we often talk about 80% going down the champion stream and 20% going down the challenger but in reality, we set the split based on the degree of risk involved, how much of a variation we’d expect to see, the size of the portfolio and the sophistication of the team managing it. This is all based on statistical sampling theories, and I won’t go into them further in this article.

Instead, let’s return to our example of a marketing analyst looking for ways to accelerate portfolio growth with a balance transfer campaign. The analyst might have seen that several competitors are offering a three-month interest-free period while they are not, and as such, they may want to the impact of following suit but also of going in with an industry-leading six-month offer.

Both of these offers will involve a guaranteed loss of revenue in exchange for an expected gain in response rates, so before going out with a large advertising campaign the analyst must decide what the most profitable approach is – the traditional balance transfer with no interest-free period or one of the two discounted offers.

The first step is to randomly assign prospects to one of three groups: 80% might be assigned to the control group and sent the undiscounted (champion) offer, while the other two might be split with 15% getting the three-month market-matching offer and 5% getting the marketing leading six months free.

From that point on, response rates must be tracked and compared to pre-agreed metrics (ideally using a profit model) so that, at the end of the test, we can calculate if the loss of interest income was sufficiently recovered by increased business through the door.

In my example, this was the middle offer, which would then be sent to a wider audience as the new champion. But, vitally, we don’t stop there. A discount can take many forms, perhaps the market would actually respond better to a welcome gift of a similar value or a cash-back rebate… so a new challenger must be designed and implemented to restart the process.

Or, in other words, we plan-implement-measure-adopt:

Plan

The planning stage should start with an awareness of the problem or opportunity at hand.  This awareness usually evolves out of an analysis of the organisation’s data, but could also come from information acquired from third parties or even just a good old-fashioned hunch. And once the business problem has been understood, it should be translated into a hypothesis that can be tested.

Ideally, this would focus on the optimisation of one or more aspects of that business’s profit model.   Customers with revolving balances are more profitable in the long run, and a short-term discount that attracts more of them will ultimately pay dividends becomes the net present value of a campaign will be increased if a discount of x% leads to an increase of at least y% in response rates.

A test must then be designed to conclusively prove or disprove this hypothesis.  To do this well, you’ll need to work through the formulas beforehand to ensure that the data points you need to make the maths work are being gathered.

Once the key test measures have been established, the experimenter must select their inclusion and exclusion criteria.  These criteria limit the test to a specific population of interest.  In the example begun above, the experimenter may choose to include only those potential customers with a clean credit record and to exclude any potential customers who have received a previous advertisement from their company within the last three months.

The final step in the planning process is to ensure that the results of the test will be statistically significant.  This is achieved, primarily, by including a sufficiently large population of customers in the test.  Bear in mind however that, since the test-and-learn technique identifies the single most profitable strategy, candidates exposed to the failing challenger strategies are value-destroying.  For this reason, the test should include only as many candidates as are necessary to ensure statistical significance so that once the champion is identified, there are more uninfluenced targets still available.  This also limits the costs and risk of the test. 

Implement

Implementing a well-planned test should then be a simple process.  The qualifying candidates are randomly assigned to either the control group or one of the test groups – it is possible to run multiple tests at once, where for example we test both the interest-free balance transfer offer and the impact of different coloured envelopes, and in such cases, a customer might be in more than one group, but great care is needed in such cases.

Importantly, everything that is not being tested must be identical across all offers. So if we’re testing the impact of the interest-free period we need to make sure that, for example, the wording of the offer should be the same regardless of the offer being made.

Similarly, assigning candidates to groups randomly ensures that the groups are effectively identical.  Unless the groups are identical, any measured difference in performance might be attributable to differences in the groups’ composition rather than to any difference in the strategies applied.

Measure

The test is then monitored to identify any variation in performance between the test and control groups.  As I mentioned above, this should not be done in an open-ended period, but rather for a pre-set time period informed by the aforementioned profit model.

Since all other factors are kept constant, superior performance in the test group must be indicative of a superior strategy.  That said, the less confident you are in your profit model, the more leeway you could leave here.

Adopt

Now is the time for the challenger to take over. With immediate effect, all qualifying candidates who were not mailed during the test should receive the offer with the three-month interest-free period.  This strategy will remain in force until a future test identifies an even more profitable marketing strategy.

Thus creating a circular pattern that should always bring the best available strategy to the fore.