B2B Marketing & Sales and the Law of large numbers

In B2B marketing and sales we often have benchmarks on the probability of an outcome occurring. Often this is in the form of a typical conversion rate. For example:

· Number of impressions required per click

· Number of sales opportunities required per win

In other words, we know how many times we normally need x to happen, for y to happen.

For example, over 3 years, you might have built up a body of data showing that the typical conversion rate of landing pages on your website is 2%. Meaning that when you create a new landing page, broadly, you expect a 2% conversion rate.

But let’s imagine that the landing page appears to be performing significantly worse than usual: at what point should you change it?

This is where it’s helpful to know about the ‘Law of large numbers’: the more times something happens, the more the results will be in line with what you expect.

Here is a chart showing what happens when a coin is tossed 200 times:

(Image credit: Statistics by Jim)

Early on, the results fluctuate significantly from what you would expect (0.5). Then as the number of times the coin is flipped increases, the results settle close to 0.5.

The law of large numbers is something that we instinctively understand: we wouldn’t expect to flip a coin 4 times and be guaranteed that it would land on heads twice.

But it’s harder to intuitively answer a question like: “if when x happens, there is a 20% chance of y outcome happening as a result, what is the chance of y outcome happening if x happens 6 times?”.

For example, if you know that typically 5 sales opportunities lead to 1 win, and you’ve now had 6 opportunities in a row without a win, is this bad luck or is there something wrong with your approach?

The formula to work this out looks like this:

1 – (1 – 0.2)⁶ = 0.74

To break it down, the 0.2 represents the 1 in 5 chance of the outcome happening, and the 6 represents the number of tries. The 0.74 represents the likelihood of the outcome occurring after 6 tries (74%).

So in this situation, you can see that there is basically a 1 in 4 chance that the fact you haven’t had a win is bad luck. Which would probably suggest it’s not worth making drastic changes or going back to the drawing board yet.

As the number of opportunities increases, the likelihood that you’re suffering from bad luck decreases:

1 – (1 – 0.2)⁷ = 0.79
1 – (1 – 0.2)⁸ = 0.83
1 – (1 – 0.2)⁹ = 0.87
1 – (1 – 0.2)¹⁰ = 0.89
1 – (1 – 0.2)¹¹ = 0.91
1 – (1 – 0.2)¹² = 0.93
1 – (1 – 0.2)¹³ = 0.95
1 – (1 – 0.2)¹⁴ = 0.96
1 – (1 – 0.2)¹⁵ = 0.96

To the extent that if you expect 1 win from every 5 sales opportunities, and you’ve had 15 opportunities without a win, there is just a 4% chance that this is due to bad luck.

Here is another example at a different end of the spectrum, in terms of probability: the likelihood that someone will click on some kind of paid digital ad. If we imagine that for a certain ad format, you typically see a 2% click through rate, what would the relationship be between the number of impressions and likelihood that someone would have clicked?

After the first 50 impressions, there’s only a 64% chance that someone would have clicked:

1 – (1 – 0.02)⁵⁰ = 0.64

In the above sum, 0.02 represents the 2% probability of a click happening, the 50 represents the number of impressions, and 0.64 is the likelihood of the outcome (a click) happening.

Here is what happens as the number of impressions increases:

1 – (1 – 0.02)⁶⁰ = 0.70
1 – (1 – 0.02)⁷⁰ = 0.76
1 – (1 – 0.02)⁸⁰ = 0.80
1 – (1 – 0.02)⁹⁰ = 0.84
1 – (1 – 0.02)¹⁰⁰ = 0.87
1 – (1 – 0.02)¹¹⁰ = 0.89
1 – (1 – 0.02)¹²⁰ = 0.91
1 – (1 – 0.02)¹³⁰ = 0.93
1 – (1 – 0.02)¹⁴⁰ = 0.94
1 – (1 – 0.02)¹⁵⁰ = 0.95

If you run an ad and you expect a 2% CTR based on past performance, and you don’t get the first click until after 150 impressions, the chances that this is bad luck and that over time the ad will perform as well as you expect is 5%. In other words, you should monitor subsequent performance closely, and probably start to think about making some changes to the creative and / or targeting.

You can see that despite the probabilities of the two outcomes being an order of magnitude apart (5 sales opportunities usually needed per win; 50 impressions usually needed per click), the distribution is very similar:

· After 5 opportunities, there’s a 67% chance of a win; after 50 impressions, there’s a 64% chance of a click.

· After 10 opportunities, there’s a 89% chance of a win; after 100 impressions, there’s a 87% chance of a click.

· After 15 opportunities, there’s a 96% chance of a win; after 150 impressions, there’s a 95% chance of a click.

The only difference is that where the probability of an outcome occurring is low, you need relatively more attempts to make it happen than if the probability is high – but there’s not a huge difference.

What are the implications of the law of large numbers for those in B2B GTM roles?

The law of large numbers strengthens the argument for a measured long-termism. You can see that mathematically, even when you have made enough attempts that you would expect an outcome to happen, there’s still only about a two thirds chance that the outcome will actually happen. Ie:

· 33.33% chance of something happening, 3 tries = 70% chance of the thing happening

· 20% chance of something happening, 5 tries = 67% chance of the thing happening

· 10% chance of something happening, 10 tries = 65% chance of the thing happening

· 5% chance of something happening, 20 tries = 64% chance of the thing happening

· 2% chance of something happening, 50 tries = 64% chance of the thing happening

· 1% chance of something happening, 100 tries = 63% chance of the thing happening

But there should be limits to this, and a sensible rule of thumb is probably when you get to 3x the number of tries that you would theoretically need to happen to lead to an outcome, there’ll be a >95% chance of the thing happening, so if it hasn’t happened, something is almost certainly wrong (or different). Ie:

· 33.33% chance of something happening, 9 tries = 97% chance of the thing happening

· 20% chance of something happening, 15 tries = 96% chance of the thing happening

· 10% chance of something happening, 30 tries = 96% chance of the thing happening

· 5% chance of something happening, 60 tries = 95% chance of the thing happening

· 2% chance of something happening, 150 tries = 95% chance of the thing happening

· 1% chance of something happening, 300 tries = 95% chance of the thing happening

B2B Marketing & Sales and the Law of large numbers

What are the implications of the law of large numbers for those in B2B GTM roles?

Richard Costeloe | richard@costeloe.io