Away, My Unbelieving Fear
Every time I convince myself that no-one could say anything nuttier than the last nutty thing I read by Seth Godin, along he comes and trumps himself. In a post today, ‘Getting to scale‘, about the differences between mass marketers and direct marketers, Mr. Godin wrote: “The direct marketer, on the other hand, must get it right in the small. That pitch letter can be tested on 100 houses and if it gets a 2% response rate, then it can be mailed to 100,000 houses with confidence.”
If a letter gets a 2% response rate from 100 recipients, it can be mailed to 100,000 houses “with confidence“? With confidence in what, exactly? A 2% response rate is just two houses.
Let’s say that mailing those letters (including printing and postage) costs us $1 each. That mailing will cost $100,000. If 2% respond, we’ll need to get $50 from each (on average) just to cover the cost of our mailing.
Let’s say that our product gets us more than $50 in profit per sale, so our mailing will be profitable if the response rate is 2% (2000 homes), but not if it’s 1.5% (1500 homes) or 1.0% (1000 homes). How confident are you now in authorizing that mailing?
If you randomly pick a number from 1 to 100 and then ask me to guess it, I’ll be right (on average) 1% of the time. If we play that game 100 times, I’ll be right (again, on average) just once (which is probably why nobody plays this silly game). But I might be right twice. Since each guess has a 1% chance of being right, I might be right on Game #11 and then, just by luck, be right again on Game #86. Winning on Game #11 doesn’t prevent me from being right for another 100 games. In every game independently I have a 1% chance of being right.
So let’s say that you mail 100 homes and 2 houses respond. Are you confident that (at least) 2000 homes will respond to your larger mailing? Of course not. The chance any individual house responds might only be 1%. (It might be 1.5%, but it’s physically impossible to see a 1.5% response rate from 100 houses.)
It’s fairly easy to show that, if you see a 2% response rate from 100 homes, the chance that only 1500 homes will respond to the larger mailing is more than 90% of the chance that 2000 homes will respond. And the chance that only 1000 homes will respond is about 70% of the chance that 2000 homes will respond.
With a sample size of only 100 homes, we simply cannot have a high degree of confidence that our response rate will be near 2% just because that’s the number we saw this one time.