top of page

Test more ideas with smaller tests to find the rare big win

Oct 10
3 min read

Updated: Oct 11

Public-domain art plate: Audubon — Louisiana Water Thrush, Birds of America (1827-1838)
Louisiana Water Thrush, a plate from John James Audubon's The Birds of America (1827-1838).

Most of your ideas will do almost nothing. A very few will do almost everything.


The study. Eduardo Azevedo and 4 co-authors, with Microsoft, studied 1,450 A/B tests on the Bing search engine from 2013 to 2016. Each test had about 19.4 million users on average.


The finding. The results had a "fat tail". This means that rare, very large wins are much more common than a normal bell curve predicts. The top 2% of ideas made 74.8% of all the gains.

        Want to read more?

        Subscribe to plaintalkbusiness.com to keep reading this exclusive post.

        bottom of page