Library

A field test with about 1 million app users found that clicks caused only about 34% of the effect of a push message. Learn how to measure the other 66%.

In a 7-day test with 1,100 Google users, search with only AI answers decreased satisfaction, trust and the number of searches. Keep the normal search results next to your AI answers.

A review of 50 experiments with 5,036 people found that, on average, more choice did not decrease purchases. The effect was large in some studies. In other studies, there was no effect.

A trial with 682 colonoscopy patients found that patients remembered a slightly longer procedure with a gentler end as less painful. These patients were also more likely to return.

Gym visits, searches for "diet" and goal sign-ups all increased after fresh-start dates. Fresh-start dates are dates such as a new week, a new month and a birthday.

In a GitHub test with 95 developers, the group with GitHub Copilot finished a coding task 55.8% faster than the group without it.

Harvard tests on travel and dating sites found that people often liked a slower search more when the site showed the work that it did. When people saw the effort, the result felt more valuable.

At Bing, a small change to the ad headlines increased revenue by 12%. Nobody tested the idea for months, because the team thought that it was not important.

A study of 7,752 gym members found that people on a monthly plan paid more than $70 a month and went 4.3 times. They paid about $17 per visit when a $10 visit pass was available.

In a Harvard test, people who built an IKEA box themselves would pay 63% more for it. The same box built by someone else was worth less to them.

When a large US company enrolled new staff in its retirement plan by default, the share in the plan went from 37% to 86%. Most people also kept the default savings rate.

In an online test, 82% of people stayed organ donors when that was the default. Only 42% chose it when they had to opt in.

A 21-month Pandora experiment with almost 35 million listeners found that each extra ad per hour cut listening time by about 2%. A short test can miss most of this cost.

A field test with 8,966 telecom customers found that a chatbot that disclosed human help got longer messages. The authors estimate that human work cost 38% to 70% more.

A Spotify test with 852,937 users found that personal podcast picks raised listening by 28.90%. But the mix of podcasts for each user became less varied.

A field experiment with 64,147 phone customers found that calls to suggest a better plan raised churn from 6.4% to 10.0%. The calls made customers think about leaving.

A field experiment with 337,724 software users found that a 7-day free trial beat a 30-day trial. Subscriptions went up by 5.59% and revenue by 7.92%.

In a test with 1,963 US adults, mild dark patterns more than doubled sign-ups for a doubtful service. Aggressive dark patterns almost quadrupled sign-ups, but made people angry.

A study of 1,450 Bing experiments found that the top 2% of ideas made 74.8% of the gains. This result favors many small tests over a few large tests.

A study of 35,918 startups found that firms with A/B testing tools got more page views and launched more products. They also stopped bad ideas faster.