Question 1

What are the key takeaways from Ronny Kohavi's episode on Lenny's Podcast?

Accepted Answer

1. Most ideas fail: 66-92% of experiments do not improve key metrics, and every team that starts experimenting is humbled by how bad humans are at predicting outcomes. 2. Define your Overall Evaluation Criterion (OEC) around lifetime value, not short-term revenue - optimizing for money alone leads to user-hostile decisions that destroy long-term growth. 3. Avoid big-bang redesigns: bundle 17 changes together and most will be negative. Instead, use one-factor-at-a-time (OFAT) to learn what works and iterate incrementally. 4. Build institutional memory by documenting surprising experiment results and holding quarterly reviews - organizations that fail to summarize learnings repeat the same mistakes. 5. Trust is the foundation of experimentation culture: if your platform produces unreliable results (inflated false positives, sample ratio mismatches), teams will lose faith in data-driven decision making entirely.

Question 2

What does Ronny Kohavi say about experimentation and data-driven?

Accepted Answer

"That simple idea increased revenue by about 12%. This thing was worth $100 million at the time when Bing was a lot smaller. And the key thing is it didn't hurt the user metrics. It's very easy to increase revenue by doing theatrics. Displaying more ads is a trivial way to raise revenue, but it hurts the user experience." - Ronny Kohavi "We are often humbled by how bad we are at predicting the outcome of experiments. This was an example of a tiny change that was the best revenue generating idea in Bing's history, and we didn't rate it properly. Nobody gave this the priority that in hindsight, it deserves." - Ronny Kohavi "Of these experiments, 92% failed to improve the metric that we were trying to move. So only 8% of our ideas actually were successful at moving the key metrics. Very humbling. I know that every group that starts to run experiments, they always start off by thinking that somehow, they're different. And their success rate's going to be much, much higher, and they're all humbled." - Ronny Kohavi

Question 3

What are Ronny Kohavi's views on experimentation, data-driven, A/B-testing?

Accepted Answer

Ronny Kohavi: "That simple idea increased revenue by about 12%. This thing was worth $100 million at the time when Bing was a lot smaller. And the key thing is it didn't hurt the user metrics. It's very easy to increase revenue by doing theatrics. Displaying more ads is a trivial way to raise revenue, but it hurts the user experience." Ronny Kohavi: "Of these experiments, 92% failed to improve the metric that we were trying to move. So only 8% of our ideas actually were successful at moving the key metrics. Very humbling. I know that every group that starts to run experiments, they always start off by thinking that somehow, they're different. And their success rate's going to be much, much higher, and they're all humbled."

Ronny Kohavi

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

TRANSCRIPT

Ronny Kohavi

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Ronny Kohavi

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

TRANSCRIPT

Ronny Kohavi

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)