Experimentation

“Generally I wouldn't let anything bleed past a quarter. You can probably get some good signal in a month or less, what I would call fishing - putting bait in the water to figure out where the fish are, not necessarily getting statistically significant repeatable solutions.”

“When you're in exploratory mode, think of it as finding the right mountain to climb. And then when you're in exploitation mode, it's like focusing your resources on climbing that mountain effectively. If you do too much exploration, you can have your team feel a little bit too scattershot. And if you do too much in exploitation, it can lead to this saturation and stagnation where you're just locally maximizing a thing.”

“I think in this early stage, we're in the community gardening phase, not the factory farming phase of this technology. And so I think what people need most is curiosity and play. You should be playing with these things and trying out different things and seeing what weird things are possible now.”

“Because we don't exactly know what capabilities will even come up soon and we don't know what's going to work technically, and then we also don't know what's going to land even if it works technically, it's much more important for us to be very humble and learn a lot more empirically and just try things quickly. And the org is set up in that way to be incredibly bottoms up.”

“I reward the learning versus the outcome. If we're constantly learning, it's okay if we make bad decisions as long as we learn from them. Even in making a bad decision, we learned something about our customers or business that we can use in another context.”

“And I would encourage everyone, if you can, look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment. And I'll bet you'd be surprised how many times the metric is different than what you thought it would be after a year.”

“Ultimately you're there to create impact. But learnings is the means. If you try and focus on impact itself you might struggle. If you focus on the things you need in terms of learnings to take you step by step, that will pave the path to creating impact.”

“Jeff Bezos is fundamentally a very scientific and analytical thinker. He said, 'I need to be scientific about this. There needs to be some mechanism for me to manage such a company. So I'm going to experiment, like a scientist would, with different ideas, different hypotheses, implement them and see what works.'”