Data Driven Product Development: Experimentation and A/B Testing (opens in new tab)

They've got a ton of really great tests. Seeing the actual change in revenue is a crazy - my favorite test was 4.2, where they added some ads to the homepage and decreased revenue.

Overall an excellent paper that both discusses specific, relevant examples as well as critical aspects of experiment design.

The fact that we could get design so wrong if we just rely on intuition surprises me everytime I re-learn it. The discussion on the cultural challenges and the things can go wrong with A/B testing were fascinating. This is wisdom from people that are battle scarred.