When he was still at Etsy in 2012, McKinley gave a talk about "Design for Continuous Experimentation" that is still one of my favorite tech talks that I've had the pleasure to attend:
http://mcfunley.com/design-for-continuous-experimentation
I skimmed the slides linked in this current talk and it didn't seem to mention the test-driven stories he brought up in his 2012 talk, the highlights which were:
1. Some of Etsy's employees were also power users when it came to shopping Etsy, and they would open up product pages in new tabs while skimming the listings. So they assumed that Etsy's normal users would prefer new tab opening to be the default upon clicking a product link. The results were an "epic failure", with A/B testing showing that "70% more people gave up and left the site after getting a new tab".
2. After spending months implementing infinite scroll for search results, which they just assumed would be a net win for user engagement, they found that it negatively impacted the number of items users bought via searching. To my knowledge, this is the largest example in which a company has rejected infinite scroll based on A/B testing. I mentioned this on my blog, and that post is by far my most visited blog post...lots of people looking for reasons to not implement infinite scroll I guess...
