Good Experiment, Bad Experiment (opens in new tab)

(reforge.com)

25 pointsxtacy6y ago4 comments

4 comments

4 comments · 3 top-level

mwexler6y ago· 1 in thread

This is philosophically interesting, but little of this really is about _running_ a good experiment. Perhaps it's about choosing which ones to run?

Kohavi's book will probably provide much more value than this kind of abstract post. See https://experimentguide.com/ for more details.

tantalor6y ago

This bit is crucial for running a good experiment:

> Good experiments use tight exposure groups

Not all your users will have the conditions necessary for the experimental treatment.

You should only look at changes in behavior where the treatment condition was true. Otherwise your experimental effect is diluted.

But your comparison between experiment and control groups must be neutral on the condition, that is, users in the control group would have seen the treatment if they were in the experiment group (counterfactual).

daveFNbuck6y ago

> Good experiments define success up-front.

This is absolutely critical. If you're not defining success up-front, you're not running an experiment. You're just doing a staged rollout. You can use the data to craft whatever story you want for most changes.

Nzen6y ago

tl;dr former Slack director of product outlines twelve principles for a/b type experiments with features, products, whatever. (ex Good experiments involve analyzing the results as opposed to 'ship/kill?') Each principle has a couple of sentences of elaboration and calls out the opposite style to avoid.

j / k navigate · click thread line to collapse