Uh... How did you choose those cutoffs? Looks like you have a significant chance of making the wrong choice.
Also:
> Once enough data is collected to start making conclusions (1000 sends per variation)
You should check out the Bayesian solution to the Multi-Armed Bandit problem. It's very close to what you are doing, but makes decisions much faster than you do because it isn't deciding to turn off a variation, merely to scale it down.
We considered a weighted decision approach but 1) were turned off by posts like http://visualwebsiteoptimizer.com/split-testing-blog/multi-a... and 2) wanted to keep moving parts to a minimum for V1.
Any thoughts?
It's a good v1 for sure, congrats!
I would ignore any non-baysian MAB posts out there. The formulation used by other approaches is one that considers an infinite number of repeated trials, which is basically an insane assumption. Epsilon greedy and UCB1 aren't optimal except with that assumption.
You should check out:
- http://www.economics.uci.edu/~ivan/asmb.874.pdf
- https://www.youtube.com/watch?v=vz3D36VXefI
- http://www.cs.cmu.edu/~deepay/mywww/papers/nips08-mortal.pdf (good benchmarks)
FWIW you are doing a weighted decision approach, it's just that you've constrained your weights to be either 0 or 1...
Cheers,
David
What that means is that the changed version tends to win the test, but then may or may not perform well. The flip side of this is that if you have a choice, have multiple variations on the same email that you rotate between so that people don't get too used to your emails.
Neural network tutorial of the same site is pretty cool too.