> I didn't say hypothesis test, I said decision rule.
I did not say that you said hypothesis test.
> If you visually inspect the posterior, your are implicitly pulling out of your ass an unknown "threshold of visual similarity".
Yes, but you are "implicitly pulling a number out of your ass" based on a lot more information. When you ask somebody to come up with a mechanical decision rule before seeing the posterior, it's unlikely that you will get as good a decision as when you just show them the posterior.
> That's incorrect. From the article: "To begin we will choose a Beta distribution prior." The computational intensiveness is not caused by the choice of prior, it's caused by the need to evaluate an integral over the joint posterior.
Ah, I was confused because they are specifying the prior in terms of a mean and standard deviation. That is a very weird way to represent a beta distribution.
> The computational intensiveness is not caused by the choice of prior, it's caused by the need to evaluate an integral over the joint posterior.
I see, they are computing expected_value(max(ctr[A]-ctr[B], 0.0)). That is still weird though. What you want to know is if it's worth it to run the test another time. So you want to compare E(final conversion rate if stop now) with E(final conversion rate if run another time), and if the latter is not much greater than the former you stop the test. Both of those have a closed form. Even better would be to compare E(final conversion rate if stop now) and E(final conversion rate if we test A) and E(final conversion rate if we test B). Then you would also automatically decide the best version to show (e.g. if the uncertainty about A is small and the uncertainty about B is big, you'll show B).
> A Dirichlet prior is also not what you'd use for more than 2 alternatives
Hm? Lets say you have a free plan, basic plan, and enterprise plan. This is a very common scenario in practice. A dirichlet prior would be the natural thing to use here, IMO.
