Ask HN: A/B/C testing - something bothers me about this one.

If you want to see which they prefer, why don't you just have all three options on the page and see which one gets the most sales? Surely you can just see how much revenue is coming in from the different outlets?

It seems to me like this isn't a great use case for a/b testing. If you have all three options and one of them never gets used, stop showing that option.

You're performing the test on one large group of people, not on lots of individual people. (Does that make sense?)

It doesn't tell you what Bob as an individual likes, but it might tell you what a collection of people prefer.

It's quite important with A/B testing to set a sample size at the beginning of the experiment and stick to it. A statistician will be able to explain why.

Because you're not testing individual people it'll be tricky to disentangle the reasons for preference from the data.

You are adding an additional variable: for example "liking iTunes or not" that can't be tested like this, so you are confusing yourself as to the meaning of the results.

You are also trying to imply meaning on the "not doing something" aspect of it "does that say anything about the alternatives". This can only be true if the alternatives are available at the point of selection. Therefore, you are right this says nothing about the alternatives.

If you really wanted to see which was the preference from all three then all three choices need to be available to choose equally. At the moment by only giving one albeit random choice you are not answering anything.

To understand this process better think of the classic "google blue" test, where the colour of the links in the search results was subtly changed for random users to find out which blue worked best.