I stopped at the first misuse of statistics, but by that point, they had taken out informative links from their web site and turned it into a landing page. Frankly, a 5% change in conversion rate is in the noise compared to the effect of targeted advertising (and I have no faith that these sloppy reporters actually did a properly controlled experiment).
Edit: Downmodded for asking for real math over marketing math?
The old conversion rate was 24.4%. The new one was 29.6%.
(29.6 - 24.4) / 24.4 = .213 = 21.3%.
Is it misleading because their number of conversions is so low? (Not trying to be snarky, I have no idea)
Reporting "a percent of a percent" is downright misleading because it is entirely dependent on the starting baseline and also because conversion rates have high natural variance. Imagine an improvement from 2% conversion (for a truly awful site) to 2.6% (for a site just as awful). That "30% improvement" just isn't. If they were intellectually honest and called it a "0.6% improvement," anyone would be able to see that the claimed improvement is well within stddev.
You appear to know words associated with statistics, but treat them as if they were magic incantations utterly disassociated from actual math.
I'll bite: what is the standard deviation in that example. I'm looking for a two part answer: a) a number, b) what the number is measuring.
Let me be intellectually honest with you: there is no basis for assuming that an improvement from 2.0% conversion to 2.6% conversion is not statistically significant.
You need sample sizes to even attempt to do the math. For example, try doing a chi-squared test on 100,000 people converting at 2% versus 5,000 converting at 2.6%.
I'm thinking you'll find that you reject the "just as awful" hypothesis with over 99% certainty.
The number that matters is "how many more users are signing up since we changed"? The answer is NOT 5.2%.
Example: I have a 1% conversion rate on 100,000 visitors. 1,000 customers! Yay! I change something and I get it up to 2% on the next 100,000 visitors. 2,000 customers.
Which is a better description: "I doubled my conversion rate" or "I increased my conversion rate by 1%?" I'd go with the former.
I think it's just a matter of using confusing labels to refer to the percentages/improvements.
In other words, you don't really care about the second derivative (the rate at which the conversion rate) is improving as much as you care about the initial rate.
To be honest you don't have to use Google Website Optimizer to notice that.
Example: I'm releasing my Rails A/B testing framework this Sunday, with a case study writeup taken from my site. My site currently allows people to defer signup for the trial by means of a guest login. That sort of option is popular here. I'm told it increases usability and user engagement, right?
Without ruining the surprise: that is a testable hypothesis. ;)
