I chose the clip based on what Dark_Shikari (x264 developer) had to say about it[1]:
It shouldn't bias too heavily towards any one encoder like many of the other standard test clips will:
a. It's relatively high motion, so it won't bias heavily against encoders without B-frames or qpel (as, say, mobcal does).
b. It's not so high motion that it would cripple video formats that don't support motion vectors longer than 16 pixels (e.g. Theora).
c. It's not something that benefits an unreasonably large amount from some of x264's algorithms (which is why I picked this and not parkrun).
[1] http://forum.doom9.org/showthread.php?t=154430
I could have done multiple test encodes, sure, but the problem in this case was that downloading several gigabytes of raw source material isn't exactly instant. And even if I tested with multiple clips, I doubt the conclusion would be that much different.
And sure, perhaps you'd get the same result on other clips. Over high-profile H264 the only obvious format feature that come to mind that could really let VP8 get ahead are the 'truemotion' intra-predictor and creative use of the synthetic reference frame (though I suppose the vp8 developers might have other suggestions) and I'd expect those features to only be big wins on a small number of clips so it wouldn't be hard to miss the cases where VP8 really shines over high profile h264.
But you (or I) could have said that without doing the test at all, and there would be 100% fewer clueless people going around claiming that something was proven here that wasn't. Your opinion (or mine) is a fine thing, but it's not proper to launder an opinion as fact by dressing it up in an inadequate test.
Given that people do hundreds of test encodes when they actually use things like x264, I think that if you want to say anything general about these encoders you have to do more than one comparison.
"The first in-depth technical analysis of VP8" http://x264dev.multimedia.cx/archives/377
*VP8 doesn't have B frames, similar to the h.264 baseline profile.
Apart from being one of the developers of open source GPL licenced x264 encoder, he and the other developers are also licencing it for proprietary use (nothing wrong with that, I think it's a great way of making money out of open source development). This means that a open source royalty free codec like VP8 (and later VP9) could cut into their revenue stream.
I'm not saying that this affected the outcome of his test, but it's still context. Also his test back then was also criticized for only using one clip, here's another test with a larger set of comparisons:
http://qpsnr.youlink.org/vp8_x264/VP8_vs_x264.html
I can't vouch for the validity of this anymore than the validity of any other test.
It's important to explore the space a bit for discussion's sake.
I really do hope that VP9 can offer the quality of H.265. VP8 just wasn't good enough to compete against H.264.
I am okay with the minor loss in quality, just for the ease of managing a lot of videos and a lot of disk space.
The standard itself doesn't specify how an encoder behaves. That leaves a lot of room for innovation on encoders. End of the day, the qp/bitrate ratio you get is encoder implementation dependent and not sometime that's specific to the standard. This is the part where most of the patents exist.
The ffmpeg guys wrote a VP8 decoder with 1400 lines of C because of how much they were able to reuse. From: http://blogs.gnome.org/rbultje/2010/06/27/googles-vp8-video-...
Of course as everything switches to android this will change completely by the end of the decade, but for streaming in 2013, h264 is the better investment.
Once they move to h265 vs vp9 the point is moot but that won't be common for at least five years. The encoding requirements for those streams will also be significant.
I wonder why couldn't they allow software usage ( Encoder and Decoder ) to be free. And only collect royalty on Hardware, and Industry Production use.
{currently at the bottom of page 4 when by pts it should still be in the top 10)
