That iphome paper used a random git checkout from the day the bitstream was frozen, and claimed it was an official release, which always struck me as somwhere on the stupid/devious spectrum.

The first actual VP9 release was about 6 months later and even then I dont think they'd done much tuning for speed. Some parts of Google get criticised for building open source products privately, but others that develop in the open have their openness abused.

I ran some experiments to investigate the discrepancy.

The largest differences were caused by:

1. A variable quantiser being used for HEVC, but not for VP9 (as you described)

2. Keyframes being forced every 30 frames for VP9 in the first paper

HEVC also had I frames added every 30 frames, but these were not IDR frames, meaning that B frames were allowed to use the information from the I frame in HEVC.

However, in VP9, true keyframes were forced every 30 frames. The way VP9 works this meant that every 30 frames it encoded both a new Intra frame, and a new golden frame.

Making both codecs use a true fixed quantizer and removing the forced intra frames made the results more like Google's own paper.

I guess the moral is to not force frequent keyframes when encoding with VP9.

At some point comparing highly efficient codecs with metrics like PSNR becomes meaningless, I'm not sure why so much emphasis is still put on it. Verification against human preference is the way forward.

I believe methods like A/B preference testing have been done for audio codecs for a few decades, idk why video didn't catch up.

In this respect I look forward to the work done by the Daala guys, seeing how they come from the highly successful OPUS audio codec seem to be very mindful of perceptual optimization.

Thanks. Even with bias, Google results looks way more realistic to me than "100x slower and worse than H.264".