Edit: As pointed out by Hupo, who did a significantly better test than the mere “for fun” one posted here, it’s apparent that H.264 still leads VP8 in terms of quality. And, even more interesting are recent developments in H.265 and VP9, also mentioned in his post, due to the fact H.265 has been approved by the ITU, essentially advancing it a step further than VP9 in terms of potential hardware support.
Given that H.265 and VP9 are right around the corner, I felt this would be a good time to get a refresher on VP8′s progress. As tested back in August 2010 on this blog, H.264 turned out to be superior. The test was also conducted on H.264′s Baseline Profile because it did made sens back then to encode web video in High Profile. Nowadays however, thanks to better CPUs and widespread hardware acceleration, we can safely encode H.264 in High Profile for web videos, which also means we can extract more quality out of it, which is a good thing. VP8′s encoding library has also significantly improved, giving us a very competitive codec even with H.264 High Profile.
The encoding was done on a video clip which has some grain, resulting in a more difficult encoding situation. The following encoders and specs have been used:
H.264
Encoder: x264 rev. 2208
Quality: Placebo High Profile in XMedia Recode
Bitrate: 2 pass @ 2 Mb/s
Resolution: 960 x 540
VP8
Encoder: libvpx 1.1.0
Quality: Best in XMedia Recode
Bitrate: 2 pass @ 2 Mb/s
Resolution: 960 x 540
Results
Source

VP8

H.264

Details
Source

VP8

H.264

Winner
As you can see, this time around however, VP8 wins (see Hupo’s counter-post for a better comparison, where H.264 wins). H.264 doesn’t look bad in practice, but VP8 manages to retain a more accurate encoding of the image. If you do a back-and-forth comparison (simply open the images in new tabs (Open image in new tab)), you’ll see what I’m talking about.
The upcoming H.265 claims to be twice as efficient as H.264, which is mind boggling if it turns out to be true, and definitely gives VP9 and Google quite the challenge. However, contrary to many’s belief, Google was able to push in some significant improvements in its VP8 encoders, enough to make it better than H.264, so there is still hope, in my opinion, that VP9 can beat H.265.
