Everything you ever wanted to know about video codecs.

David Field puts the nuts and bolts of the most popular video codecs under the microscope; then pits the codecs against eachother.

Back in the Paleolithic age – just after man had discovered stone tools and how to load CD-ROMs into caddies – stunning pixelated video played back from our mighty 486s. In short, we have video codecs to thank for that and everything that evolved from it.

In the many years since that early time, video quality has increased thanks to ever-expanding storage capacities and processing power – the two building blocks of high quality digital video. At the same time we also developed a taste for better quality from physical media and video, with a bitrate small enough to be streamed over our measly bandwidth limits.

All codecs (a contraction of code/decode), whether they be for video, audio or some other type of data, exist to change a long bit series into a shorter bit series and back again, or at least into something that closely resembles the original. This magic process is called ‘compression’ (or ‘decompression’ in the latter case).

It’s taken its sweet time, but high definition video is finally here in the form of HD-DVD and Blu-ray. Although they’re both a huge jump up from DVDs, there’s very little difference between the two formats: They provide 1920 x 1080 (or 1080p) resolution, have considerably more space and higher transfer speeds than DVDs and both use the same codecs to decrease the size of the original video so it can be shoehorned onto the disc.

If you are wondering why optical discs still need to use compression to hold a movie when they can now store anywhere from 15 to 50GB and transfer data at a minimum of 36Mb/s, ask this question instead: How much space do you need for true high definition video?

Movies in space
Here’s a fun and jaw-dropping fact about digital video: At a post-production house, an uncompressed two-hour film in digital cinema resolution and quality will clock in at about 12 terabytes, less 9 to 18 gigabytes for the accompanying 16 channels of 48 or 96kHz audio.

Some of this can be explained away when you consider digital cinema’s 4096 x 2160 (or 4K) resolution, but the data rate is still monstrous – far too high for commercial cinemas to read and project, let alone store. This is why digital films are perfectly – or ‘losslessly’ – compressed to no more than 500GB, resulting in visually identical footage that requires a bit of decoding processor muscle.

Even after you account for the drop in resolution from 4K to 1080p, it’s still clear that no consumer format has enough space to deliver this kind of perfectly reproduced image quality. And that’s just the film – we haven’t even thought about the space needed for the extra features we’ve come to expect from our discs yet. This is where ‘lossy’ codecs come into play. They’re much more complex than lossless codecs, and we’ll examine them after we’ve looked at the basics of compression.

Slice and dice
Compression in general exploits patterns that exist in data sequences. If lengthy patterns can be replaced with a more concise placeholder, the sequence will become smaller without any information being lost.

If you’re thinking of ZIP and RAR files right now, you’re on the right track. Conceptually, lossless video codecs resemble a RAR archiver designed for video. There’s a hard limit on how much data you can remove, which is governed by the laws of information entropy. To sidestep these hard mathematical limits, we have to be prepared to sacrifice some data first.

Lossy codecs not only use straight compression, they exploit the way we perceive video and the way it is constructed. They predict what will happen from one frame to the next, then deliberately leave data out, then take a mathematical guess at what to fill in the blank spaces with later on. It sounds scary, but in practice all they do is throw away information from the picture that is hard for your eyes to perceive and easy for your brain to miss.

What’s in a frame?
At the most basic level, a video is just a series of images (in this case, frames) displayed one after the other at a constant rate. The rate determines the look and smoothness of the video; 24fps for a movie, 25fps for PAL and 29.97fps for NTSC.

Like a bitmap, every frame of video is made up of a grid of pixels arranged along the X and Y axes. Think of a video as thousands of frames positioned sequentially down the Z axis that are displayed one at a time, every 1/24, 1/25 or 1/30 of a second. Video sequences differ slightly from computer images as the pixels aren’t made up of RGB values – they are stored in a YCrCb colour space. This meshes a full resolution greyscale (luma, or Y) layer with two layers of colour (chroma, or the red component Cr and the blue component Cb).

Generally, video contains a full resolution luma image and half resolution chroma images, which are scaled up and layered over the full resolution luma image. This effectively colours in four pixels of a detailed greyscale image with one pixel of colour. The technique is called chroma subsampling, and it works because our eyes are much more sensitive to the brightness of a signal than its colour. This little biological quirk not only came in handy when colour was added to the first black and white TV signals, but it’s also used to reduce the bandwidth of a video stream.

All MPEG formats use 4:2:0 chroma subsampling to halve the video bandwith of a high quality master source. Not only is it very hard to tell the difference between a half bandwidth 4:2:0 signal and a full bandwidth 4:4:4 signal, material encoded with any MPEG codec (which encompasses most of what we watch) is subsampled at 4:2:0. You literally won’t know what you’re missing unless you see a film shot and projected digitally with fearsome gear like the Arriflex D20 and Sony’s Cinealta projector.

And yes, we’re purposely sweeping the whole film thing under the rug.