Everything you ever wanted to know about video codecs.

The Moving Picture Experts Group, or MPEG, is easily the biggest player in the lossy video codec game. It’s been around for the last 18 years, has a lot of imitators and a legacy of codecs used in all facets of the industry. MPEG-1 drove VCDs. MPEG-2 drove DVDs. If it hasn’t already, MP3 should win the award for the most publicly recognisable IT acronym in the world.

The MPEG-4 specification is more of a container standard than a codec. It’s comprised of 23 parts, not all of which are dedicated to video. Part 2 is the standard video codec – the same compression scheme that both DivX and XviD are based on. Part 10 is a more advanced video codec – known formally as H.264/MPEG-4 AVC. H.264 is one of the three codecs that drives both HD-DVD and Blu-ray.

Aside from throwing away half the colour information from a high quality master to save space, codecs (MPEG-based and others) use other techniques to approximate data from a video file before compressing it. Here are the most common techniques:

Spatial compression
Using a technique similar to that of JPEG to compress still images, the size of a single image can be reduced. This does reduce the image size, but not greatly, mostly relegating spatial compression to the domain of lossless codecs. It still serves a fairly major purpose in the process of lossy encoding though, as you’re about to find out.

Temporal compression
Apart from some epilepsy-inducing Japanese music clips, all video is continuous. From one frame to the next, most of the detail stays the same: Think of the background, the walls or, if he’s been cast, Keanu Reeves’ expression. Why waste precious storage space to store tens of thousands of images of Keanu Reeves if he looks exactly the same in 90 percent of them?

This is exactly what temporal compression does. A keyframe (known in the MPEG world as an I-frame) that contains all the important data is drawn, and in the following frames whatever doesn’t change is left out and filled in with data from the keyframe. Temporal compression literally means compression in time (as opposed to spatial compression, which is literally compression in space) and it can slice away massive amounts of data.

Keyframes are reference points in a movie made of a spatially compressed image that is used by subsequent frames to fill in parts of the image that stay the same from frame to frame, letting them be replicated instead of repeatedly encoded. As such, the best position for a keyframe in a sequence is when the sequence suddenly changes, such as a cut between a dim indoor game of cards and an outdoor shot of a car about to run into the house where said game is being played.

Motion compensation
Temporal compression has some limits, in that it doesn’t deal with camera pans very well. When an object moves from one location in a frame to another, the scene has to be updated even if the element itself doesn’t change. MPEG-2 introduced motion compensation, which uses vectors to push static elements around a frame in time instead of redrawing them.

Motion compensation and temporal compression are the keys to reducing data, and the reason why when you jump to a point in a video on an old machine it takes a second to play. The codec has to render the keyframe first then recreate all the frames between it, as well as the frame you requested.

Discrete Cosine Transformation
Although you’ll need a degree in computer science to fully understand it, the DCT is the heart of a lossy encoder. In short, it’s a process of approximating numbers with a degree of accuracy. During an encode, every number in a series is simply halved and the remainders thrown away. When it comes time to decode, they are doubled, resulting in values that closely resemble the original.

The technique works better on bright areas than dark ones, which is why compressed blacks don’t always look as smooth as their non-compressed counterparts, although codecs can control this.

Macroblocks
It’s inefficient, especially when video is stored in a YCrCb colour space, to refer to each pixel individually. Macroblocks are groups of (usually) 8 x 8 pixels that have their values converted to a formula. Because neighbouring pixels normally look very similar, the formula determines the average brightness of the pixels and how they change relative to the average value of the macroblock.

The quantisation matrix
Here’s another tricky concept deeply rooted in hardcore mathematics. Think of a quantisation matrix as a palette of values that controls how the pixels in a macroblock are converted from pixels to a formula. Modifying it modifies the way pixel information is thrown away.

Multipass encoding & variable bitrates
Thanks to the world of audio codecs, you should already be familiar with the concept of variable bitrates. In short, they let you open the bandwidth taps during the more visually intensive scenes and use less bandwidth for the more placid scenes.

Multipass encoding is an extension of this; it lets the codec plot where to position keyframes and how to allocate its bandwidth resources as the footage progresses.