Digital Media

< CS101

JPEG Images

JPEG is a free and open standard for storing digital images, such as you would take with a digital camera. JPEG is a "lossy" compression format, detailed below, allowing an image to be adjusted, losing some detail but requiring less space in the process.

JPEG is an incredibly successful standard, allowing computers, phones, printers, TVs, email, blogs, .. to exchange image files and understand each other. Some commonly understood standard format for what is "an image" is needed, and JPEG is the mostly widely used one.

JPEG stands for the Joint Photographic Expert Group, a technical committee which drafted the standard. I doubt it was possible at the time to understand how widespread and critical this format would become.

JPEG is a "lossy" format, meaning that the level detail preserved when a JPEG is saved is adjustable. Say the quality levels are in the range q1, q2, .. q10, q10 corresponding to very little compression and high visual quality and q1 corresponding to very aggressive compression with lower visual quality. In reality the scale terminology is not exact across systems, sometimes described as 0-10, or 1-100. An image saved with q10 saves the maximum detail, but the resulting file takes up the most space. An image can be saved with a lower quality level, causing it to lose some detail, but take up less space. Or in other words, q1 is more compressed, and q10 is less compressed. JPEG is very smart about the way it loses detail, so saving at something like q7 is a normal thing to do without losing appreciable detail.

Here are three versions of the flowers.jpg image

Here is the image as it originally came out of my camera. I believe the camera uses about q7 internally. This image takes up 48 KB. The "raw" form of the image takes up about 490 KB, so q7 is saving us about 10x space. JPEG works quite well.
flowers at q7

Here is the image saved at q5, taking up 29 KB. I cannot see obvious differences between this version and the one above, although there must be some tiny differences.
flowers at q5

Here is the same image, saved at q1, taking up 14KB (about 4x fewer bytes than the q7 version). Generally you don't want to compress this much. If you zoom way in, you can see the results of the compression in this version:
flowers at q1

There are two things you notice in JPEGs as the shed detail:

Considering the the q1 version takes about 4x fewer bytes than the q7 version, JPEG does a good job keeping the basic look of the scene when asked to use less space.

Images - How Many Bytes?

How many bytes does an image take up? The main issue is just plain size -- how many pixels. The 457 x 360 flowers image has 164520 pixels. Say each pixel takes three bytes (one for each color channel), that's 493560 bytes or about 493 KB. If we double the width, so the image is 800 x 600, how many more bytes is that? Doubling the width multiples the bytes by 4, just like doubling the side of a rectangle increases the area by a factor of 4.

Note that the q7 flowers image up there takes up 48 KB (10x smaller than 493 KB), so even at relatively high quality levels, JPEG saves a huge amount of space -- that's why cameras use it internally.

My cheap little "12 megapixel" digital camera takes images which are 4000 by 3000 pixels, stored with JPEG compression of about q7. Each resulting image takes up about 3 MB.

Web Page Images

One case where you might care about image compression is a web page. Suppose you are publishing a web page that features 3 200KB images. The 600 KB of data that needs to be downloaded for the page make it feel kind of slow for visitors. One approach would be compression -- maybe if the images are more aggressively compressed, they would only take up 100 KB apiece, speeding the whole thing up. But of course you have to be careful that the specific material in the images does not look noticeably bad from the compression. Mostly, images only look noticeably bad at very aggressive compression levels -- q2 or q3.

The other important design idea is: don't feel you need to load your site up with images on every square inch. Visual design hint: less is more. Have text and other elements, and a few images here and there where it counts. Look at the google search page -- less is more!

GIF and PNG Images

GIF and PNG (Portable Network Graphics) are "lossless" image formats, recording every pixel exactly. They are used for non-photographic images, like little solid color borders. GIF was popular early on with the Internet. However parts of it were patented, but the patents was not enforced. When the patent was enforced more aggressively, the community created the free and open and PNG standard as an alternative, and it also performs a little better than GIF.

Audio Formats

MP3 is the dominant audio format (good example of a "network effect", a later topic). MP3 is lossy, like JPEG. Raw CD audio takes up about 10 MB per minute (this is how it is stored on an audio CD .. no compression). MP3 gets that down to about 1 MB per minute while still sounding pretty good. As with JPEG, you can choose the level of compression, say 2 MB per minute to keep more detail, or 512 KB per minute if space is at a premium.

MP3 is patented, and legitimately so. (Nick's opinion) Many modern software patents are garbage .. just patenting obvious solutions. However, MP3 is legitimate: it uses complex and non-obvious techniques to get its excellent 10x compression while still sounding good. If a device or software plays or produces MP3s, a license fee is due to the patent holders, on the order $1-$2 per instance. With each MP3 device you have owned over the years .. you have in effect paid this fee each time. Licensing of devices which can play video is similar.

AAC is a more modern form of MP3 used by Apple with their dominant iPod/iTunes system. AAC also features DRM (Digital Rights Management) features, which allow the original content owner to control how the purchaser can use the content they have "bought".

To provide a free and open alternative, the "Ogg Vorbis" audio format was developed. It is a free and open standard and royalty free and typically performs better than MP3. Aside: some geeks have a way of choosing tragically bad names for consumer facing technology, incorporating some obscure reference or joke. I think Ogg Vorbis would be much more widely adopted if they had chosen a better name.

Video Formats

A video is basically a series of images -- 20 to 60 per second, plus an audio "track". Video data takes up a lot of bytes, but computers have now become powerful enough to handle video. Very roughly speaking, say compressed video of about DVD quality takes about 2 GB per hour (roughly 30 MB per minute, or 0.5 MB per second). In reality, there is a very large range of video sizes -- HD video takes more space, smaller YouTube video takes less space. Video compression is complicated and the techniques are heavily patented.

MPEG (Motion Picture Experts Group) standardizes some video formats in the industry, and the MPEG-LA (Licensing Authority) handles collecting patent royalties, which are significant.

MPEG-2 is used in DVDs and some satellite TV systems, originally released in 1995. Compression techniques have gotten significantly better since then.

MPEG-4 and particular the "h.264" compression system is very good at producing good looking video with the minimum bytes. Most digital video cameras, phones, and Blu-ray disks use h.264 internally to compress and store the video data. Patent fees are paid by the manufacturer to produce an encoder or decoder in hardware or software.

h.264 Obnoxious Licensing Terms

One surprising thing about the h.264 licensing is that it does not come with an unrestricted right to distribute your own video. You have properly bought a video camera (paying for the patents), and produced your video and stored it on your hard drive. However, if you want to make a web site or whatever that distributes the video to many people, you may have to pay additional royalties for each minute distributed of your video. There may be exceptions if your video is distributed for free, however these terms have been changed over time, so really you have to consult a lawyer to see what you are permitted to do with your video. These license restrictions strike me as unusually obnoxious and certainly out of step with how you usually think of having your own data file. My guess is that MPEG did not have a lot of competition, resulting in these one-sided terms.

Open Video Format - WebM

In support of a free and open internet, Mozilla, Wikipedia, Google and others have been working on a true, free and open video compression scheme to compete with h.264 -- free to encode or decode video, and free to distribute the video however its owner wishes. Those are the sorts of terms under which the internet has thrived (and some poor villager in Africa should be able to watch an internet video without first paying a $1 for a license. The most recent iteration, mostly driven by Google, is the WebM format and project: WebM project.

Bottom line: convert your mpeg-4 video (i.e. h.264) to .webm format, and then you don't have to worry about the licensing for yourself or your audience. However, Safari and IE do not support WebM without installing extras. Google is doing the right thing here, and Microsoft and Apple should be embarrassed to be so clearly on the wrong side of internet history. In Firefox and Chrome, everything will just work.

HTML5 Video Tag

Before HTML5, there was not a good way to embed video, so everyone used Flash with all its problems. However, this has changed with the new HTML5 video tag to embed video right inside an HTML page.

Here's our earlier hard-drive example video embedded with a <video> tag as follows (the video is kind of big). Note how similar this is to putting in an image tag.
<video src="how-a-hard-drive-works.webm" controls="1"> WebM video </video>

Flash Video

So how did video work for the pre-HTML5 web? The Flash plugin. Flash is a proprietary format produced by Adobe. It is specific to Adobe, not a standard, and it is patented. Flash supports animation (you see it to make ads in web-pages a lot), and it also supports video. Internally, the most recent versions of flash use h.264 for video compression.

The Trouble with Flash

Since Flash is a proprietary product of Adobe, the only way software on a computer or phone or whatever supports it is when Adobe puts engineering resources into producing Flash "player" software for that system. Adobe's competence at this has been uneven, being at times a source of both crashing and security problems. Realistically, Adobe mostly cares about Windows (90% market share), so Flash support from Adobe on other platforms has tended to be worse.

A famous example was that, at one time, a significant fraction of all the crashes on Mac OS X were due to the Flash player crashing due to its own bugs. This made the Apple look bad to the consumer who cannot realize who is at fault, but Apple had little recourse. The story was that, understandably, Steve Jobs was extremely irritated at this situation. You can also imagine a tricky hypothetic situation where Apple and Adobe are strong competitors ... so how motivated is Adobe to fix problems that make Apple look bad? (not that that's what happened here... ordinary engineering mistakes is always a pretty good explanation!). This shows the problems of having a proprietary piece be a critical part of a public information network like the internet.

With a standard, like JPEG, if Apple wants good support for JPEG in their browser, Apple writes it .. they do not need anyone's permission, they can just do it. TCP/IP, HTML, HTTP, JPEG, PNG ... that's how all the important parts or the internet work, as free and open standards with no permission required. It is likely that HTML5 (a free and open standard), will eventually displace Flash.