Handsum: An LQIP Image File Format

Nigel Tao

Low Quality Image Placeholders (LQIPs) are very small (both in terms of file size and pixel dimensions) images that load very quickly, providing immediate visual feedback while full-resolution images load slowly in the background. They are small-in-file-size enough that it’s feasible to base64-encode and inline them into a web page, minimizing network round-trips and improving page load times.

Existing LQIP techniques include the Blurhash and Thumbhash custom image file formats, as well as just using low-quality JPEG or WebP and very small pixel dimensions. This blog post introduces Handsum, another LQIP custom image file format.

Handsum files are also a fixed size-in-bytes: 48 bytes at the lowest quality setting and 147 bytes at the highest. For a desktop, native or command-line app, if you’re storing your thumbnail images in a database, this lets you use fixed-size columns, which can be faster and simpler to iterate over.

Handsum is based on the Discrete Cosine Transform (DCT), also used by the Blurhash algorithm, the Thumbhash algorithm and JPEG itself. Understanding how the relatively simple Handsum format works should help you if you ever wanted to understand how the relatively complicated JPEG format works (the JPEG specification is a 186-page PDF file; Handsum is simpler).

Web Page Example

Here’s an annotated screenshot of a Handsum web page demo. It uses a Handsum decoder written in C, compiled to Wasm.

Comparison

Here’s Handsum compared to PNG, Thumbhash (which is pretty similar to Blurhash), WebP, ETC2 and JPEG. You might want to open this image in its own tab, zoom in and pan around.

There’s a lot going on here. Let’s break it down.

Each row concerns a famous image, as seen on Wikipedia: earthrise, la-grande-jatte, lincoln, mona-lisa, parliament, pearl-earring, starry-night, tsunami, van-eyck, water-lillies.

The first column, “Original”, is the image itself, scaled to fit inside a 32×32 pixel bounding box (while preserving the aspect ratio).

For example, the 500×745 pixel Mona Lisa image is resized to 21×32.

Every other column is that original image after round tripping: encoding in some lossy fashion, then decoding.

The numbers (like 1368 or 615) above each cell is the encoded size-in-bytes. More bytes are clearly correlated with more quality. The question is how small (in terms of size-in-bytes) can you get while keeping acceptable quality.

The second column, “PNG 16”, simply scales the image down to fit inside a 16×16 pixel bounding box, encoding it losslessly as a PNG. (Yeah, we could possibly get a slightly smaller size-in-bytes if we pngcrushed it, or used WebP lossless, but it’s probably not that big a difference). Decoding produces a within-16×16 image (e.g. a 11×16 Mona Lisa) which we then upsample (with a bi-linear filter) back to 32 pixels wide or high.

The third column uses the Thumbhash LQIP codec. This produces extremely small encodings, weighing only 24–27 bytes.

The next four columns use Handsum, at each of its four quality settings. The “16” in “Hsum 16 q=1” means that, like “PNG 16”, the codec per se produces something that fits inside a 16×16 bounding box, so we upsample to 32 to get an image that’s better compared with the other cells.

The four columns after that use WebP lossy, based on VP8, at quality 0, 25 and 75. The “WebP 16” columns encode (and thus decode) within-16×16 (and, again, we upsample). The “WebP 32” column encodes within-32×32 image (and no upsampling is needed on decode).

The next two columns use ETC2 (wrapped in a PKM / PACKMAN container), the Ericsson Texture Compression format, mandatory in OpenGL ES 3.0 (Appendix C.1 of the linked spec) and similar to other texture formats like BCn / DXTn / S3TC. ETC2 is designed to be decoded on GPUs (with high parallelization), but you can decode on CPUs too, like any other image format. ETC2 is an interesting and under-appreciated image format, but diving into that is out of scope of this blog post. Again, “16” means encoding the lower-resolution source image and upsampling after decode, “32” means encoding the higher-resolution source image.

The last two columns are JPEG, at quality 75 (which is roughly comparable to, but not exactly the same as, WebP’s “quality = 75”). Again, “16” vs “32” means lower-res-and-upsample vs higher-res.

At Handsum’s q=1 (lowest quality), it’s roughly comparable to a bigger (higher file size), better (better visual quality) Thumbhash. Bigger is in relative terms. The absolute difference is barely 24 bytes. By “better visual quality” I mean that, if you told me to guess the famous image based on the thumbnail, then for Mona Lisa or Girl with a Pearl Earring, I’d say that that’s plausible for Handsum (q=1) but implausible for Thumbhash. Still better than the modern potato art that WebP 16 q=0 produces, though.

At Handsum’s q=4 (highest quality), it’s roughly comparable to WebP Lossy (VP8) and ETC2, for a given size-in-bytes budget. One difference, though, is that at a given quality setting (and at c=3, see below), every Handsum image file is the same size-in-bytes: q=1 files are always 48 bytes, q=2 files are 75, q=3 files are 123 and q=4 files are 147.

If you wanted to make an embedded app for displaying a catalog of artworks (oil paintings, audio records, etc.), you could budget 256 bytes per catalog entry, for example. After assigning 50 bytes each to the artist name and the artwork name (ellipsizing if necessary) and a few more bytes to other metadata (time stamp, track length, etc.), that could leave you 150 bytes for the thumbnail. If you were using WebP for encoding the 16×16 thumbnails, q=75 will fit at or under 150 bytes most of the time, but not all of the time. WebP encoders also typically have a quality setting but not a byte-budget setting, so generating a small-enough (in terms of size-in-bytes) WebP takes a little dance. In contrast, with Handsum q=4, you’ll get 147 bytes every time, guaranteed.

How Handsum Works

Handsum has three color settings (c=1 means Gray, c=3 means RGB or, equivalently, YCbCr and c=4 adds an alpha channel). This blog post only discusses c=3. The reference implementation’s code comments has more details on c=1 and c=4.

This blog post also focuses mostly on the q=4 (highest) quality setting, as it has the simplest DCT implementation.

For any source image, such as a 500×745 pixel Mona Lisa, the encoding algorithm reduces it to 144 bytes of pixel data (at q=4). This becomes 147 after prepending a a 3-byte header: a 15-bit magic signature, a 2-bit color, a 2-bit quality and a 5-bit aspect ratio.

Handsum encoding can be decomposed into a sequence of stages. Decoding is simply reversing the stages in the opposite order, plus a post-processing “Loop Filter” stage (see below). When decoding gets to Stage 0 (Resize), it’s not a full reversal. The decoded image’s longest dimension (width or height) is 16 pixels and the 5-bit aspect ratio (in the header) gives the shorter dimension.

Handsum is not a general purpose image format, only a thumbnail image format.

Stage 0: Resize

Resizing to 16×16, at 3 bytes RGB (Red, Green, Blue) per pixel, starts us off at 768 bytes of data. Encoding’s later stages operate on this 16×16 workspace regardless of the original source image’s aspect ratio.

Stage 1: YCbCr Conversion

Converting from RGB to YCbCr uses the same formulae as JPEG / BT.601:

Y = (+0.2990 * R) + (+0.5870 * G) + (+0.1140 * B) Cb = (-0.1687 * R) + (-0.3313 * G) + (+0.5000 * B) Cr = (+0.5000 * R) + (-0.4187 * G) + (-0.0813 * B)

This transformation from 3 bytes RGB per pixel to 3 bytes YCbCr per pixel is lossless (and perfectly reversible) in infinite-precision real numbers, and only slightly lossy in finite-precision computer numbers.

A (theoretically) lossless transformation doesn’t save any bytes per se, as we’re still at 3×16×16 = 768. Linear algebra enthusiasts would recognize this as a (lossless) change of basis. But this sets us up for later stages to better compress the 16×16 image.

Compression means bringing that 768 number down. But, for a fixed-size-in-bytes output (768 / 144 is a 5.333× compression ratio), lowering that number necessitates discarding (or losing, in the “lossy compression” sense) information. Switching to YCbCr gives us better opportunities to discard information that people mostly don’t notice.

Stage 2: Chroma Downsampling

Human vision is more sensitive to luma and less sensitive to chroma, so we can be more aggressive in discarding chroma channel information (to achieve better compression ratios).

This stage simply drops 75% of the two chroma channels’ data (which is 50% of the overall 768), downsampling Cb and Cr from 16×16 to 8×8. We’re now at (16×16 + 8×8 + 8×8) = 384 total bytes, a 50% saving.

This is the classic 4:2:0 chroma subsampling, as used by WebP images and is also very popular for JPEG images. For those who already know how JPEG works, Handsum’s 16×16 pixel workspace is essentially one JPEG 4:2:0 MCU (Minimum Coded Unit).

Stage 3: DCT

We slice up our 16×16 luma and 8×8 chroma grids into 2×2 blocks, 4 samples per block, and apply the Discrete Cosine Transform to each block independently. This transforms 4 samples to 4 samples, per block. Like Stage 1, this is just a change of basis. Like Stage 1, this doesn’t lower the byte count per se (and would be a lossless transform, with infinite-precision real numbers), but again, it unlocks later opportunities.

Handsum q=4 uses 2×2 blocks (and so 4 coefficients per block). Lower Handsum quality settings use 4×4 or 8×8 blocks (and JPEG’s DCT also uses 8×8). DCT is applicable to any of those block sizes, but the DCT formulae are simplest for 2×2 and we’ll go into more detail about that further below.

Stage 4: Quantization

We started off using 8 bits (the half-open range 0 .. 256) per RGB sample. Quantization from 8 to 4 bits (the half-open range 0 .. 16) is conceptually as simple as doing value >>= 4 on the encode side and value *= 0x11 on the decode side.

In practice, Handsum’s quantization is a little more complicated:

DC and AC components (see below) are de-quantized slightly differently.