Compositing & Blending • Niklas Gadermann

Your browser is a miracle! Every pixel you see on this page is the result of stacking hundreds of elements on top of each other and deciding, millions of times, pixel by pixel, what the result should look like. Two important steps in this process are compositing and blending, and in this post, we'll get to see what they are, how they work, and where they can be useful when building fun effects on the web. What you'll learn:What compositing is, and what it has to do with blend modesWhat a blend mode is and how you can reason about what it will doWhich blend modes exist and which ones are commonly useful in real-world scenariosHow blend modes can be used in CSS, and problems that you might run into Let's dive in! Compositing Before we can talk about what a blend mode is, we have to talk about compositing. Compositing is "the process of combining an element with its backdrop", according to the W3C specification. What does this mean? The pixels you see on your screen while you're reading this were drawn by your browser. This webpage consists of many HTML elements making up many layers, and the browser draws each layer and then combines them. If we simplify a bit, it looks like this: 0 layers composited The step of combining these layers with each other is called compositing. We'll call the layer that is drawn onto during compositing the backdrop, and the layer that is being drawn the source. Let's look at how the browser might composite the input textbox of your favorite LLM tool, which consists of many different elements. We'll pretend that each element is a layer, but do note that in reality, layers are made up of multiple elements. Once a layer is painted, operations like transforming or fading it only require re-compositing, not re-painting, which saves time and increases performance. For our general intro to compositing though, what exactly constitutes a layer in the browser is not of huge concern. Move the slider to pan through the compositing steps, and view the different compositing layers by clicking the small 3D button in the top corner. Empty In this example, we simply draw the current layer on top of the backdrop and call it a day.

For a browser, this is sufficient. But compositing can be much more powerful: In different contexts, we can choose from many other ways to composite the source layer with the backdrop, like drawing it behind, or only drawing the overlap of both layers. Let's look at the different operations which are possible. When we combine two images, each pixel falls into one of four regions, depending on whether this pixel exists in both the source and the backdrop, only one of them, or neither: SourceBackdropSource onlyBackdrop onlyBothNeither For each of these regions, we can choose what the composited pixel should be; for example, for the region in which only the source has visible pixels, we can either display these pixels in the composited image, or display nothing. The following table shows the available options: RegionNameDescriptionSource OnlyThe pixel is present in the source but not the backdrop.2 Options: Choose between showing source or nothingBackdrop OnlyThe pixel is present in the backdrop but not the source.2 Options: Choose between showing backdrop or nothingBothThe pixel is present in both the source and the backdrop.3 Options: Choose between showing source, backdrop or nothingNeitherThe pixel is not present in either the source or the backdrop.1 Option: Show nothing (as there is nothing to composite) The Porter-Duff Operators If we count all the possible permutations, there are a total of 2×2×3×1=122 \times 2 \times 3 \times 1 = 12 ways to composite the four regions of two overlapping images. These 12 options are called the Porter-Duff compositing operators, after Thomas Porter and Tom Duff of Lucasfilm, who came up with them in 1984. They define, mathematically, how to combine two pixels during compositing. The formula to calculate the resulting color for a pixel when compositing two layers is given by: Co=Fs×Cs+Fb×CbC_o = F_s \times \textcolor{#3b82f6}{C_s} + F_b \times \textcolor{#f59e0b}{C_b} \\ Variables with the letter CC are colors, which are given as a 3-dimensional vector in the form (R,G,B)(R, G, B).

CoC_o is the output color, and Cs\textcolor{#3b82f6}{C_s} and Cb\textcolor{#f59e0b}{C_b} are the source and backdrop colors, respectively. We can calculate the resulting opacity αo\alpha_o in the same way: αo=Fs×αs+Fb×αb\alpha_o = F_s \times \textcolor{#3b82f6}{\alpha_s} + F_b \times \textcolor{#f59e0b}{\alpha_b} Let's address the elephant in the room: What are the factors FsF_s and FbF_b? These are set depending on the compositing operator in use, and they are central to determining how the compositing operation will behave. For example, if we only want to keep the source color when both the source and backdrop have a color set, then we can set Fs=1F_s = 1 and Fb=0F_b = 0. This will simplify the color formula to Co=CsC_o = \textcolor{#3b82f6}{C_s} and the opacity formula to αo=αs\alpha_o = \textcolor{#3b82f6}{\alpha_s}, which shows us that only the source color remains with its original opacity. What really helped make these click for me was thinking about them as a sort of maximum contribution for both the source color and the backdrop color. For example, when we set Fb=0F_b = 0, then the backdrop color can never contribute anything to the result. However, if we set Fb=1−αsF_b = 1 - \textcolor{#3b82f6}{\alpha_s}, then the backdrop color could contribute fully to the result, depending on the opacity of the source at this pixel.

If we think back to the 12 operations we came up with above, we can now express them as maximum contribution factors FsF_s and FbF_b for each operator: NameFsF_sFbF_bDescriptionClear0000Clear imageSource1100Only source is visibleBackdrop0011Only backdrop is visibleSource Over111−αs1 - \alpha_sSource covers backdropBackdrop Over1−αb1 - \alpha_b11Backdrop covers sourceSource Inαb\alpha_b00Source visible in intersectionBackdrop In00αs\alpha_sBackdrop visible in intersectionSource Out1−αb1 - \alpha_b00Source visible outside intersectionBackdrop Out001−αs1 - \alpha_sBackdrop visible outside intersectionSource Atopαb\alpha_b1−αs1 - \alpha_sBackdrop with source visible in intersectionBackdrop Atop1−αb1 - \alpha_bαs\alpha_sSource with backdrop visible in intersectionXOR1−αb1 - \alpha_b1−αs1 - \alpha_sIntersection is cleared Because the raw formulas can be rather confusing, it's much easier to play with this yourself. In the following playground, you can choose the Porter-Duff operator of your choice and explore how the formulas behave for each pixel in the two overlapping squares: CompositingBackdropSource100%100%InspectorBackdroprgb 245, 158, 11α 1.00Sourcergb 59, 130, 246α 1.00Resultrgb 0, 0, 0α 0.00 Opacity and Coverage I said before that the α\alpha variables used in the general compositing formula describe the color's opacity, but that is only partially true. In reality, α\alpha actually combines the opacity and the coverage. But what is the coverage of a pixel? When shapes are rendered by the browser, their edges don't fit perfectly on the pixel grid onto which they are rendered: In these cases, the rendering step will determine how much of a pixel is covered by a shape.

This percentage value is the coverage. The color itself might also not be fully opaque, so the final alpha value α\alpha is given by α=opacity×coverage\alpha = \text{opacity} \times \text{coverage} So a fully opaque pixel at the edge of a shape might have an α\alpha value of 1×0.5=0.51 \times 0.5 = 0.5, but so would a semi-transparent pixel with full coverage: 0.5×1=0.50.5 \times 1 = 0.5. In the end, the origin of the alpha value does not really matter. It's just important to know that the α\alpha calculations in the compositing formula are used even for fully opaque objects in cases where they don't align perfectly with the pixel grid which they are rendered onto. So next time you see a smooth rendered edge on a fully opaque font or rounded shape, you'll know that it only appears smooth because the compositing step combined the backdrop and source colors into faded color somewhere in between, depending on the pixel coverage. Neat! Compositing in Practice So, this is compositing. But what can it do for you? Unfortunately, in the DOM, you don't really get any control about Porter-Duff operators. Rendering simply uses source over compositing, which is sure to be the most intuitive and definitely what you would expect. But you cannot control it, it simply is. In canvas land on the other hand, you get a say in all of this! On a canvas context, the globalCompositeOperation property can be set to change the compositing operator used when drawing new things onto the canvas. This way, you can combine different Porter-Duff operators to draw foreground elements, mask them if needed, and then draw the backgrounds behind them later. To illustrate this, we'll draw a little Pac Man ghost using only simple shapes and compositing operators. Move the slider to see the little ghost come to life. At each step, the next shapes to be composited are shown as a dashed red outline. Head circle (source-over) This uses a few of the compositing operators we saw before: We carve out areas of our ghost for its "feet" using destination-out, and apply the pellets and black background at the end behind the finished ghost using destination-over.

I encourage you to get creative with this; I'm sure there are lots of fun ideas left to explore! Blending Now that we know how the pixels on our screen are composited, we can look at how they are blended. When we composite two layers, blending happens in the region where the source and backdrop overlap (the Both region from before), and it allows us to change what the resulting color will be. SourceBackdropBlendinghappenshere Conceptually, the blending step happens before the compositing step: For every pixel in the region, the source color is first blended in place with the backdrop color, which results in a new source color. This new color is then composited with the backdrop to receive the final pixel color. Here is what that process looks like for a single pixel: From now, let's focus on the blending step: The general formula for blending the source pixel with the backdrop pixel is: Cs = (1−αb) × Cs + αb × B(Cb,Cs)C_s\;=\;(1 - \alpha_b)\,\times\,C_s\;+\;\alpha_b\,\times\,B(C_b, C_s) The key thing to notice is that CsC_s shows up on both sides of the equation: The source color is replaced with a new one before being composited later. That newly created color combines the original CsC_s (if the backdrop is not fully opaque) with whatever B(Cb,Cs)B(C_b, C_s) produces. So what is this? BB is what we'll call our blending function: It takes two colors as a parameter, and returns a new color. That's it. In theory, we can choose any function for BB that we like. We could have a blending function that always returns hotpink, and this would totally work out. In practice though, we don't get to choose BB freely. Instead, the tools we use – in our case, the browser – give us some presets for BB from which we can choose. These presets are the common blend modes that you might have encountered before, and we can refer to them by name: lighten, darken, etc. So now that we know that blend modes are essentially just named functions, that begs the question: What are the underlying functions?