(iterate think thoughts): Painting with Gaussians

Last year I built an edge-aware pixelation tool to turn images into pixel art by deforming a grid so that it follows image edges instead of naively using a fixed grid over the picture. Using an edge adapter grid bent to inform the color and brightness of the pixels worked well for keeping edges crisp and preserving details.I later realized the same edge information could also be applied in a context of digital painting. A painting program has to figure out where to put brush strokes, how big should they be, and which direction should they flow in. Much of that is already encoded in the edges since they mark the boundaries between regions. These are the contours around areas of objects that a brush would trace, and their absence indicates generally flat areas where a few broad strokes should suffice.And so, I set off to see if I could make a program that paints in the style of a digital painting where marks derived from the image structure would resemble brush strokes. My goal was to make an interactive tool where you drag sliders around and the painting reforms itself in front of you. The project was also a great opportunity for me to test drive Jolt and see how well it works for building a non-trivial project.In this post, I'll walk you through how it all came together. We'll see what ideas worked and which ones didn't. Most importantly, we'll find out whether the end result actually ends up resembling anything like a painting.The first question we need to consider is what a brush stroke is exactly in computational terms. A stroke of oil or acrylic is an elongated mark which has a center of color that fades toward its edges, and its orientation is the product of a brush being dragged across a canvas. It's translucent at the edges, and strokes overlap, allowing a painter to lay down broad blocks of color first, then build detail on top with smaller and more translucent marks to add finer detail.It turns out that a 2D Gaussian splat maps onto this idea surprisingly well. It has a mean which is where the stroke lands, a covariance matrix representing how it's stretched and rotated, along with a color and opacity. The covariance can be used to encode the brush direction and its elongation with the major axis pointing along the stroke, and the minor axis across it. Rendering a field of splats with standard over-compositing where each one is occluding what's behind it by its alpha gives you a similar effect to a natural painting model that allows marks to layer and blend together. Of course, you don't get the same fidelity of actual paint, so the effect is closer to digital painting using a tool like GIMP or Krita.There are already some implementations of this idea such as DrawingWithGaussians and 2d-gaussian-splatting-Art. However, both of them use a gradient descent approach where they seed random splats, and then iteratively nudge their positions, shapes, and colors until the rendered field has the appearance of a target image. That's the well known approach which is both slow and opaque. The worst part is that the end result ends up being simply a lossy reconstruction of the input image rather than looking like any sort of a painting.Since I already had the solution for extracting edge information from the image, I didn't see the point of evolving the image blindly. Instead, the extracted edges can be used to guide the painting process because they tell us where the details are along with the orientation of the strokes. Between detail density and pixel colors I had all the information that I'd need without having to resort to gradient descent. I'd basically just need to trace the existing image. How hard could it be really?So, I started following the reference rasterizer which uses additive blending where: pixel = background + Σ(intensity × color). Turns out, this approach works in the fitting regime because the optimizer learns colors that compensate for overlap. Unfortunately, seeding thousands of splats directly from pixel colors and rendering them additively creates a lot more overlap. With 1,200 splats on a 64×64 image, the sum hit 22.06 in some pixels, creating pure white blobs all over the image. Luckily, the problem can be solved by using the standard over-operator from alpha compositing to make each splat occlude what's behind it by its alpha so that the summed color never exceeds 1.0. Another benefit of this approach is that it cleanly separates color sampled from the image and opacity.Edges Tell You Where to PaintEvery image in this section is the same photo run through the same pipeline, with one idea switched off at a time — same source, same stroke budget, same base size — so each step shows exactly what that one idea buys.I started using the following source photo, and recorded the progress as I continued to improve the app to illustrate what each idea buys. Let's see how the painting evolves as new tricks are added to the mix.I got a rather sad output which looked like a uniform mosaic with my initial renderer. Every splat had the same size, aspect ratio, and rotation, producing a regular grid of identical blobs. Not really looking like much of a painting so far.An actual painter would vary their strokes using a few broad strokes for flat regions such as the sky or a smooth surface. Then, along edges and in textured areas like eyes or fabric, a smaller brush gets used to make numerous finer strokes that follow the contours of the objects.One way to emulate this is by using a structure tensor to compute the image gradient, encoding how much and in which direction the color changes for each pixel. A 2×2 tensor is formed from the gradient outer product, and blurred over a neighborhood. Importantly, the tensor's eigenvectors will tell you three key things. The major eigenvector points across the contour, providing the direction of the strongest gradient. The minor eigenvector points along the edge, giving the direction of the brush stroke. And coherence, which is the ratio of eigenvalues, tells you whether the edge is a crisp contour or isotropic mush.Each splat gets its own covariance from this tensor at its position, and gets elongated along the edge, with elongation being proportional to coherence. Flat areas stay round while the edges become thin, directional strokes that trace the contours of the objects in the scene. This is the classic painterly rendering trick from Litwinowicz and Hertzmann. With it in place, the rendering started to resemble something that looks like brushwork if you squint a bit. Here, the fur and the hat brim pick up direction, and the whiskers start to appear.But here, I hit another problem because the structure tensor uses luminance gradients which are grayscale, making it blind to isoluminant color edges such as red lips against pale skin or a blue sign on a grey wall. Luckily, the Di Zenzo color tensor can be used to compute Sobel gradients per RGB channel. Their outer products can then be summed into one tensor, giving a chroma edge that drives orientation as strongly as a luma edge.Edges Aren't EnoughWhile the structure tensor solves the problem of figuring out orientation, it tells you nothing about the density of the region. And without knowing that, it's not possible to figure out how many strokes need to go in that region and how small should they be.You might be thinking that you could just use edge strength to figure this out, and decide on the number of strokes to use based on that. But doing so ends up missing the texture of the objects entirely.

So, relying solely on doing edge analysis makes it impossible to reproduce many of the important details present in the original image.This is where the Haar wavelet, which I discussed in this post, comes into play. Running a multi-scale 2D Haar decomposition on the luminance produces a detail energy map by summing the absolute detail coefficients across scales for each cell. The map will contain high values in textured and edgy regions, and low values in ones lacking detail. But raw wavelet energy still has the problem of being absolute.