A whirlwind tour of systemd-nspawn containers

In the last yearly update, I talked about isolating my self-hosted LLMs running in Ollama as well as Open WebUI in systemd-nspawn containers and promised a blog post about it. However, while writing that blog post, a footnote on why I am using it instead of Docker accidentally turned into a full blog post on its own. Here’s the actual post on systemd-nspawn. Fundamentally, systemd-nspawn is a lightweight Linux namespaces-based container technology, not dissimilar to Docker. The difference is mostly in image management—instead of describing how to build images with Dockerfiles and distributing prebuilt, read-only images containing ready-to-run software, systemd-nspawn is typically used with a writable root filesystem, functioning more similarly to a virtual machine. For those of you who remember using chroot to run software on a different Linux distro, it can also be described as chroot on steroids. I find systemd-nspawn especially useful in the following scenarios: When you want to run some software with some degree of isolation on a VPS, where you can’t create a full virtual machine due to nested virtualization not being available1; When you need to share access to hardware, such as a GPU (which is why I run LLMs in systemd-nspawn); When you don’t want the overhead of virtualization; When you want to directly access some files on the host system without resorting to virtiofs; and When you would normally use Docker but can’t or don’t want to. For reasons, please see the footnote-turned-blog post. In this post, I’ll describe the process of setting up systemd-nspawn containers and how to use them in some common scenarios. Update (2025-05-19): added safer Alpine container creation instructions and some more guidance on setting up Alpine Linux containers. Start with a chroot systemd-nspawn, like all containers, requires a separate rootfs in a directory. Since it doesn’t come with an image manager to build images for you, you’ll need to use the traditional Unix approach of creating a chroot. It is, after all, chroot on steroids. For systemd-nspawn, the typical convention is to put these rootfses under /var/lib/machines/[hostname], where the hostname is that of the container. For this exercise, I’ll just use the name of the OS as the hostname. Each Linux distro has its own way of creating a new rootfs, so we’ll go through some of the more common ones. I am testing all this on Debian, but there’s no reason why you wouldn’t be able to do any of this on another distro by substituting the equivalent packages. Still, I’ll probably go deeper into Debian because I am more familiar with it. debootstrap on the Debian family debootstrap has featured on this blog many times, mostly when doing crazy things like installing Debian by hand from rescue systems. For Debian-based distros like Ubuntu or Debian itself, this is the tool to use to create a rootfs. On a Debian-based OS, install it with sudo apt install debootstrap. It may also be packaged on other distros, or you can just download it from Debian’s repositories: $ curl -L https://salsa.debian.org/installer-team/debootstrap/-/archive/master/debootstrap-master.tar.gz | tar xz ... $ cd debootstrap-master/ $ ./debootstrap I: usage: [OPTION]... <suite> <target> [<mirror> [<script>]] ... Either way you installed debootstrap, let’s create the rootfs in /var/lib/machines/debian: $ sudo debootstrap --include=systemd,dbus,locales,bash-completion,curl bookworm /var/lib/machines/debian I: Target architecture can be executed I: Retrieving InRelease ... I: Base system installed successfully. Here, we’ve installed bookworm, but feel free to substitute this for the latest Debian release. To avoid silly issues, I ensured that systemd and dbus are installed, along with locales to avoid weird warnings in apt, bash-completion so I don’t hate my life when typing in commands by hand, and curl so you can download stuff later. Some of these require a bit of additional configuration later, which we’ll do once we boot into the container. Installing RHEL-derivatives with dnf First, install dnf and rpm if you are on another distro, e.g. on Debian: $ sudo apt install dnf rpm ... Setting up dnf (4.14.0-3+deb12u1) ... Setting up rpm (4.18.0+dfsg-1+deb12u1) ...

...

$ sudo mkdir -p /var/lib/rpm $ sudo rpm --initdb Then, we make a Fedora 41 chroot as an example: $ sudo mkdir -p /var/lib/machines/fedora/etc/dnf $ sudo tee /var/lib/machines/fedora/etc/dnf/dnf.conf > /dev/null [fedora] name=Fedora $releasever - $basearch metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch gpgkey=https://getfedora.org/static/fedora.gpg [updates] name=Fedora $releasever - $basearch - Updates metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch gpgkey=https://getfedora.org/static/fedora.gpg $ sudo dnf --releasever=41 --best --setopt=install_weak_deps=False --repo=fedora --repo=updates --installroot=/var/lib/machines/fedora install dnf fedora-release glibc glibc-langpack-en iproute iputils less ncurses passwd systemd systemd-networkd systemd-resolved util-linux vim-default-editor ... Transaction Summary ================================================================================ Install 137 Packages Total download size: 62 M Installed size: 206 M Is this ok [y/N]: y ...

Complete!