Memory Safe Context Switching

Support for ucontext APIs is new since release 0.680. If you want to play with setcontext, getcontext, makecontext, and swapcontext then you have to build from source.

This document describes how Fil-C supports longjmp, setjmp, setcontext, getcontext, makecontext, and swapcontext in a totally memory-safe way. In particular, no misuse of those APIs in Fil-C can lead to stack corruption or any other violation of Fil-C's capability model.

These APIs are widely used:

longjmp and setjmp are used in C programs to implement exception handling. It's especially common to use them to implement exceptions "thrown" from signal handlers. getcontext, setcontext, makecontext, and swapcontext (aka the ucontext APIs) are used to implement coroutines and fibers. For example, Boost uses ucontext as part of its fiber implementation.

The ucontext APIs are less commonly used than longjmp/setjmp and some OSes (like Darwin) have deprecated them. However, they remain well supported in glibc.

Implementing these APIs in a way that preserves memory safety is hard since their misuse can result in restoring a dangling stack. For example, you could either setjmp or getcontext within some function, and then do any of the following things:

Return from that function. At this point, the context that was saved will attempt to restore a stack frame that no longer exists. Exit from the thread. At this point, the context that was saved will attempt to restore execution on a stack that has been freed.

Even more friendly APIs like makecontext and swapcontext can be straightforwardly misused:

You can use makecontext to create a context that points to some stack, then free that stack, and then either swapcontext or setcontext to that context. In Yolo-C, this will result in running on a dangling stack. Fil-C makes this not an error. You can call swapcontext with the second argument being the context that is currently executing. This might happen if you confuse the first and second arguments. In Yolo-C, in the best case, this will behave like a longjmp; in the worst case, it will result in executing on a dangling stack. In Fil-C, this is a safety error that panics your program.

In Yolo-C, execution on a dangling stack results in the most confusing kinds of crashes, since the debugger won't even be able to print a stack trace! Worse, if the program has subtle bugs in its handling of contexts, then an attacker could exploit those bugs to cause the program to do whatever the attacker likes. In Fil-C, execution on a dangling stack is not possible: all such cases are either panics at the point where you misused longjmp or one of the ucontext APIs, or they are reliably legal execution because of how Fil-C manages stacks.

Fil-C implements setjmp/longjmp and the ucontext APIs quite differently.

Making setjmp/longjmp Memory Safe

There is an impressive amount of depth to the depravity of setjmp. Before going into the details of how Fil-C implements setjmp/longjmp, we need to discuss exactly what makes this function so amazingly evil.

setjmp saves the context as it was at the moment when it was called so that when longjmp is called later, setjmp will return a second time. It is the fact that it returns twice that makes it so vile, and so we need to understand the implications precisely.

An Example

Consider this simple program:

#include <setjmp.h> #include <stdio.h>

int main(int argc, char** argv) { volatile int x = 42; jmp_buf jb; if (setjmp(jb)) { printf("x = %d\n", x); return 0; } x = 666; longjmp(jb, 1); printf("Should not get here.\n"); return 1; }

This program prints:

x = 666

And then exits. The flow is:

On the first call to setjmp, it returns 0 and saves its caller's context in jb. Then we set x to 666 and longjmp to jb with the value 1. setjmp returns 1, so we printf and exit.

Note that we have to mark x as volatile for the program to reliably print 666. Otherwise, the compiler is allowed to optimize the access to x and have it return 42 instead.

This might happen in the following ways:

The compiler could constant fold x to 42. This will happen in the example if we remove volatile and use any optimization level above -O0. Then x = 42 gets printed. Say that constant folding doesn't happen, maybe because we insert a asm("" : "+r"(x)) right after the definition of x. In that case, the compiler could register-allocate x in a callee-save register, in which case the register ends up saved by setjmp. This also leads to x = 42 being printed. Say that we experience register pressure for some reason, and x doesn't make it into a callee-save register, but instead gets spilled. At any optimization level above -O0, the compiler will split x into two variables: one for x = 42 and one for x = 666, and the printf will reference the first one (since x = 42 dominates the printf). Those two variables will almost always get separate spill slots. Hence, when we come out of the setjmp the second time, reading x will still give 42.

Three things to reflect upon:

To get the property that x's value is observed to be 666 in the printf, we need to make sure that the compiler treats x as a stack allocation rather than a variable. Using volatile achieves this. Also, passing a pointer to x to anywhere is likely to accomplish this. Spill slots are not the same as stack allocations. If a variable is stack-allocated, then it will get one stack allocation. If a variable is spilled, it may get multiple spills (often, a separate spill per assignment). The compiler is allowed to analyze the lifetime of spill slots and stack allocations. It's allowed to reuse spill slots. How does the compiler know that the x = 42 spill slot should stay alive until the longjmp happens? How come it won't get reused, resulting in x having either 666 or any random garbage when we fall out of the setjmp a second time?

Here's a more diabolical version of the example that triggers spilling of x to two different spill slots (one for 42 and one for 666) in gcc, clang, and filcc.

#include <setjmp.h> #include <stdio.h>

int main(int argc, char** argv) { int x = 42; asm volatile("" : "+r"(x)); jmp_buf jb; int a = 1, b = 2, c = 3, d = 4, e = 5, f = 6, g = 7, h = 9, i = 10; /* Force some spilling */ asm volatile("" : "+r"(a), "+r"(b), "+r"(d), "+r"(e), "+r"(f), "+r"(g), "+r"(h), "+r"(i)); if (setjmp(jb)) { asm volatile("" : "+r"(a), "+r"(b), "+r"(d), "+r"(e), "+r"(f), "+r"(g), "+r"(h), "+r"(i)); printf("x = %d\n", x); return 0; } x = 666; void (*jump)(jmp_buf, int) = longjmp; asm volatile("" : "+r"(x)); asm volatile("" : "+r"(jump), "+r"(a), "+r"(b), "+r"(d), "+r"(e), "+r"(f), "+r"(g), "+r"(h), "+r"(i)); jump(jb, 1); asm volatile("" : "+r"(jump), "+r"(a), "+r"(b), "+r"(d), "+r"(e), "+r"(f), "+r"(g), "+r"(h), "+r"(i)); asm volatile("" : "+r"(x)); printf("Should not get here.\n"); return 1; }

This program will print x = 42 even though x is not constant folded or register-allocated.

Note that all of the examples so far work in Fil-C. Even the inline assembly that we're using to obfuscate variable values works in Fil-C, and has the desired effect.

What Is Even Happening

Let's take a look at how simple setjmp is by looking at the musl implementation on x86_64:

__setjmp: _setjmp: setjmp: mov %rbx,(%rdi) /* rdi is jmp_buf, move registers onto it */ mov %rbp,8(%rdi) mov %r12,16(%rdi) mov %r13,24(%rdi) mov %r14,32(%rdi) mov %r15,40(%rdi) lea 8(%rsp),%rdx /* this is our rsp WITHOUT current ret addr */ mov %rdx,48(%rdi) mov (%rsp),%rdx /* save return addr ptr for new rip */ mov %rdx,56(%rdi) xor %eax,%eax /* always return 0 */ ret

This is only saving the callee-save registers, plus the stack pointer and instruction pointer as they were at the callsite. It's not saving the stack itself.

Later, when longjmp is called, the register state is restored with only one difference: %eax (the return value register) will get the argument passed to longjmp.

Hence, the most basic safety issue with setjmp is that if we call it and then return from the function that had called it, the context saved by setjmp is not valid to longjmp to. Jumping to such a context will result in a torn machine state:

The callee-save registers, stack pointer, and instruction pointer will be exactly as they had been at the time that setjmp had been called. The stack contents - in particular, the frame that the stack pointer is pointing at - will be whatever they were at the time that longjmp had been called.

longjmp is only safe if it's called at a time when the stack frame used by setjmp could not have possibly been overwritten, since that is the only way to guarantee that the register state restored by longjmp matches the stack frame that the stack pointer points to.

The easiest way to guarantee this is to ensure that longjmp is only called from within the function that called setjmp, or from some function called by the function that called setjmp (transitively).

But that's not all!

The compiler has to know that setjmp returns twice to ensure that spill slots are not reused unsoundly. In fact, compilers detect calls to setjmp and treat the functions that call it specially by disabling any optimization that would lead to a reuse of spill slots. This is surfaced a bit to compiler users with the returns_twice attribute.

Let's consider our diabolical example, but with the setjmp call obfuscated:

#include <setjmp.h> #include <stdio.h>

int main(int argc, char** argv) { int x = 42; asm volatile("" : "+r"(x)); jmp_buf jb; int a = 1, b = 2, c = 3, d = 4, e = 5, f = 6, g = 7, h = 9, i = 10; asm volatile("" : "+r"(a), "+r"(b), "+r"(d), "+r"(e), "+r"(f), "+r"(g), "+r"(h), "+r"(i)); int (*setjump)(jmp_buf) = setjmp; asm volatile("" : "+r"(setjump)); if (setjump(jb)) { asm volatile("" : "+r"(a), "+r"(b), "+r"(d), "+r"(e), "+r"(f), "+r"(g), "+r"(h), "+r"(i)); printf("x = %d\n", x); return 0; } x = 666; void (*jump)(jmp_buf, int) = longjmp; asm volatile("" : "+r"(x)); asm volatile("" : "+r"(jump), "+r"(a), "+r"(b), "+r"(d), "+r"(e), "+r"(f), "+r"(g), "+r"(h), "+r"(i)); jump(jb,