SOCKMAP - TCP splicing of the future

2019-02-187 min readRecently we stumbled upon the holy grail for reverse proxies - a TCP socket splicing API. This caught our attention because, as you may know, we run a global network of reverse proxy services. Proper TCP socket splicing reduces the load on userspace processes and enables more efficient data forwarding. We realized that Linux Kernel's SOCKMAP infrastructure can be reused for this purpose. SOCKMAP is a very promising API and is likely to cause a tectonic shift in the architecture of data-heavy applications like software proxies. Image by Mustad Marine public domainBut let’s rewind a bit. Birthing pains of L7 proxies Transmitting large amounts of data from userspace is inefficient. Linux provides a couple of specialized syscalls that aim to address this problem. For example, the sendfile(2) syscall (which Linus doesn't like) can be used to speed up transferring large files from disk to a socket. Then there is splice(2) which traditional proxies use to forward data between two TCP sockets. Finally, vmsplice can be used to stick memory buffer into a pipe without copying, but is very hard to use correctly.Sadly, sendfile, splice and vmsplice are very specialized, synchronous and solve only one part of the problem - they avoid copying the data to userspace. They leave other efficiency issues unaddressed.betweenavoid user-space memoryzerocopysendfiledisk file --> socketyesnosplicepipe <--> socketyesyes?vmsplicememory region --> pipenoyesProcesses that forward large amounts of data face three problems:Syscall cost: making multiple syscalls for every forwarded packet is costly.Wakeup latency: the user-space process must be woken up often to forward the data. Depending on the scheduler, this may result in poor tail latency.Copying cost: copying data from kernel to userspace and then immediately back to the kernel is not free and adds up to a measurable cost. Many tried Forwarding data between TCP sockets is a common practice. It's needed for:Transparent forward HTTP proxies, like Squid.

Reverse caching HTTP proxies, like Varnish or NGINX.Load balancers, like HAProxy, Pen or Relayd.Over the years there have been many attempts to reduce the cost of dumb data forwarding between TCP sockets on Linux. This issue is generally called “TCP splicing”, “L7 splicing”, or “Socket splicing”.Let’s compare the usual ways of doing TCP splicing. To simplify the problem, instead of writing a rich Layer 7 TCP proxy, we'll write a trivial TCP echo server.It's not a joke. An echo server can illustrate TCP socket splicing well. You know - "echo" basically splices the socket… with itself! Naive: read write loop The naive TCP echo server would look like: while data: data = read(sd, 4096) writeall(sd, data) Nothing simpler. On a blocking socket this is a totally valid program, and will work just fine. For completeness, I prepared full code here. Splice: specialized syscall Linux has an amazing splice(2) syscall. It can tell the kernel to move data between a TCP buffer on a socket and a buffer on a pipe. The data remains in the buffers, on the kernel side. This solves the problem of needlessly having to copy the data between userspace and kernel-space. With the SPLICE_F_MOVE flag the kernel may be able to avoid copying the data at all!Our program using splice() looks like: pipe_rd, pipe_wr = pipe() fcntl(pipe_rd, F_SETPIPE_SZ, 4096);

while n: n = splice(sd, pipe_wr, 4096) splice(pipe_rd, sd, n) We still need wake up the userspace program and make two syscalls to forward any piece of data, but at least we avoid all the copying. Full source. io_submit: Using Linux AIO API In a previous blog post about io_submit() we proposed using the AIO interface with network sockets. Read the blog post for details, but here is the prepared program that has the echo server loop implemented with only a single syscall.

Image by jrsnchzhrs By-Nd 2.0 SOCKMAP: The ultimate weapon In recent years Linux Kernel introduced an eBPF virtual machine. With it, user-space programs can run specialized, non-turing-complete bytecode in the kernel context. Nowadays, it's possible to select eBPF programs for dozens of use cases, ranging from packet filtering, to policy enforcement.From Kernel 4.14 Linux got new eBPF machinery that can be used for socket splicing - SOCKMAP. It was created by John Fastabend at Cilium.io, exposing the Strparser interface to eBPF programs. Cilium uses SOCKMAP for Layer 7 policy enforcement, and all the logic it uses is embedded in an eBPF program. The API is not well documented, requires root and, from our experience, is slightly buggy. But it's very promising. Read more:LPC2018 - Combining kTLS and BPF for Introspection and Policy Enforcement Paper Video SlidesOriginal SOCKMAP commitThis is how to use SOCKMAP: SOCKMAP or specifically "BPF_MAP_TYPE_SOCKMAP", is a type of eBPF map. This map is an "array" - indices are integers. All this is pretty standard. The magic is in the map values - they must be TCP socket descriptors.This map is very special - it has two eBPF programs attached to it. You read it right: the eBPF programs live attached to a map, not attached to a socket, cgroup or network interface as usual. This is how you would set up SOCKMAP in user program: sock_map = bpf_create_map(BPF_MAP_TYPE_SOCKMAP, sizeof(int), sizeof(int), 2, 0)

prog_parser = bpf_load_program(BPF_PROG_TYPE_SK_SKB, ...) prog_verdict = bpf_load_program(BPF_PROG_TYPE_SK_SKB, ...) bpf_prog_attach(prog_parser, sock_map, BPF_SK_SKB_STREAM_PARSER) bpf_prog_attach(prog_verdict, sock_map, BPF_SK_SKB_STREAM_VERDICT) Ta-da!

At this point we have an established sock_map eBPF map, with two eBPF programs attached: parser and verdict. The next step is to add a TCP socket descriptor to this map. Nothing simpler: int idx = 0; int val = sd; bpf_map_update_elem(sock_map, &idx, &val, BPF_ANY); At this point the magic happens. From now on, each time our socket sd receives a packet, prog_parser and prog_verdict are called. Their semantics are described in the strparser.txt and the introductory SOCKMAP commit. For simplicity, our trivial echo server only needs the minimal stubs. This is the eBPF code: SEC("prog_parser") int _prog_parser(struct __sk_buff *skb) { return skb->len; }

SEC("prog_verdict") int _prog_verdict(struct __sk_buff *skb) { uint32_t idx = 0; return bpf_sk_redirect_map(skb, &sock_map, idx, 0); } Side note: for the purposes of this test program, I wrote a minimal eBPF loader. It has no dependencies (neither bcc, libelf, nor libbpf) and can do basic relocations (like resolving the sock_map symbol mentioned above). See the code.The call to bpf_sk_redirect_map is doing all the work. It tells the kernel: for the received packet, please oh please redirect it from a receive queue of some socket, to a transmit queue of the socket living in sock_map under index 0. In our case, these are the same sockets! Here we achieved exactly what the echo server is supposed to do, but purely in eBPF.This technology has multiple benefits. First, the data is never copied to userspace. Secondly, we never need to wake up the userspace program. All the action is done in the kernel. Quite cool, isn't it?We need one more piece of code, to hang the userspace program until the socket is closed. This is best done with good old poll(2): /* Wait for the socket to close.

Let SOCKMAP do the magic. */ struct pollfd fds[1] = { {.fd = sd, .events = POLLRDHUP}, }; poll(fds, 1, -1); Full code. The benchmarks At this stage we have presented four simple TCP echo servers:naive read-write loopspliceio_submitSOCKMAPTo recap, we are measuring the cost of three things:Syscall costWakeup latency, mostly visible as tail latencyThe cost of copying dataTheoretically, SOCKMAP should beat all the others:syscall costwaking up userspacecopying costread write loop2 syscallsyes2 copiessplice2 syscallsyes0 copy (?)io_submit1 syscallyes2 copiesSOCKMAPnoneno0 copies Show me the numbers This is the part of the post where I'm showing you the breathtaking numbers, clearly showing the different approaches. Sadly, benchmarking is hard, and well... SOCKMAP turned out to be the slowest. It's important to publish negative results so here they are.Our test rig was as follows:Two bare-metal Xeon servers connected with a 25Gbps network.Both have turbo-boost disabled, and the testing programs are CPU-pinned.For better locality we localized RX and TX queues to one IRQ/CPU each.The testing server runs a script that sends 10k batches of fixed-sized blocks of data. The script measures how long it takes for the echo server to return the traffic.We do 10 separate runs for each measured echo-server program.TCP: "cubic" and NONAGLE=1.Both servers run the 4.14 kernel.Our analysis of the experimental data identified some outliers. We think some of the worst times, manifested as long echo replies, were caused by unrelated factors such as network packet loss. In the charts presented we, perhaps controversially, skip the bottom 1% of outliers in order to focus on what we think is the important data.Furthermore, we spotted a bug in SOCKMAP. Some of the runs were delayed by up to whopping 64ms.

Here is one of the tests: Values min:236.00 avg:669.28 med=390.00 max:78039.00 dev:3267.75 count:2000000 Values: value |-------------------------------------------------- count 1 | 0 2 | 0 4 | 0 8 | 0 16 | 0 32 | 0 64 | 0 128 | 0 256 | 3531 512 |************************************************** 1756052 1024 | ***** 208226 2048 | 18589 4096 | 2006 8192 | 9 16384 | 1 32768 | 0 65536 | 11585 131072 | 1 The great majority of the echo runs (of 128KiB in this case) were finished in the 512us band, while a small fraction stalled for 65ms. This is pretty bad and makes comparison of SOCKMAP to other implementations pretty meaningless. This is a second reason why we are skipping 1% of worst results from all the runs - it makes SOCKMAP numbers way more usable. Sorry. 2MiB blocks - throughput The fastest of our programs was doing ~15Gbps over one flow, which seems to be a hardware limit. This is very visible in the first iteration, which shows the throughput of our echo programs.This test shows: Time to transmit and receive 2MiB blocks of data, via our tested echo server. We repeat this 10k times, and run the test 10 times.