Nobody Is Listening on Port 8125

There's a Fedora box on my desk named stick. An app on it has been firing StatsD datagrams at 127.0.0.1:8125 ten times a second for about a week now, the way apps have done since Etsy taught everyone to in 2011. Nothing is listening on 8125. $ ss -lun | grep 8125 $ Grafana has the request rate anyway. This is the first post in a series where we go down the list of exporters everyone runs next to Prometheus and replace them, one at a time, with a yeet script. The pitch for the series is short: you do not need fifty exporters in six languages, each with its own port, its own config format, and its own opinion about vendoring. You need one binary that can read the kernel. We're starting with statsd_exporter because it's the one where the replacement stops being the same thing and turns into something kind of better. TL;DR statsd_exporter is a translator. Your app pushes StatsD lines at it over UDP, it holds them as Prometheus metrics, Prometheus pulls. It's maintained by the Prometheus project itself, so this is an official piece of the stack we're pulling out. We replaced the listener with an eBPF program that watches port 8125 at the traffic control layer of every interface, and a JavaScript decoder that turns the lines into Prometheus families. The app keeps doing fire-and-forget UDP. Nothing changes on the app. Nothing gets installed on the host except yeet. And the metrics show up whether or not anyone is there to receive the datagrams. You'll learn: Why a StatsD receiver is just a packet sniffer with a socket in front of it. The 165 lines of C that read the wire, and why the same mechanism works for any line protocol you point it at. Redis, memcached and Graphite are later posts in this series and they reuse this probe unchanged. How yeet:telemetry turns a script into a Prometheus target without the script ever touching a socket. What you give up, because you do give up a couple things. The whole thing is open source. To run it on your own boxes: Ok so what does statsd_exporter actually do Less than you'd think. It binds a UDP socket, reads datagrams, splits them on newlines, and parses lines that look like this: app.requests:1|c app.queue.depth:22|g app.request.time:143|ms app.users:20|s app.errors:1|c|@0.1|#service:api,region:us Then it keeps a counter, gauge, histogram or set per name, runs the names through a YAML mapping file so your dots turn into labels, and serves the whole thing on :9102/metrics. That's it. That's the exporter. It's a perfectly good exporter and I've run it for years without thinking about it once, which is about the nicest thing you can say about infrastructure. But look at what it eats. UDP datagrams. On a well-known port. In a plaintext line format. Usually on the same host as the app. By the time that socket gets the bytes, the kernel has already had them, looked at them, and decided where they go. The listener is the second reader of every packet. So we asked the first one. 165 lines of C The kernel half is a tcx program clipped onto the ingress and egress hooks of every interface that's up, loopback included. It does three things: find the transport header, check if either port is one we care about, copy a bounded chunk of the payload into a ring buffer. Done. /* The ports worth capturing, filled by the script. */ struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 64); __type(key, __u16); __type(value, __u8); } ports SEC(".maps"); static __always_inline int wanted(__u16 sport, __u16 dport) { return bpf_map_lookup_elem(&ports, &sport) != NULL || bpf_map_lookup_elem(&ports, &dport) != NULL; } static __always_inline int handle(struct __sk_buff *skb, __u8 dir) { /* ... walk Ethernet or raw IP, then IPv4 or IPv6, to the L4 header ... */ __u16 sport = 0, dport = 0; bpf_skb_load_bytes(skb, l4, &sport, 2); bpf_skb_load_bytes(skb, l4 + 2, &dport, 2); sport = bpf_ntohs(sport); dport = bpf_ntohs(dport); if (!wanted(sport, dport)) return TCX_NEXT; /* ... TCP data offset or the 8-byte UDP header, then the payload ... */ struct wire_event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0); if (!e) return TCX_NEXT; e->ts = bpf_ktime_get_ns(); e->sport = sport; e->dport = dport; e->ifindex = skb->ifindex; e->dir = dir; e->total_len = plen; e->captured = cap; if (bpf_skb_load_bytes(skb, poff, e->data, cap) < 0) { bpf_ringbuf_discard(e, 0); return TCX_NEXT; } bpf_ringbuf_submit(e, 0); return TCX_NEXT; } That ifindex is there for a reason. A packet on loopback goes through the egress hook on lo, and then, because the kernel just hands it right back to itself, through the ingress hook on lo. Same bytes, two events. TCP you can dedupe on the sequence number. UDP has no sequence number. So on loopback the collector keeps the egress copy and drops the other one, and every datagram counts exactly once: if (ev.dir === DIR_INGRESS && ev.ifindex === lo) return; Look at the ports map too. There's no port number anywhere in the C. The script fills the map at startup from its arguments, and 8125 is contentionally what statsd used, but there's no reason why it couldn't be anything else. Because the kernel side has no idea what StatsD is, the same mechanism covers anything else that speaks a line protocol on a known port. It matches a port and copies up to 512 bytes. Parsing is a JavaScript problem, and JavaScript is cheap. The collector on stick already runs the Redis, memcached and Graphite decoders off this one probe, each with its own flag, and those get their own posts. The next protocol costs a port number and a decoder. Everything else stays in the kernel. ACKs, TLS, your Zoom call, none of it ever crosses into userspace. And the program returns TCX_NEXT on every path, so it never drops or rewrites a packet. It reads. That's the whole job. yeet loads it from a script with yeet:bpf. The attach target is whatever the system graph says is up: import { BpfObject, HashMap, RingBuf } from "yeet:bpf"; const { data } = await yeet.graph.query(`{ network_interfaces { index is_up } }`); const tcx = { kind: "tcx", ifindex: data.network_interfaces.filter((i) => i.is_up).map((i) => i.index) }; const control = await new BpfObject({ exe: "../bpf/bin/wire.bpf.o", base: import.meta.dirname }) .bind("events", { kind: "ringbuf", btf_struct: "wire_event" }) .bind("ports", { kind: "hash" }) .attach("on_ingress", tcx) .attach("on_egress", tcx) .start(); const ports = new HashMap(control, "ports"); for (const port of [8125, 8126]) await ports.update(port, 1); // from yeet.args, really await new RingBuf(control, "events").subscribe(onEvent, () => dropped.inc()); btf_struct: "wire_event" is the entire deserializer. The daemon reads the struct layout out of the object's BTF and hands onEvent a plain object with sport, dport, data and friends already on it. Same trick in the other direction for the map: update(8125, 1) is a __u16 key and a __u8 value because the C said so. We did not write a byte parser. We're never going to write a byte parser. The decoder is boring on purpose Here's the StatsD half with the plumbing cut out: request(ev, bytes) { this.datagrams.inc(); for (const line of latin1(bytes).split("\n")) { const m = /^([^:]+):([^|]+)\|([a-z]+)(?:\|@([0-9.]+))?(?:\|#(.*))?$/.exec(line.trim()); if (!m) { this.lines.labels({ status: "invalid" }).inc(); continue; } const [, rawName, rawValue, type, rate, tags] = m; const name = rawName.replace(/\./g, "_"); const labels = parseTags(tags); // #service:api,region:us const sample = 1 / (Number(rate) || 1); // @0.1 counts for ten this.#apply(name, type, rawValue, Number(rawValue), labels, sample); } } #apply is a switch on the type letter. c bumps a counter by value * sample. g sets a gauge, or nudges it if the value came with a sign. ms, h and d observe a histogram, with milliseconds scaled to seconds. s throws the value in a set and publishes the size. That's the StatsD spec. All of it. It fits on a napkin. Families get created the first time a name shows up, which is the part statsd_exporter makes you do in YAML ahead of time: #family(name, type, labels) { const key = `${type}:${name}`; let f = this.#series.get(key); if (!f) { const opts = { help: `StatsD ${type} ${name}, read off the wire.`, labels: labels.names }; if (type === "histogram") opts.buckets = LATENCY_BUCKETS; f = this.telemetry[type](name, opts); this.#series.set(key, f); } return f; } this.telemetry is a Telemetry registry. The first app.errors line that shows up with #service:api,region:us on it locks in the label set for app_errors_total. A later line with different tags gets counted under statsd_exporter_lines_total{status="rejected"} instead of taking the collector down. Same rule the real exporter has, we just didn't make you write it in a config file. Ok but the isolate can't open a socket Correct. The isolate a yeet script runs in cannot bind a port. It also can't fetch, which we went on about last time and still think is the best thing about it. The only listeners anywhere in this picture belong to the daemon, and the daemon speaks HTTP and WebSocket and that's the list. So how does Prometheus scrape the thing? Two pieces. The collector runs as a SharedWorker and calls telemetry.serve(), which answers snapshot requests over a message port. The scrape endpoint is a separate script, and it's tiny: import { render } from "yeet:telemetry"; import { emit } from "../../lib/emit.js"; import { pick, upAs } from "../../lib/snapshot.js"; // Everything the wire collector holds that is not another protocol's. const OTHERS = ["redis_", "memcached_", "graphite_"]; emit(await render({ worker: "../../lib/collectors/wire.js" }, ({ worker: w }) => ({ ...upAs(w, "statsd_exporter_up", "1 when the wire collector answered."), ...Object.fromEntries(Object.entries(w).filter(([k, f]) => k !== "yeet_worker_up" && !OTHERS.some((p) => k.startsWith(p)) && !/^Graphite/.test(f?.help ?? ""))), }))); render({ worker }) connects to the shared worker, asks for its registry,