Profiling

In the previous article we took apart the reflect package and found that its magic is mostly the compiler leaving very good notes — type descriptors frozen into read-only data at build time, and a package that knows how to walk them. The whole article was about reading metadata that was already sitting in memory before main even started.Today we shift the perspective. Profiling is the runtime catching your program in motion — sampling what it’s doing and where it’s spending its time, then accumulating that into something you can open with go tool pprof. The heart of every sample is a call stack, produced with the same unwinder we saw in the Stacktraces article . So profiling is really live moment-catching sitting on top of build-time stack-reading.But Go doesn’t have just one type of profile — it provides five: CPU, heap, block, mutex, and goroutine. At first they look like five unrelated subsystems, but they all share the same skeleton, and once you see it, the whole thing collapses into one idea repeated five ways. (A sixth, a goroutine-leak profile, is on its way in Go 1.27 — Alex Rios has a great series of posts digging into it — but we’ll stick to the five that ship today.) Let’s start there.The deepest similarity between all five profiles is the thing you actually walk away with: the file. No matter which profile you collected, what lands on disk is the same format — and it’s surprisingly simple: the pprof profile, a gzip-compressed protocol buffer . It’s not even Go-specific; it’s the same format Google’s C++ profiler (gperftools) emits, and one go tool pprof shares with the wider pprof ecosystem. So before we look at how each profile is collected, let’s look at what they’re all collected into.At the top level it’s one Profile message, and the part that matters is just a handful of repeated fields:message Profile { repeated ValueType sample_type = 1; // what each number in a sample means repeated Sample sample = 2; // the actual data repeated Mapping mapping = 3; //

How the data gets thereWhichever profile you’re collecting, one thing is constant: somewhere along the way the runtime captures a call stack with the unwinder, and at the very end it all comes out as the pprof file we just saw. How a stack gets from the one to the other, though, is not the same across the five — and that difference is the real story.The five sort into three collection models:CPU records asynchronously. A signal interrupts a running thread, the handler captures the stack and logs it into a ring buffer, and a separate background goroutine drains that buffer as it fills. A genuine streaming pipeline.Heap, block, and mutex record in place. When the event fires, the stack is hashed into a long-lived table of per-stack records and the matching record’s counters are bumped right there. Nothing streams and nothing drains in the background — the data just sits and accumulates, and the profile is assembled on demand, the moment you ask for it.

Goroutine doesn’t record during execution at all. There’s no trigger and no buffer. When you ask for the profile, the runtime walks every live goroutine’s stack right then, in one snapshot.So the thing that really changes from profile to profile isn’t a single knob — it’s the whole collection model. Let’s walk the five, grouped by the model they use.CPU profiling: the signal-driven oneThis is the most distinctive of the five because its trigger comes from outside the program entirely: the operating system interrupts a running thread with a signal, roughly 100 times a second, and asks “what were you doing right now?”When you call pprof.StartCPUProfile, the runtime sets up the timers it needs to interrupt the running threads — so that time spent burning CPU gets sampled — and starts a background goroutine to collect the results (src/runtime/pprof/pprof.go:888). Hold on to that background goroutine — we’ll see in a moment why it matters.Arming the timers is the easy part — the interesting bit is what happens when one of them fires.Catching a tickOn every timer tick, the program gets interrupted: the thread is yanked out of whatever it was doing, mid-instruction, to run the handler, and then it’s expected to pick up exactly where it left off. That puts the handler in a very awkward spot — it can’t allocate memory, and it can’t take an ordinary lock (src/runtime/proc.go:5748).The handler then captures the call stack with the unwinder, fills in the sample’s values and your labels (the ones you can attach with pprof.Do), and stores it. But remember, it can’t allocate or lock — so it needs somewhere to put the sample that demands neither. That’s exactly what it has: a structure designed for this constraint, a lock-free, preallocated ring buffer (src/runtime/profbuf.go:91), which can be appended to without allocating and without taking a lock. Being a ring buffer, though, it’s finite: if it ever fills up faster than it’s emptied, there’s no room for the next sample — so rather than make the interrupted thread wait for space, the buffer simply drops samples and counts how many it lost (losing a sample is fine; freezing a thread is not).

Which sounds like a problem — nobody wants to drop samples on the floor. And that’s exactly where the background goroutine we started earlier earns its keep.Its job is to drain the ring buffer as fast as it can — not to write anything out yet, but precisely so the ring rarely fills up and samples aren’t lost. Each batch it pulls out is folded into an in-memory map keyed by call stack (the profMap), where identical stacks are deduplicated and consolidated on the spot, bumping the count on the matching entry. The output file isn’t touched during the session at all. Only when you call StopCPUProfile, once the last samples have drained, does the builder make a single pass over that map and convert it into the pprof format — emitting every sample, location, function, and string in one go — and close the file you open with go tool pprof.Let’s see the whole thing visually:As we can see here, roughly 100 times a second the interrupted threads drop call stacks into the ring buffer, the background collector drains those and aggregates them into the profMap, and eventually — when you stop profiling — it’s all transformed and written out to the pprof file.We’ve followed a single stack all the way from interrupt to file. But a stack on its own isn’t a profile — what makes it one is the numbers that ride alongside it.The values each sample carriesThe whole point of a CPU profile is to know how much CPU each stack uses — and that’s where the value[] slot on each sample comes in. A CPU sample carries two values: a plain count of how many times the timer caught this exact stack, and an estimate of CPU time in nanoseconds. The neat part is that the runtime never actually measures time — it only counts ticks. The nanosecond figure is just the count multiplied by the sampling period, and since the timer fires at 100 Hz, each tick stands for about 10 ms. So a stack caught 50 times is credited with roughly 500 ms, and “this function used 3.2 seconds” is really ~320 ticks times 10 ms each (src/runtime/pprof/proto.go:348).CPU profiling’s trigger came from outside the program. The next profile’s comes from deep inside it.

Heap profiling: the allocator-sampled oneHeap profiling shares its bones with CPU — the unwinder captures a stack, and the result is the same pprof file — but it belongs to the in-place model, not the streaming one. A trigger fires, the stack is folded into a long-lived table right there, and the profile is assembled only when you ask for it. So we’ll focus on what makes it different, and three things do: where the samples accumulate, what triggers one, and what data each one collects. Let’s take them in that order, starting with the storage (which the block and mutex profiles share too).A table of per-stack bucketsCPU profiling needed that lock-free ring buffer because it records from inside a signal handler, where it can neither allocate nor take a lock. Heap profiling records from inside the allocator instead — still a delicate place, but not a signal handler — so it can take a lock. And being able to lock is what lets it do something the CPU handler couldn’t: aggregate on the spot. So instead of a streaming buffer it uses a hash table of records keyed by the call stack, called buckets (src/runtime/mprof.go:75). Identical stacks collapse into the same record: a million allocations at the same line in your code don’t create a million entries, they all find the one bucket for that stack and bump its counters.Let’s see how this changes the lifecycle of collection:As we can see in the image, allocations are aggregated directly into the bucket table as they happen — no draining goroutine and no Start/Stop window, the bucket table just lives in the runtime and is always accumulating. Then, whenever the profile is required, it’s exported in a single pass to the pprof file.That’s where the samples accumulate. Now, what decides when to take one?The trigger: the allocator, sampledThe trigger is the allocator itself: every allocation goes through runtime code, so there’s nothing to interrupt — the runtime just notices and records the sample right there (profilealloc, src/runtime/malloc.go:2238).Recording every allocation would be far too expensive, so it only samples some of them — on average one for every MemProfileRate bytes allocated, 512 KiB by default.