Goroutine Leak Profiles - The Go Programming Language

Go’s concurrency features are powerful and easy to use, but that same ease can sometimes lead even seasoned developers to make mistakes. Fortunately, the Go ecosystem comes equipped with useful tools for debugging, e.g., the race detector, but even existing tools may miss some concurrency bugs, such as the topic of this article, the goroutine leak. Goroutines synchronize or exchange information via shared concurrency primitives, e.g., channels, locks, and wait groups. While communicating, goroutines often block on these primitives, as in, wait until some condition is met; ubiquitous examples include waiting to acquire a held mutex, or receive a message over a channel. Goroutines can also block on operating system operations, like reading from a network socket or a file. We may consider a goroutine leaked if it is blocked, but the conditions needed to unblock it can never be met. Over time, an accumulation of leaked goroutines degrades performance through excessive memory usage (by the leaked goroutines themselves or the memory they reference), as well as CPU usage from the garbage collector, especially if GOMEMLIMIT is in use. Goroutine leaks can be notoriously difficult to detect. In unit testing, the most significant breakthroughs include the open-source library goleak, which can instrument individual tests to signal any un-terminated goroutines after the test wraps up as suspicious. Similarly, Go 1.25 introduced the synctest package to the standard library; it can significantly improve the quality of unit tests in concurrent code by giving Go developers more control over the ordering of concurrent events in order to reliably test hard-to-reproduce scenarios. Unfortunately, neither approach can check for goroutine leaks in production systems, especially at larger scales, which may behave in ways unaccounted for by tests. Goroutine profiles are a rudimentary way to check for operations that block too many goroutines, or analyze growth trends. However, goroutine profiles cannot distinguish between goroutines which are leaked, and those which are temporarily blocked in high numbers by design, e.g., as caused by increased traffic in a microservice. Likewise, leaks which are low in number may slip by undetected for many years. Go 1.27 introduces the goroutine leak profiler, a flexible and lightweight mechanism for finding goroutine leaks in running Go programs, including production systems. Unlike previous approaches, which require human analysis, this mechanism is precise and generates little-to-no false positives. The trade-off is that it is limited to a subset of goroutine leaks: goroutines permanently blocked on channels or primitives in the sync package. Luckily for us, this already covers a very large subset of goroutine leaks, as we’ll see in our examples. In the following sections, we showcase how to use the feature, followed by some additional examples of detectable leaks, and a description of the underlying implementation and trade-offs. Example: concurrent workers Consider a function that processes work items concurrently: type result struct { res workResult err error } func processWorkItems(ws []workItem) ([]workResult, error) { // Process work items in parallel, aggregating results in ch. ch := make(chan result) for _, w := range ws { go func() { res, err := processWorkItem(w) ch <- result{res, err} }() } // Collect the results from ch, or return an error if one is found. var results []workResult for range len(ws) { r := <-ch if r.err != nil { // This early return may cause goroutine leaks. return nil, r.err } results = append(results, r.res) } return results, nil } Because ch is an unbuffered channel, each worker goroutine blocks when sending its result until the main goroutine receives from the channel. If processWorkItems returns early due to an error, the receiving loop terminates, and all remaining sender goroutines block forever. This example is emblematic of a common mistake discovered in real Go programs, including Uber production services. Let’s see how we can find these leaks by using the new goroutine leak profiler. Debugging with the goroutine leak profiler The profile is available through the runtime/pprof package, as the goroutineleak profile type, or by installing the profile handlers defined by the net/http/pprof package. If you already have net/http/pprof set up in your service, then you don’t need to do anything else! The profile will be automatically made available for collection at the /debug/pprof/goroutineleak endpoint on whatever host and port the handlers are installed. Let’s put our concurrency bug in context and set up the net/http/pprof package. This way, you can try it yourself! package main import ( "errors" "log" "net/http" _ "net/http/pprof" "time" ) type workItem int type workResult int func processWorkItem(w workItem) (workResult, error) { time.Sleep(10 * time.Millisecond) if w == 5 { return 0, errors.New("simulated error") } return workResult(w * 2), nil } type result struct { res workResult err error } func processWorkItems(ws []workItem) ([]workResult, error) { ch := make(chan result) for _, w := range ws { go func() { res, err := processWorkItem(w) ch <- result{res, err} }() } var results []workResult for range len(ws) { r := <-ch if r.err != nil { return nil, r.err } results = append(results, r.res) } return results, nil } func main() { // Start pprof server go func() { log.Println(http.ListenAndServe("localhost:6060", nil)) }() // Repeatedly trigger the leak for { items := []workItem{1, 2, 3, 4, 5, 6, 7, 8, 9, 10} _, err := processWorkItems(items) if err != nil { log.Printf("Error processing items: %v", err) } time.Sleep(time.Second) } } Build the program above, then run it: $ go build -o leaky $ ./leaky Collecting the profile It won’t take long for the program to start accumulating leaks, which you can then view by using the web UI at http://localhost:6060/debug/pprof. Alternatively, you can collect the goroutine leak profile using curl, and then examine it with go tool pprof: $ curl http://localhost:6060/debug/pprof/goroutineleak > leak.prof $ go tool pprof leak.prof Type: goroutineleak Time: 2026-03-01 13:19:49 UTC Entering interactive mode (type "help" for commands, "o" for options) (pprof) list processWorkItems Total: 116 ROUTINE ======================== main.processWorkItems.func1 in .../main.go 0 116 (flat, cum) 100% of Total . . 31: go func() { . . 32: res, err := processWorkItem(w) . 116 33: ch <- result{res, err} . . 34: }() The profile reveals the goroutines leaked at ch <- result{res, err} (line 33), pinpointing the culprit operation. Notably, the longer the program is running, the larger the number of leaked goroutines. Addressing the leak This leak can be simply fixed by giving ch a buffer: ch := make(chan result, len(ws)) This allows all the work item goroutines to send a message without blocking in the event of a premature return of processWorkItems. We list more real-world examples in this section. Implementation This section is for those interested how leak detection works under the hood of the goroutine leak profiler. For details strictly pertaining to performance overhead and limitations, skip ahead to this section. Core concept Let’s start with an initial observation: if a goroutine is blocked over some concurrency primitive that no other goroutine has access to (in this case, via a reference in memory), then it is obviously leaked. This is already a strong lead, we can generalize it further into a definition for when a goroutine is not leaked, a property we term as liveness. We formally define liveness, an inductive property as follows: A goroutine is live if: it is not blocked by a concurrency primitive, or at least one concurrency primitive that blocks it is referenced by another live goroutine. In the trivial case, goroutines which are not blocked are obviously not leaked. In the inductive case, the underlying assumption is that any goroutine which is not leaked may eventually use concurrency primitives it references to unblock any other goroutines blocked by those primitives. To find all live goroutines, we start from the obviously live unblocked goroutines and trace any references they hold, i.e., through their local variables, to find the concurrency primitives they have access to. We then incrementally include any goroutines blocked over those primitives as live, and repeat the process until no additional live goroutines are discovered. Fortunately for us, the Go runtime already computes memory reachability through the garbage collector (GC), so the next step is to adapt the GC to suit our purposes. You can quickly compare the two GCs with the following diagrams: A complete overhaul of the GC is not necessary. The Go runtime uses a concurrent tri-color mark-and-sweep garbage collector, (now with the Green Tea variant!), so its MO already neatly aligns with our goals. Only a few key changes are needed: In the initial phases, the regular GC marks all goroutines (and global variables) as reachable, such that they would never be considered garbage, i.e., they are mark roots. We change it to instead only include unblocked goroutines, since these are guaranteed to be live. This is followed by the marking phase, where the GC traces objects referenced (transitively) by the mark roots, and marks them as usable memory. Even though we do not modify this phase directly, the changes in step 1. implicitly ensure that the GC only marks memory referenced by live goroutines. The marking phase is finalized by inspecting all the blocked goroutines not included as mark roots in step 1. If a goroutine is blocked by at least one concurrency primitive that has been marked in step 2., it is added as a mark root, and the GC resumes the marking phase from step 2. This coincides with the inductive step in the definition of liveness. Once all live goroutines have been discovered, any goroutine which has not been added as a mark root has its status set to