Finding zombies in our systems: A real-world story of CPU bottlenecks

13 min readApr 15, 2026--Vaibhav Shankar; Staff Software Engineer | Raymond Lee; Staff Software Engineer | Chia-Wei Chen; Staff Software Engineer | Shunyao Li; Sr. Software Engineer | Yi Li; Staff Software Engineer | Ambud Sharma; Principal Engineer | Saurabh Vishwas Joshi; Principal Engineer | Charles-A. Francisco; Senior Engineer | Karthik Anantha Padmanabhan; Director, Engineering | David Westbrook; Sr. Manager, EngineeringOne day in early 2025, the Kubernetes platform team at Pinterest (PinCompute) got a ping from our partners on the ML platform team. Their Ray-based training jobs , which often take hours of computation on expensive GPU hardware, were crashing. Not every time, but often enough that it was becoming noticeable. Their logs indicated that their distributed training jobs were seeing intermittent loss of network connectivity, and that ultimately caused their jobs to crash. Their ask was simple:Why is this happening?Can you please make it stop?What started there led to a more than three-month-long investigation and a great lesson in profiling performance bottlenecks. Read on to learn from our fun story about CPU bottlenecks, AWS network drivers, and yes, how we discovered Zombies in our system!Background: Ray at PinterestAt Pinterest, Ray has risen as the backbone of our next-gen ML training and inference. Over the past few years, it has enabled us to scale systems, accelerate experimentation, and significantly boost the performance of models powering our diverse ML workloads.We have previously shared deep dives on our progress, including: Ray Infrastructure (provisioning ray cluster on in-house K8s clusters at scale [blog]), Batch Inference with Ray (scaling to hundreds of nodes [blog][talk]), Ray for Training (distributed dataloaders and throughput optimization [talk]), and Last-Mile Data Processing (reducing experimentation cycles [blog 1][blog 2]).Today, we run more than half of the offline ML workload company-wide on Ray, provisioning tens of thousands of Ray clusters per month, a feat made possible only by a robust Kubernetes environment.Network Model & ChallengesPress enter or click to view image in full sizeFigure 1: Ray architecture at PinterestWhat makes the network stability challenging lies in Ray’s unique network model.Ray operates as a highly “network-active” system. A Ray cluster generates constant, intensive inter-pod gRPC traffic that is fundamental to the cluster’s operation, with the following two distinct layers:Control Plane: Handles stateful operations, such as node health check, task submission, actor scheduling, and the maintenance of Object References.Data Plane: Handles the high-volume transfer of values within the Object Store. Our Large-scale ML training relies on this plane to move data rapidly between nodes.Because this traffic is highly distributed and latency-sensitive, the impact of network instability is often non-deterministic, manifesting across various components of Ray Cluster:Job Hanging: Caused by actor state corruption following brief network interruptions. [github issue]ObjectFetchTimedOutError / ObjectLossErrorActorDiedErrorNode failed the health check and crashed…All of these occurrences resulted in one common outcome: our Ray Training jobs would crash (some use cases with > 25% Success Rate drop), resulting in loss of expensive compute hours and significant slowdown in Model building and experimentation. After grinding for over a month seeking solutions for individual issues in the Ray stack, the ML Platform team realized it was necessary to turn our attention to look for more lower level network issues with our friends from the PinCompute team.Symptom 1: Network driver resetsAt Pinterest, our Kubernetes clusters are backed by AWS EC2 instances, which leverage the ENA Network driver (ref) as a standard traffic component. This Network driver works with AWS Elastic Network Interfaces (ENIs) and sets up receive and transmit queues for buffering packets. Our first symptom that something was wrong was identifying that whenever the ML training jobs failed with network connectivity issues, it correlated with a Network driver ‘reset’, as seen in our system logs.[] ena 0000:20:03.0 eth0: TX q 5 is paused for too long (threshold 5000000). Timesince last napi 6596000 usec. napi scheduled: 1[] ena 0000:20:03.0 eth0: napi handler hasn't been called for a long time but is scheduled# .... Bunch of stats excluded....[] ena 0000:20:03.0: ENA Large LLQ is disabled[] ena 0000:20:03.0: Device reset completed successfully, Driver info: Elastic Network Adapter (ENA) v2.11.0gFrom the reference docs:Q: What is [the] ENA device reset?A: ENA device reset is a self healing mechanism that is triggered when the driver detects unexpected device behavior. Example of such behavior could be an unresponsive device, missing keep-alive events from the device, Tx completions timeouts, netdev timeout etc. The device reset is a rare event, lasts less than a millisecond and might incur loss of traffic during this time, which is expected to be recovered by the transport protocol in the instance kernel.Ok, so the driver saw Tx threads paused for an extended period of time (hardcoded to 5s in AWS ENA Kernel drivers), and caused the device to be reset, which could cause some packet drops. A typical reason for resets was documented as CPU starvation, i.e, when the Network driver’s threads don’t get CPU time for several seconds. So perhaps something CPU intensive was starving out the Network driver threads?Symptom 2: CPU utilizationOur next observation was that some of the machines where we saw network resets exhibited high system CPU usage and that correlated nicely with the CPU starvation theory in the ENA documentation. We speculated that our training jobs were leveraging inefficient memory allocators and that was resulting in High page faulting.Press enter or click to view image in full sizeFigure 2: Page faults per second on impacted machinesWe did what many reasonable people would do:We tried using Huge pages (by turning on TransparentHugePages) to reduce page faulting.We experimented with more efficient memory allocators like jemallocWe tried to give the training jobs their own CPU cores by providing them CPU affinity via taskset.Out of desperation, we played with interrupt pinning for ENA drivers by steering network interrupts to other cores.Nothing worked. While we saw some drops in overall CPU utilization and page faulting from the memory allocators and huge pages settings, the network resets continued. They sometimes happened very early in a training job run and sometimes several hours into their execution. Across 100s of training job runs, it was hard to predict when exactly we’d see a network reset, if at all.One mitigation did work, albeit briefly and it’s everyone’s favourite IT crowd advice: Yes, we turned it off and on again. When we rebooted machines with high amounts of resets, they were able to support running ML jobs just fine.. that is until they weren’t. We clocked it at approximately one week of uptime, after which the network resets returned on the rebooted machines.Symptom 3: Availability zone differencesTo further understand the problem, the ML platform team started emitting metrics whenever an ENA reset was observed. Once the metrics were available, the team noticed something odd — the network resets were happening on machines in one AWS Availability zone only and all their jobs with identical parameters were running just great on other zones.Press enter or click to view image in full sizeFigure 3: Network resets per Availability ZoneThe PinCompute team runs zonal clusters (one Kubernetes cluster per Availability zone) but when the team looked at our cluster configurations across different zones, they seemed identical. They were running the same version of Kubernetes and the same system image. So, did we get a bad hardware batch!? We reached out to our excellent AWS support team and after several engagements, were convinced that the issue was definitely not on the AWS side. Their analysis was clear: there was something on our machines in the us-east-1a zone, which was heavily using the CPU and causing the network threads starvation. So why would one availability zone’s machines only exhibit this network reset behaviour?Profiling attempts: perf and mpstatWe decided it was time to stop with high level metrics and start profiling what was actually using the high amounts of CPU. Performance engineers know all about perf and its versatility. perf is a Linux profiler that can provide insights into ‘hot’ code paths and a call stack indicating CPU time spent by a particular process on a machine. Initially, our rudimentary snapshots of perf revealed the same suspected actors: Page faulting and some heavy computation from our ML jobs. However, this didn’t indicate CPU starvation all on its own.Press enter or click to view image in full sizeFigure 4: perf snapshot on an impacted machineWe realized that for CPU starvation to happen, it may take as little as one CPU core to be heavily utilized and block an unlucky network thread that was scheduled onto that core. Moreover, we realized that our GPU machines had 96 vCPU cores, which meant that an overall perf view told us very little about what was happening in each individual core.To address this, we used mpstat to get an overview of per core utilization on a per-second basis for an hour to identify if specific cores were using up large amounts of CPU. In our offline analysis, we found that sometimes, a single CPU core (in the following screenshot, CPU 39) was often using 100% of its system CPU for multiple seconds! This also correlated with when a network reset happened. We were finally closing in on the root cause!Press enter or click to view image in full sizeFigure 5: 100% System CPU utilization on a single core (Core 39) when profiled per second.Given these network resets were happening at unpredictable times and we lacked