Fleet: Hierarchical Task-based Abstraction for Megakernels on Multi-Die GPUs

View PDF HTML (experimental) Abstract:Modern GPUs adopt chiplet-based designs with multiple private cache hierarchies, but current programming models (CUDA/HIP) expose a flat execution hierarchy that cannot express chiplet-level locality or synchronization. This mismatch leads to redundant memory traffic and poor cache utilization in memory-bound workloads such as LLM inference. We present Fleet, a multi-level task model that maps computation to memory scopes. Fleet introduces Chiplet-tasks, a new abstraction that binds work and data to a chiplet and enables coordination through its shared L2 cache. Wavefront-level, CU-level, and device-level tasks align with existing abstractions, while Chiplet-tasks expose a previously unaddressed level of the hierarchy. Fleet is implemented as a persistent kernel runtime with per-chiplet scheduling, allowing workers within a chiplet to cooperatively execute tasks with coordinated cache reuse. On AMD Instinct MI350 with Qwen3-8B, Fleet achieves 1.3-1.5x lower decode latency than vLLM at batch sizes 1-8 through persistent kernel execution and per-chiplet scheduling. At larger batch sizes, cooperative weight tiling increases L2 hit rate (from 12% to 54% at batch size 32 and from 39% to 61% at batch size 64), reducing HBM traffic by up to 37% and delivering 1.27-1.30x speedup over a chiplet-unaware megakernel baseline.

Subjects: Hardware Architecture (cs.AR) Cite as: arXiv:2604.15379 [cs.AR]   (or arXiv:2604.15379v1 [cs.

AR] for this version)   https://doi.org/10.48550/arXiv.2604.15379 arXiv-issued DOI via DataCite Submission history From: Sangeeta Chowdhary [view email] [v1] Wed, 15 Apr 2026 21:49:03 UTC (68 KB)