[Tech] Why MLA and MTP Fight Each Other: Attention Through Arithmetic Intensity

I was reading this post by Su Jianlin when one sentence stopped me: Besides the KV cache, decoding now has another variable — MTP, or speculative decoding, whose idea is to trade compute for speed.

But MLA behaves during decoding like an MQA with head_dims=512+, and has already consumed most of the compute up front, so “MLA+MTP” tends to lose out. My first reaction was: what does that mean? Why would MLA “consume compute up front” during decode? And why should it conflict with MTP that its decode FLOPs come out equivalent to a head-dim-512+ MHA?