Cohere details its megakernel serving engine for North Mini Code

Cohere published a long-form engineering write-up of the megakernel serving approach behind North Mini Code: fusing the whole inference step into persistent GPU kernels to cut launch overhead and memory traffic, reporting 1.58x faster LLM serving on H100 (vendor-measured). The post covers the kernel design trade-offs and how it composes with the serving stack.

Persistent-kernel / megakernel serving is no longer a single-lab curiosity — DeepSeek's FlashMLA lineage, academic megakernel work and now Cohere's production deployment make it a multi-vendor engineering direction. For serving teams the post is a concrete reference implementation of what step-level fusion buys.