Cohere published a long-form engineering write-up of the megakernel serving approach behind North Mini Code: fusing the whole inference step into persistent GPU kernels to cut launch overhead and memory traffic, reporting 1.58x faster LLM serving on H100 (vendor-measured). The post covers the kernel design trade-offs and how it composes with the serving stack.
Cohere details its megakernel serving engine for North Mini Code
2026-09-08Verified announcementWATCH
Cohere published a long-form engineering write-up of the megakernel serving approach behind North Mini Code: fusing the whole inference step into persistent GPU kernels to cut launch overhead and memory traffic, reporting 1.58x faster LLM serving on H100 (vendor-measured). The post covers the kernel design trade-offs and how it composes with the serving stack.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
Persistent-kernel / megakernel serving is no longer a single-lab curiosity — DeepSeek's FlashMLA lineage, academic megakernel work and now Cohere's production deployment make it a multi-vendor engineering direction. For serving teams the post is a concrete reference implementation of what step-level fusion buys.
Evidence & related material
Technical details & scores
| Approach | megakernel / persistent-kernel serving engine for North Mini Code |
|---|---|
| Claim | 1.58x faster LLM serving on H100 (vendor-measured) |
| Context | same direction as DeepSeek FlashMLA lineage and academic megakernel work |
Technical impact 3
Engineering value 3
Adoption signal 2
Maturity 3
Verification cost 3
Risk 2