A controlled study pretrains attention, Mamba and hybrid 3B block-diffusion models on the same 300 billion tokens and exposes them through one cached decoding interface. Only the Mamba state cache stays constant with context length. At 256K tokens it uses about 7.5 GB and 6.8 ms per step, versus 82 GB and 29 ms for attention, yielding 2.6 times single-stream and 14 times aggregate throughput. Mamba and hybrid models retain retrieval at 8–16 times training length while attention collapses at twice the training length, with no measured quality loss.
Constant-state cache makes block-diffusion context length independent
2026-09-14Frontier watchWATCH
A controlled study pretrains attention, Mamba and hybrid 3B block-diffusion models on the same 300 billion tokens and exposes them through one cached decoding interface. Only the Mamba state cache stays constant with context length. At 256K tokens it uses about 7.5 GB and 6.8 ms per step, versus 82 GB and 29 ms for attention, yielding 2.6 times single-stream and 14 times aggregate throughput. Mamba and hybrid models retain retrieval at 8–16 times training length while attention collapses at twice the training length, with no measured quality loss.
Availability: no public checkpoints or implementation repository linked
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams exploring diffusion language models, the cache architecture can erase much of their long-context serving penalty. The result is expensive but well controlled across equal-scale pretraining runs. Without released checkpoints or serving code, it remains a research direction to watch rather than a deployable option.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Training | three 3B block-diffusion models, 300B tokens each, shared data/tokenizer/schedule/objective |
|---|---|
| Architectures | full attention, bidirectional Mamba-2, and hybrid with attention every five Mamba layers |
| Cache Scaling | Mamba state O(1) in sequence length; attention KV cache O(L) |
| At 256k | Mamba about 7.5 GB and 6.8 ms/step; attention 82 GB and 29 ms/step |
| Throughput | Mamba 2.6x single-stream and 14x aggregate throughput at 256K |
| Availability | no public checkpoints or implementation repository linked |
Technical impact 4
Engineering value 3
Adoption signal 1
Maturity 1
Verification cost 5
Risk 2