Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context? (arXiv 2609.08574, Wed 9 Sep digest) builds SinkProbe, a diagnostic suite measuring sink mass, massive activation, position-resolved recall and the recency gap, and applies it to four small models that differ only in how they mix tokens — including gated attention (which cut first-token attention from 46.7% to 4.8%) and Kimi K3's gated attention + Kimi Delta Attention + Attention Residuals behind a one-million-token window, eight times past the range where these diagnostics had been reported. The paper maps where each mechanism holds and where sinks and positional-recall pathologies reappear at extreme range.
Hybrid attention mechanisms do not fully fix attention sinks at 1M tokens
2026-09-10Frontier watchWATCH
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context? (arXiv 2609.08574, Wed 9 Sep digest) builds SinkProbe, a diagnostic suite measuring sink mass, massive activation, position-resolved recall and the recency gap, and applies it to four small models that differ only in how they mix tokens — including gated attention (which cut first-token attention from 46.7% to 4.8%) and Kimi K3's gated attention + Kimi Delta Attention + Attention Residuals behind a one-million-token window, eight times past the range where these diagnostics had been reported. The paper maps where each mechanism holds and where sinks and positional-recall pathologies reappear at extreme range.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
The open-weight wave is shipping 262k-1M-token hybrid-attention models, and serving teams design prefix caching and eviction around assumptions about what attention does at those lengths. SinkProbe turns those assumptions into measurable quantities: check whether recall is actually uniform across the advertised window before committing cache and memory budgets.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2609.08574, Wed 9 Sep 2026 digest |
|---|---|
| Diagnostics | sink mass, massive activation, position-resolved recall, recency gap |
| Models | four small models differing only in token-mixing, incl. gated attention and Kimi K3's gated + KDA + Attention Residuals at 1M |
Technical impact 3
Engineering value 3
Adoption signal 2
Maturity 2
Verification cost 3
Risk 1