Dust trains small transformers without backpropagation
Dust trains small transformers without backpropagation
2026-10-06Frontier watchTRIAL
Dust estimates transformer gradients using token-wise activation perturbations and only forward evaluations. The authors report performance competitive with backpropagation in small pretraining experiments and much better efficiency than weight-space evolution strategies. Public MIT-licensed code reproduces the estimator up to a 37.7M-parameter setup, but the authors say the method still needs orders-of-magnitude efficiency gains before becoming practical.
What this means for you
For training-research teams, Dust offers a concrete way to study non-backprop credit assignment and architectures with non-differentiable components. It is a research experiment, not a replacement for production pretraining.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Method | independent activation perturbations per token create a virtual population; reward-weighted noise estimates layer errors |
|---|---|
| Reported Scale | experiments up to 243M parameters and 1B-token checkpoints; public minimal implementation uses 37.7M parameters |
| Public Trial | single-GPU 1M-token run at population 256; paper configuration uses up to eight GPUs |
| Limitation | authors say orders-of-magnitude more efficiency is needed for practical replacement of backprop |
| License | MIT |
Technical impact 4
Engineering value 2
Adoption signal 2
Maturity 2
Verification cost 3
Risk 2