Ai2 released Olmo-core 3, a redesigned open MoE training stack that keeps experts resident on GPUs and combines data, tensor, pipeline and expert parallelism. Ai2 ran a 1.2T-parameter configuration with 58.36B active parameters per token across 512 NVIDIA B300 GPUs. The code is fully open, but the published throughput uses random routing and short measurements rather than sustained training.
Ai2 opens Olmo-core 3 for trillion-parameter MoE training
2026-10-01Verified announcementTRIAL
Ai2 released Olmo-core 3, a redesigned open MoE training stack that keeps experts resident on GPUs and combines data, tensor, pipeline and expert parallelism. Ai2 ran a 1.2T-parameter configuration with 58.36B active parameters per token across 512 NVIDIA B300 GPUs. The code is fully open, but the published throughput uses random routing and short measurements rather than sustained training.
What this means for you
For teams training their own MoEs, this is an inspectable, adaptable large-scale reference stack rather than a vendor-only system. Validate routing balance, checkpoint recovery, communication share and sustained throughput on 8–32 existing GPUs before scaling out.
Evidence & related material
Technical details & scores
| Architecture | DDP-based MoE stack with expert, tensor and pipeline parallelism |
|---|---|
| Largest Test | 1.2T total parameters, 58.36B active per token, 512 NVIDIA B300 GPUs |
| Peak | 858 TFLOP/s/GPU in short random-routing tests |
| Expert Scaling | 8 to 128 experts, four selected per token; 6B to 47B total capacity with under 5% throughput loss in one benchmark |
| Precision | MXFP8 support |
| License | Apache-2.0 code |
Technical impact 4
Engineering value 4
Adoption signal 2
Maturity 3
Verification cost 5
Risk 3