Hugging Face Releases Olmo-core 3 for Trillion-Parameter MoE Training

Training large artificial intelligence models demands substantial computing resources, which increases costs and energy consumption. This dynamic often places advanced model development out of reach for academic researchers and smaller laboratories. Mixture-of-experts (MoE) architectures offer a more efficient alternative by containing many parameter components without requiring every input to utilize all of them simultaneously. However, scaling MoEs introduces distinct communication and coordination challenges across GPU clusters, which can reduce the computational advantages of activating only a portion of the model per input.

To address these hurdles, Hugging Face has released Olmo-core 3, a redesigned open training framework for large language models. The updated system shifts from fully sharded data parallelism to a distributed data parallelism approach. This modification keeps experts resident on the GPUs and routes relevant data directly to them, thereby eliminating the need for repeated weight gathering. In preliminary tests on eight NVIDIA B300 GPUs, a 47-billion-parameter MoE achieved a throughput of 52,000 tokens per second per GPU, representing roughly 2.7 times the throughput of the earlier implementation.

Scaling and Optimization Techniques

Olmo-core 3 combines several methods to distribute large MoE architectures across hardware clusters efficiently. Expert parallelism spreads experts across GPUs, pipeline parallelism divides model layers across groups of devices, and a distributed optimizer shares optimizer states instead of duplicating them fully on every card. Furthermore, the framework incorporates rowwise expert parallelism and GPU-resident routing to minimize data rearrangement overhead, alongside grouped GEMM to execute small expert computations together.