Community
Hugging Face publishes a handbook on training Mixture-of-Experts models with Megatron-LM
The handbook covers a Megatron-LM runbook, MoE layer design choices and serving and RL, drawing on about 800k GH200 GPU-hours on the Alps cluster at CSCS. Its 55 figures, scripts and recommended flags are released alongside it.
Every outlet that ran it 1 article
r/MachineLearningCommunityLead