Community
Developer releases open repo for self-distillation with EMA teacher under ZeRO-3 and LoRA
The repo uses a shared frozen backbone with two LoRA adapters, one trained student and one EMA teacher, and includes DeepSpeed configs for tight GPU memory. Tested on Qwen3.5 and ToolAlpaca.
Every outlet that ran it 1 article
r/MachineLearningCommunityLead
Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]