Research · Oct 30, 2026, 10:59 UTC

Soft-MTP paper proposes a fix for noisy multi-token prediction gradients

The arXiv paper reports Soft-MTP, a one-line change to multi-token prediction training that removes a noisy gradient residual, improving perplexity and speculative decoding from 167M to 1.2B parameters with no added parameters.

• Topicresearch
• Topicbenchmark