Research · Oct 28, 2026, 00:00 UTC

Study tests whether truncated backpropagation hurts looped transformer scaling

Across nearly 600 training runs, a fitted scaling law favors full backpropagation through time over truncation when loop counts are jointly optimized. Two fully trained loops were identified as the best looped setting, with about 27% fewer non-embedding parameters and 8% more training compute than…

• Topicresearch