Research
Specter proposes quantized speculative offloading for faster MoE inference
The research system combines speculative decoding with expert offloading to ease PCIe bottlenecks. It reports an average 1.50x TPOT speedup, up to 1.88x, over the prior state-of-the-art under single-request inference.
Every outlet that ran it 2 articles
r/LocalLLaMACommunityLead
[Paper] Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
r/LocalLLaMACommunity
35B MoE running at 20.5 tok/s on an iPhone, without skipping experts