Research
Developer shares a 2-bit residual speculative decoding idea for models that exceed GPU memory
The idea keeps a 2-bit copy of a model in VRAM and the remaining 6 to 8-bit residual weights in system RAM, using the 2-bit copy as a draft. The author reports quality and acceptance numbers from PyTorch tests, with no tokens-per-second results yet.