Community · Oct 29, 2026, 17:53 UTC

TKintegers K5 ternary weight format reports 3.2x prefill speedup over TorchAO FP8 on NVIDIA L4

The author reports K5 hit 3,402.85 tokens/s prefill versus 1,061.58 for FP8 and 26.75 versus 8.63 tokens/s decode on Qwen2.5-0.5B-Instruct. The results are for this model, workload and implementation only and have not been independently replicated.

• ModelQwen2.5-0.5B-Instruct
• Topicresearch
• Topicchips
• Topicopen-weights