Disaggregated Quantization: Specializing LLM Prefill and Decode Formats Cuts TTFT by 1.78x

Background and the Problem

/ Background & Pain Points LLM inference inherently exhibits opposing physical bottlenecks across its two execution phases: prefill is compute-bound over thousands of prompt tokens, demanding maximum Tensor Core arithmetic throughput, while autoregressive decode is memory-bandwidth-bound, constrained by memory traffic per token. Modern quantization schemes enforce uniform formats across both phases, inevitably sacrificing either compute efficiency during prefill or memory bandwidth during decode.

Architecture and How It Works

/ Architectural Highlights Researchers from ISTA and Neural Magic formulated Disaggregated Quantization (DQ):

1. Phase-Specialized Arithmetic and Weights: Deploys compute-native NVFP4 weights to maximize arithmetic throughput during prefill, paired with 2-3 bit ultra-compact weights to minimize memory bandwidth consumption during decode;
2. Activation Decoupling on Decode: Demonstrates that eliminating activation quantization strictly during decode restores task accuracy without increasing serving latency;
3. Offloaded Disaggregated Prefill (ODP): For memory-constrained single-GPU setups, ODP streams specialized prefill weights asynchronously from NVMe SSDs, completely overlapping I/O transfer latency with prompt execution and freeing memory immediately for decode caches.

Benchmarks and Measured Results

/ Benchmark & Evaluation Extensive evaluations across Qwen 3 and Gemma 3 on llama.cpp and vLLM reveal breakthrough performance:

• Time-to-First-Token (TTFT): Slashes prefill latency by 1.78x over standard weight-only baselines at 8K prompt length on Qwen3.8-27B;
• Accuracy Recovery in Low Bits: Adding an NVFP4 prefiller to a 1-bit GGUF decoder boosts MMLU-Pro by +32.5 points and MMMU-Pro by +35.3 points without modifying the decode checkpoint;
• Trillion-Parameter Scale: Empirically validates shared-weight format disaggregation up to 2.8 trillion parameters under disaggregated serving configurations.

Getting Started for Developers

/ Developer Practical Guide

• Technical Reference: Detailed proofs and profiles are available in arXiv preprint 2609.26333;
• vLLM Integration: Ideal for vLLM disaggregated serving clusters by provisioning NVFP4 on Prefill workers and 2-3 bit quantization on Decode workers;
• Edge Deployment: ODP streaming enables running 27B-70B models with long-context windows on single consumer GPUs with NVMe storage.