Autoregressive large language models generate one token at a time, leaving GPUs idle and making inference slow, especially for long Chain‑of‑Thought reasoning. Standard speculative decoding uses a small draft model that still proposes tokens sequentially, limiting real‑world speedups to about 2–3×. DFlash changes this by using a lightweight block diffusion model as the drafter. Instead of producing tokens one by one, DFlash creates an entire block of tokens in a single forward pass. The large target model then verifies the whole block in parallel, keeping the output lossless.
Because the draft cost stays flat as the block grows, DFlash can employ a deeper, more expressive draft network without adding latency. It injects target hidden states into every draft layer’s key and value projections, allowing the accepted length to grow with draft depth. In experiments on Qwen3‑8B, DFlash delivers up to 6.08× lossless single‑stream speedup, outperforming EAGLE‑3’s 2× average. On NVIDIA Blackwell GPUs serving gpt‑oss‑120b, throughput rises up to 15× at a fixed interactivity target, about 1.5× higher than EAGLE‑3 under the same conditions.
Use cases include coding agents that need fast interactive replies, reasoning models where long Chain‑of‑Thought traces dominate generation time, and high‑throughput serving environments where latency savings translate directly into lower cost. Adoption is simple: replace the speculative config with a DFlash checkpoint in vLLM or call spec_generate from the Transformers library; no application refactor is required.
#AI #ML #LLM #Inference #Performance #Productivity