Tencent’s AngelSpec tackles a real‑world pain point: a single speculative drafter cannot serve both open‑ended chat and structured code/math workloads efficiently. In chat, many continuations are plausible, so long draft proposals are quickly rejected and waste compute. In code and math, token sequences are highly predictable, benefiting from longer, block‑parallel drafts that amortize verification cost.
AngelSpec resolves this by shipping two specialized drafters instead of one compromise.
1. An autoregressive multi‑token prediction (MTP) model trained on diverse conversation data, optimized for short, high‑entropy proposals.
2. A block‑parallel DFlash family model strengthened with code‑ and mathematics‑focused samples, built to exploit long, deterministic spans.
Training is aligned with inference through a Training‑Time Test scheme: the MTP block is shared across depths but unrolled autoregressively during training, using arg‑max predictions from the previous depth as input (instead of ground truth). The target backbone and its language‑model head stay frozen, and training uses target‑model rollout so the drafter learns from the exact token choices and hidden‑state trajectories it will see at serving time.
Results show the approach works where it matters most. At temperature zero, mean acceptance rises from 52.8 % to 66.4 % and mean accepted length grows from 2.58 to 2.99 tokens. The biggest gains appear in deeper draft positions—p3 acceptance jumps from 0.29 to 0.71 on GSM8K and from 0.39 to 0.76 on HumanEval—while the first‑position rate stays essentially unchanged.
For production traffic, AngelSpec’s D‑cut mechanism further tunes verification budget, delivering up to 15.7 % extra throughput at high concurrency with minimal acceptance loss.
In short, teams facing heterogeneous LLM serving loads can now deploy two lightweight, purpose‑built drafters that together give higher acceptance, longer useful drafts, and better throughput without sacrificing lossless verification.
#AI #Product #ML #LLM #SpeculativeDecoding #Efficiency