Developers who run language models locally often hit a wall: decoding speed stalls user experience while memory budgets stay tight. The delay appears each time the model proposes a token and waits for the next forward pass, turning simple chats or agent loops into noticeable lag. Teams that need on‑device coding assistants, offline copilots, or robotics controllers cannot rely on cloud APIs because of latency, privacy, or cost constraints, and they cannot afford to sacrifice accuracy for a modest speed gain.
Liquid AI’s DSpark draft checkpoints address this exact tension. Each drafter is a ~300 million‑parameter companion that proposes a block of nine candidate tokens in a single forward pass. The larger target model then verifies the whole block with one additional pass. Greedy decoding guarantees the output stream matches the baseline model exactly, so benchmark scores stay unchanged. The only practical cost is the extra memory needed to hold the drafter—about 655 megabytes in BF16 for the 2.6 B variant.
In practice the approach yields up to 3.18× higher token‑per‑second throughput on an H100 and up to 2.87× on an M4 Max MacBook Pro. Speedup follows the acceptance rate of the drafted tokens; on predictable workloads such as MATH500 the gain exceeds three‑fold, while on more varied tasks like GSM8K it drops to roughly 1.3×. The trade‑off is therefore workload dependent but always preserves correctness.
Deployment is straightforward for anyone who self‑hosts. The weights ship as Safetensors or GGUF files and work with day‑one support in llama.cpp and SGLang. No hosted inference service is required; users simply attach the drafter when launching their server. Licensing follows the LFM Open License v1.0, which permits free commercial use for entities under $10 million annual revenue—ideal for indie developers, startups, and SMBs. Larger organizations must negotiate a separate license.
The drafters fit naturally into local coding assistants, on‑device agents that reason before each tool call, single‑user chat sessions, and offline copilots running on laptop‑class hardware. For multi‑tool function‑calling agents the technique cuts average latency by about 57 % on the LFM2.5‑2.6B model, delivering a tangible responsiveness boost without altering the model’s behavior.
#AI #LLM #Inference #SpeculativeDecoding #EdgeAI #Product