Large language models need to keep key‑value vectors for every token during decoding. This KV cache grows linearly with sequence length and batch size, quickly exceeding the model’s own memory footprint and making decoding bandwidth‑limited. Shrinking the cache is the most direct way to cut cost and latency.
Three families of techniques have emerged. TurboQuant applies a data‑oblivious random rotation followed by an optimal scalar quantizer and a 1‑bit QJL transform on the residual. It needs no calibration, works on any model, and achieves near‑lossless quality at 3–4 bits per channel, giving a theoretical distortion close to the information‑theoretic limit.
OSCAR takes the opposite route: it computes an attention‑aware rotation from a one‑time offline calibration, then spreads importance with a Hadamard transform and bit‑reversal. It ships as a full system with mixed‑precision paged cache, fused Triton kernels, and pre‑computed rotations for popular models. At roughly 2.3 effective bits it matches BF16 quality on several benchmarks and delivers up to 8× KV‑cache memory reduction and ~3× faster decoding.
EpiCache tackles multi‑turn conversation memory by processing history in blocks, clustering turns into semantic episodes, routing queries to the relevant episode, and allocating memory adaptively per layer. It reports up to 40% accuracy gains over simple eviction, near‑full‑cache accuracy at 4–6× compression, and up to 3.5× lower peak memory. Because it decides which tokens to keep, it can be combined with either TurboQuant or OSCAR for compounding savings.
Choosing a method depends on constraints. If you need a model‑agnostic, training‑free solution in the 3–4 bit range, TurboQuant is the go‑to. If you require deployable INT2 on supported models with production‑ready kernels, OSCAR leads. For long multi‑turn sessions where history accumulates, add EpiCache to either quantizer for extra memory and accuracy benefits. The approaches are orthogonal and can be stacked to meet bit‑width, portability, and conversation‑length goals.
#AI #Product #LLM #MachineLearning #Tech #Innovation