Video diffusion transformers (DiTs) suffer from a single dominant cost: self‑attention. A 5‑second 720p clip creates roughly 70 K tokens, and attention consumes more than 60 % of generation time on modern GPUs. Two issues keep the speed low even when the matrix multiplies are moved to low‑bit Tensor Cores. First, after smoothing queries and keys, the value term still carries about 80 % of the output error because a few outlier tokens dominate the quantization scale. Second, the softmax that follows the QK product runs in FP32; its exponential and FP8 cast become the longest pipeline stage on Blackwell and Hopper chips.
A practical fix tackles both problems without retraining. V‑Smart groups similar value tokens online with k‑means, permutes keys the same way to keep causal correctness, subtracts the mean of each 128‑token hardware block, quantizes only the residual in per‑channel E4M3 (8‑bit) or NVFP4 (4‑bit), and restores the mean using the row‑sum already kept by the online softmax. This removes up to 36 % of block energy after sorting, adds only 0.125 bit per element, and costs roughly 3‑4 % of attention time because the grouping is refreshed only on the first quarter of denoising steps and reused for four steps.
ExpCast‑FP8 replaces the FP32 exponential‑then‑cast path with a single fused multiply‑add that writes the FP8 probability byte directly from the log‑domain score. An E4M3 byte approximates 8 log₂(v)+56, so the operation needs only one multiply‑add plus a constant bias. The method matches the exact byte on 79.6 % of values and deviates by at most one code elsewhere, yielding a per‑row total‑variation bound under 3.64 % (measured average 1.6 % on Wan2.2). It works for the 8‑bit kernel; NVFP4 lacks a clean log‑to‑code map.
Together V‑Smooth and ExpCast‑FP8 cut attention latency by more than half on B200 and H200, restore most of the lost fidelity, and require no extra training or model changes. Deploy the kernels as drop‑in replacements for existing attention layers and immediately see faster video diffusion generation.
#AI #VideoAI #Diffusion #AttentionOptimization #LowBitQuantization #GPUPerformance