Two recent open‑weight releases show how leading Chinese labs have independently arrived at the same efficient blueprint for large multimodal models. Teams face three recurring problems: ballooning compute as context grows, memory pressure from ever‑larger KV caches, and instability after post‑training when positional cues are weakened. The shared recipe solves each issue.
First, they keep three linear‑attention layers for every full‑attention layer. Linear layers replace the growing cache with a fixed‑size recurrent state, so token‑level compute stays flat no matter how long the input. The remaining quarter of layers perform precise retrieval but only on a compressed view of the history.
Second, both models attach a tiny indexer that scores chunks of context after a 4× compression and keeps the top 2048 tokens. This limits the expensive attention to a constant budget, cutting KV cache size by roughly fourfold and attention compute by about three times compared with a dense transformer.
Third, the residual path is split into four parallel gated streams instead of a single stream. Gates decide what each block reads and writes, which smooths flow, reduces activation outliers and enables FP8 storage without hurting quality.
The only disagreement is positional encoding: one team removed rotary embeddings, trusting the linear layers to carry position; the other kept them after observing that a NoPE model failed to stop generating during post‑training. For most pipelines, retaining RoPE in the sparse layers is the safer choice.
Finally, both train with the Muon optimizer, splitting fused projection matrices before orthogonalization, and they skip batch‑size warm‑up after measuring it adds overhead without benefit.
Adopting this 3:1 linear/full split, a 4‑token compressed indexer capped at 2048 tokens, four gated residual streams, and Muon with matrix splitting gives practitioners a proven path to high‑performance, low‑cost models that scale to million‑token contexts while preserving reasoning and stability.
#AI #MachineLearning #LLM #Efficiency #ProductDevelopment #Innovation