Perplexity’s Lily engine shows how a tightly scoped inference stack can overcome the common hurdles of running large mixture‑of‑experts models on Apple silicon. Teams often hit a wall when they try to reuse generic frameworks like PyTorch or MLX: the extra abstraction layers add memory copies, CPU‑GPU synchronizations, and prevent the kernel fusions needed for maximum throughput. Lily sidesteps these issues by being a single‑process Rust runtime that talks directly to Metal, eliminating any intermediate tensor library.
The model it serves, Qwen3.6‑35B‑A3B, is a 35‑parameter MoE that activates roughly three billion weights per token. On a typical Mac the bottleneck is not arithmetic but moving weight bytes from unified memory to the GPU. Lily’s solution is to keep the expert routing histogram, prefix scan, scatter and block map inside one GPU command batch, so no MoE layer stalls for the CPU. During prefill it fuses the 4‑bit dequantization step into the grouped GEMM, expanding weights only in threadgroup memory and never allocating a full bfloat16 copy in unified memory. This yields a 77 % speedup on a 512‑token prompt.
For decode, where weight reuse is minimal, Lily focuses on reducing bytes moved per token. It overlaps independent kernels in a concurrent Metal pass, writes the selected token directly into the next step’s GPU‑resident slot, and fuses four kernel chains to keep intermediates in registers. GQA packing lets four query heads share a threadgroup, loading each KV row once, while a fixed‑block attention layout gives extra gains at longer contexts—up to 40 % faster at 128 K tokens.
On a 40‑core, 128 GB M5 Max Lily averages 1.23 × the prefill throughput and 1.35 × the decode throughput of MLX‑LM’s best path, with consistent advantages across prompt lengths from 256 to 128 K tokens. The trade‑off is a narrow scope: Lily only runs the Qwen3.6‑35B‑A3B checkpoint on Apple silicon, but that focus is precisely what delivers the measured performance boost.
For developers needing low‑latency, on‑device LLM serving on Macs, Lily offers a practical blueprint: strip away unnecessary layers, keep critical data on the GPU, fuse quantization and routing, and tune memory access patterns for the specific hardware.
#AI #LLM #Inference #AppleSilicon #PerformanceOptimization #OpenSource