Many teams building retrieval ranking or embedding pipelines notice that even though their models run on GPU in a few milliseconds the overall request latency stays high The reason is that every input must first be tokenized on the CPU For small models such as rerankers or classifiers tokenization can become the dominant cost especially when batch sizes are large and hundreds of candidates are scored per query
Perplexity AI addressed this by rewriting their Unigram tokenizer from scratch in Rust and open sourcing the result The new encoder produces zero steady state heap allocations reaches a p50 latency of about 63 microseconds for a 514‑token input and is roughly five times faster than the Hugging Face tokenizers crate In production the change cut CPU utilization by five to six times and shaved double‑digit milliseconds off reranker latency
Three focused optimizations made this possible First they replaced the HashMap based trie with a double‑array trie which removes hashing and pointer chasing turning each byte step into two array reads an add and a compare Second they added a per node bitmap of valid child transitions and packed the bitmap base offset token ID and score into a single 64‑byte cache line so each step needs only one memory load Third they backed the trie with 2 MB huge pages via mmap with MAP_HUGETLB reducing TLB misses and page‑table walks especially for longer inputs
The practical takeaway for engineers is to profile the tokenization stage when serving small transformer models Eliminating per encode allocations alone can halve latency Moving to a cache line aligned double‑array trie with bitmap validation brings another two to three times speedup and using huge pages helps when inputs exceed a few thousand tokens The implementation is available under an MIT license and can be integrated into existing Rust inference stacks or called from C/C++ through a simple FFI
#AI #Product #LLM #Tokenization #Performance #Rust