Frontier open-weight models are now matching proprietary systems in capability, but running them still assumes datacenter‑class GPU clusters. Individual developers and small teams end up paying high cloud inference bills while their own machines sit idle. FreeToken solves this mismatch by turning a personal computer into a unified, elastic inference platform. It continuously maps computation and model state onto whatever GPU, CPU, memory and interconnect bandwidth the machine actually has, rather than treating the device as a fixed‑size GPU.
The system works with three concrete mechanisms. First, a bandwidth‑adaptive q* policy splits each cache miss between PCIe transfers and CPU execution based on measured B_P and B_H, so the saturated link leaves residual bandwidth for CPU work without approximation. Second, semantic‑aware caching uses full‑layer double buffering during prefill and a shared LRU expert cache during decode, keeping expert output bit‑exact and avoiding router changes. Third, elastic memory management rebuilds the GPU expert cache at safe points without restarting the engine, allowing the first request to be served from a cold cache.
FreeToken is already deployable. It is released under Apache‑2.0, available on PyPI as freetoken (install with pip install freetoken[accel]), and provided as a one‑click desktop app for Windows and Linux. The CLI ft serve exposes OpenAI‑ and Anthropic‑compatible endpoints on port 1919, and ft launch claude wires up popular agent frameworks against your own box. On an RTX 5090 it sustains 77‑83 tok/s on Qwen3.6‑35B‑A3B and 22‑25 tok/s on DeepSeek‑V4‑Flash, delivering 1.5‑2.3× the decode throughput of llama.cpp, Ollama or KTransformers while keeping worst‑case time‑to‑first‑token under 44 seconds. Even an 8 GB RTX 4060 laptop serves a 35B model at 39.3 tok/s, surpassing typical production decode medians for many cloud‑based agents.
For solo developers, startups and SMB engineering teams whose token costs now exceed the price of the GPU they own, FreeToken offers a practical path to run frontier models locally, keep data on‑premises, and avoid ongoing cloud fees. Enterprises can use it for air‑gapped or regulated workloads without replacing datacenter infrastructure.
#AI #Product #LLM #Inference #OpenSource #DevTools