YaFF is Yandex’s open‑source zero‑copy wire format for Protobuf messages released under Apache 2.0. It keeps the original .proto file as the single source of truth and only changes how the data is laid out in memory. This lets you read fields without a parsing step while still being able to convert back to regular Protobuf when needed.
The main problem YaFF solves is the CPU cost of Protobuf parsing in high‑load backends. Parsing can consume double‑digit percentages of a server’s CPU, translating to thousands of cores at scale. The usual zero‑copy alternative, FlatBuffers, requires a separate schema and conversion layer, breaking Protobuf semantics and adding maintenance overhead. Teams often find the migration cost too high compared to the performance gain.
YaFF offers four layouts that trade read speed for schema flexibility. Fixed is a packed struct with no header and a frozen schema, ideal for small inlined primitives. Flat adds a two‑byte header and supports limited schema evolution, delivering ~3.8× faster reads than FlatBuffers on hot data while staying within 1.2× of a raw C++ struct. Sparse uses a meta table for unrestricted evolution, suited for sparse schemas. Dynamic, the default, chooses Flat or Sparse at runtime, giving unrestricted evolution with low overhead for general use.
Adoption is incremental. You can introduce YaFF in a single hot path, keeping the rest of the system on standard Protobuf. Two‑way conversion at the edges lets you serialize a Protobuf message into a YaFF buffer, read fields directly with no parse step, and convert back to Protobuf when a consumer needs the parsed message. This approach has already yielded 10–20% CPU savings in Yandex’s advertising recommendation system at production scale.
YaFF works best in environments where you control both producer and consumer—recommendation and ad‑serving backends, memory‑mapped indexes, search indexes, feature stores, and feed services. A planned columnar layout will extend its benefits to analytics and ML pipelines.
To start, add YaFF to your build via CMake or Conan, run protobuf generation followed by YaFF code generation, and link the yaff::core and yaff::proto libraries. The generated types live in the protoyaff namespace, providing a familiar Protobuf‑like API without the parsing cost.
#AI #Product #Performance #Protobuf #ZeroCopy #YaFF