Netflix’s TimeSeries Abstraction stores petabytes of temporal event data in Apache Cassandra. When a single TimeSeries ID accumulates too many events its Cassandra partition becomes wide and read latency climbs from a few milliseconds to several seconds. Wide partitions cause timeouts, increase garbage collection pressure, drive up CPU usage and create thread queueing, which hurts the reliability of low‑latency dashboards and paginated queries.
To address this Netflix introduced a dynamic repartitioning pipeline that works at the level of individual TimeSeries IDs. The pipeline has four stages. First, every read counts the bytes it scans; when the count exceeds a configurable threshold a lightweight event is sent to Kafka that marks the partition as a candidate for splitting. Second, a planner reads the whole partition once, checkpoints progress in a wide_row metadata table and builds a split plan that assigns event buckets to child partitions while keeping the total order intact. Third, the split is performed asynchronously using a strategy such as EventBucketPartitionSplitStrategy, which caps the number of event buckets per child to control read amplification. Fourth, the system validates the split by comparing pre‑ and post‑split checksums and only marks the operation COMPLETED when the hashes match; shadow runs with Data Bridge Spark jobs confirm correctness.
Reads first consult an in‑memory Bloom filter that consults the wide_row cache in single‑digit microseconds. On a hit the cached metadata routes the query to the appropriate child partitions; the original wide partition is retained as a fallback for partial failures. The approach needs no application code changes and works on immutable partitions initially.
Results show average read latency for previously wide partitions dropping from seconds to low double‑digit milliseconds, tail latency falling to about 200 milliseconds, and 500 megabytes‑plus partitions staying available for paginated queries. CPU usage and thread queueing are reduced, making the Cassandra cluster more stable.
#AI #Product #Cassandra #TimeSeries #DataEngineering #Performance