Papers for

video streaming platform operators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

ByteDance improves recommender storage with faster low-cost system

USDB: Scaling Storage for Next-Generation Recommender Systems at ByteDance

Abstract: Recommendation models consume increasingly long, attribute-rich user histories, making small continuous updates and repeated large reads a central storage asymmetry. We present USDB, ByteDance's unified sequence storage practice for streaming and batch workloads. CascadeLSM time-based compaction stages materialize application-specific histories; differentiated replicas share persistent files; partial dumps reconstruct inference inputs from captured recent state and versioned history; and a hybrid columnar format accelerates projected reads. These designs reuse maintenance results across ingestion, inference, and training. Production migration records show 85.8% lower mean write latency, while normalized deployment accounting shows 42.1% lower infrastructure cost. Controlled comparisons with an Optimized RocksDB sequence baseline show faster long-history reads across projections and client loads. Aggregation pushdown sustains 8.5x the throughput of ordinary reads under a constrained receive budget. Production reconstruction and direct-file training results demonstrate how shared sequence maintenance supports fresh serving and repeated historical access.

Mon 2 NovDatabases
The gist
Storing and updating long user activity histories for recommendation is very challenging due to frequent small changes and large read demands. The authors present USDB, a new storage system that efficiently keeps track of these histories for both real-time and batch uses. Their approach reduces how long writes take and cuts infrastructure costs, while making reads of long user histories faster than before. It also allows fresh recommendations and repeated access to past data without extra work.
Open → 2611.02618v1