Skip to content
HN On Hacker News ↗

USDB: Scaling Storage for Next-Generation Recommender Systems at ByteDance

▲ 49 points • 8 comments • by 1equalsequals1 • 2d ago • HN discussion ↗

Pangram verdict · v3.3

We believe that this text is a mix of AI and human-written content.

58 %

AI likelihood · overall

Mixed
42% human-written 58% AI-generated
SEGMENTS · HUMAN 2 of 3
SEGMENTS · AI 1 of 3
WORD COUNT 218
PEAK AI % 100% · §2
Analyzed
Nov 3
backend: pangram/v3.3
Segments scanned
3 windows
avg 73 words each
Distribution
42 / 58%
human / AI fraction
Verdict
Mixed
Pangram v3.3

Article text · 218 words · 3 segments analyzed

Human AI-generated
§1 Human · 0%

Authors:Xiaoran Zhu, Jieming Cai, Weiping Yu, Xi Chen, Shu Cai, Jiahao Li, Yifan Xiao, Yan Deng, Yisha Zhou, Yu Lin, Shiru Ren, Siqiang Luo, Peng Xu, Junjun Chai View PDF HTML (experimental) Abstract:Recommendation models consume increasingly long, attribute-rich user histories, making small continuous updates and repeated large reads a central storage asymmetry.

§2 AI · 100%

We present USDB, ByteDance's unified sequence storage practice for streaming and batch workloads. CascadeLSM time-based compaction stages materialize application-specific histories; differentiated replicas share persistent files; partial dumps reconstruct inference inputs from captured recent state and versioned history; and a hybrid columnar format accelerates projected reads. These designs reuse maintenance results across ingestion, inference, and training. Production migration records show 85.8% lower mean write latency, while normalized deployment accounting shows 42.1% lower infrastructure cost. Controlled comparisons with an Optimized RocksDB sequence baseline show faster long-history reads across projections and client loads. Aggregation pushdown sustains 8.5x the throughput of ordinary reads under a constrained receive budget. Production reconstruction and direct-file training results demonstrate how shared sequence maintenance supports fresh serving and repeated historical access.

§3 Human · 0%

Comments: 17 pages, 18 figures, 5 tables Subjects: Databases (cs.DB) Cite as: arXiv:2611.02618 [cs.DB] (or arXiv:2611.02618v1 [cs.DB] for this version) https://doi.org/10.48550/arXiv.2611.02618 arXiv-issued DOI via DataCite (pending registration) Submission history From: Weiping Yu Mr. [view email] [v1] Mon, 2 Nov 2026 06:45:03 UTC (432 KB)