GitHub - guycipher/keybench: A scriptable, extensible performance tool for sorted key value stores.
Pangram verdict · v3.3
We believe that this document is primarily human-written, with some AI-generated content detected
AI likelihood · overall
MixedArticle text · 1,688 words · 7 segments analyzed
keybench ========
A scriptable, extensible performance tool for sorted key value stores.
You write the workload in Lua. keybench drives it across one or more storage engines, times every operation, and reports throughput and latency. The same script runs unchanged against every engine, so a comparison measures the engines and not the harness.
What it measures ================
keybench reports two rates and a latency distribution.
wu/s workload units per second. One unit is one call to your run() function. This is the rate of whole operations as your script defines them, a "view cart" or a "checkout", regardless of how many key touches each one costs.
ops/s primitive operations per second. One primitive op is a single put, get, del, range, or scan call, or one key handled inside an mget, mput, or mdel. A range or a scan counts once no matter how many rows it returns. This is the rate of raw key touches.
When every unit is one primitive op the two rates are equal and the report prints one throughput line. When a unit does several primitive ops, such as a batch of B keys or a cart checkout that scans then deletes, the report prints wu/sec and ops/sec on separate lines. For a batched verb ops/sec is exactly B times wu/sec, because each unit performs B key touches, so batching shows as ops/sec rising with B while wu/sec falls, the difference being the fixed per call cost spread over more keys.
Latency is recorded per operation kind as a distribution, not an average. Each of put, get, del, range, mget, mput, and mdel has its own histogram, and a scan is timed into the range histogram.
The report gives p50, p99, p99.9, and the max for each.
How it works ============
Five parts, each replaceable.
The engine owns concurrency. keybench spawns the worker threads, splits the op budget across them, and joins them. A worker runs your script in its own Lua state with its own random seed. The harness never holds a lock around an engine call, so a serialized engine reports as serialized and a parallel one reports as parallel. The Lua script stays single threaded and never reasons about locks.
The workload is a Lua table. A file returns a table with a name, an optional load function, and a run function. load seeds the store before timing, and the harness runs it on every worker thread at once, each filling its own shard, so a large dataset loads in parallel. run is the measured unit and the harness calls it until the op budget or the time budget is spent. Both receive a ctx table carrying users, items, ops, seed, batch, thread, threads, and the current iter.
The store is one verb set over every engine. Your script calls a global kv table. The same eight verbs reach whichever engine is selected. keybench times the call, files the sample in the right histogram, and counts the primitive ops.
The backend is a self registering plugin. Each engine lives under backends/ with its own build fragment and registers itself before main runs. Selecting one by name, or several for a comparison, is a command flag. Adding one is a new directory and a vtable, no edit to the core.
The reporter is a sink. A run broadcasts its metadata, its probes, its aggregated points, and its live samples to every reporter you asked for. A console reporter prints a human report. A tsv reporter writes machine rows. A timeline reporter writes one row per live sample. You can run several at once.
Building ========
keybench vendors its own Lua, so the default build needs only a C compiler and pthreads.
make
That produces ./keybench with the in memory skiplist engine compiled in. The skiplist needs no external library and is the reference engine.
To compile in the persistent engines, name them.
make ROCKSDB=1 add the RocksDB engine make TIDESDB=1 add the TidesDB engine make BACKENDS="skiplist rocksdb tidesdb"
No special allocator is linked by default, keybench uses the system one. If a distro rocksdb or tidesdb was itself built against jemalloc, link the same allocator first with ALLOC=jemalloc so the whole malloc family agrees, which avoids a crash when a database is reopened across sweep points. ALLOC=tcmalloc works too, and ALLOC_LIBS="..." links any other allocator or a custom path. The engine library and header locations are also overridable, so a custom install works.
make ROCKSDB=1 ROCKSDB_CFLAGS=-I/opt/rocksdb/include ROCKSDB_LIBS="-L/opt/rocksdb/lib -lrocksdb" make TIDESDB=1 TIDESDB_CFLAGS=-I/opt/tidesdb/include TIDESDB_LIBS="-L/opt/tidesdb/lib -ltidesdb" make ROCKSDB=1 TIDESDB=1 ALLOC=jemalloc
To build against a system Lua instead of the vendored copy.
make LUA_SYS=1
Other targets.
make run build, then run the cart workload once make all-wl build, then run every workload in workloads/ make clean
Running =======
The smallest run names one workload file.
./keybench workloads/mixed.lua
By default this drives 200000 units on one thread against the skiplist and prints a console report.
The shape of that report.
keybench 0.1.0: 1 workload(s), engine=skiplist seed=1 repeat=1 threads: 1 batch: 1 [probe: system] host agpmastersystem os Linux 6.2.0-39-generic x86_64 cpu 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz cpu_cores 16 ram 46.8 GiB page_size 4096 data_fs(/tmp) 88.6 GiB free of 296.0 GiB data_dev /dev/sdb4 on / (ext4) data_disk solid state data_disk_model WDC WDS500G2B0A [probe: build] keybench 0.1.0 compiler 12.3.0 std C 201112 (optimized) allocator jemalloc (glibc 2.37)
=== mixed === engine=skiplist threads=1 batch=1 (median of 1 run) op count p50 p99 p99.9 max put 60404 462 ns 1.43 us 11.71 us 15.48 us get 99617 474 ns 1.46 us 11.84 us 69.14 us del 19955 470 ns 1.48 us 9.41 us 13.41 us range 20024
2.32 us 8.26 us 15.55 us 26.98 us get hit rate: 92.2% throughput: 315896 /sec (1 op per unit)
The system probe records the machine. The build probe records the compiler and allocator. The per op table gives the latency distribution. The hit rate is the fraction of gets that found a key. The throughput line gives wu/s and, when it differs, ops/s.
Live progress =============
While a point runs, keybench streams a status line to stderr once a second, so a long point looks alive rather than hung. It first prints a seeding line for the untimed load phase, then one line per second, each its own line so the output scrolls as a log rather than rewriting one spot. A time bounded point shows the elapsed and budget seconds and the current rate.
seeding cart on rocksdb 11.1.1 with 8 threads ... done (1.8s) cart rocksdb 11.1.1 t8 b1 1.0/5s 776154 wu/s cart rocksdb 11.1.1 t8 b1 2.0/5s 626071 wu/s cart rocksdb 11.1.1 t8 b1 3.0/5s 578264 wu/s
A count bounded point shows the elapsed seconds, the rate, and the percent of the op budget done.
mixed rocksdb 11.1.1 t1 b1 2.0s 161301 wu/s 43% (341639/800000) mixed rocksdb 11.1.1 t1 b1 3.0s 147362 wu/s 61% (489498/800000) mixed rocksdb 11.1.1 t1 b1 4.0s 138006
wu/s 78% (627934/800000)
The status goes to stderr and only when stderr is a terminal, so it never lands in a piped or redirected log and never mixes into the stdout report or the tsv streams.
Choosing the work =================
--ops N total units across all threads, then stop. Default 200000. --secs S run for S seconds instead of a fixed count. --users N size of the user population a workload may address. --items N size of the catalog a workload may address. --seed N base RNG seed. Thread t uses seed plus t.
These split into two kinds. --ops and --secs are the two ways to bound a run and you use one of them, since giving --secs clears the op count and runs the clock. --users, --items, and --seed are not a choice between, they always apply, each handed to the workload through ctx to size its population, size its catalog, and seed its RNG. So a run is one bound flag plus any of the three knobs at once.
Sweeping and comparing ======================
A comma list in a knob turns one run into a grid. keybench runs every point in the grid and, when there is more than one, prints a summary table sorted as it ran.
--backend L one engine, or a comma list to compare. e.g. skiplist,rocksdb --threads L one thread count, or a comma list to sweep. e.g. 1,2,4,8 --batch L one batch size, or a comma list to sweep.
e.g. 1,8,64 --repeat N run each point N times and report the median.
The grid is the cross product of engines, thread counts, and batch sizes. Only a workload that opts into batching, by returning batched true, varies with the batch knob, so the others run once at batch one rather than repeating an identical measurement across the batch list. Each cell is run --repeat times and reduced to a median, which damps the noise of a single run. An example that compares two engines as they scale.
./keybench --backend skiplist,rocksdb --threads 1,4,16 --repeat 3 \ --data-dir /mnt/disk workloads/mixed.lua
Name several files to run them in sequence, each isolated in its own fresh store.
./keybench workloads/cart.lua workloads/mixed.lua workloads/scan.lua
Reporters and live timeline ===========================
--report S a comma list of sinks, each name or name:path. --timeline S sample every S seconds while a run is in flight. --report-dir D create D/<timestamp>/ and bundle every artifact there.
The sink names are console, tsv, and timeline. A bare name, or a name whose path is a single dash, writes to stdout. A name with a path writes to that file.
--report console,tsv:points.tsv,timeline:tl.tsv
console prints the human report above. tsv writes one row per operation per grid point, ready for a spreadsheet or plot. timeline writes one row per live sample, one metric per row, so you can watch throughput, CPU, resident memory, disk bytes, and engine internals evolve over the run. The timeline sink only produces rows when --timeline names an interval.
--report-dir is the convenient whole. It stamps a fresh directory, writes the console report, the points tsv, the timeline tsv, and a replay config into it, and tells you where.
Probes ======
A probe records context the benchmark itself does not produce.