Making Coroutines Routine: Building a Scalable TPC-C Client in C++
Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,448 words · 4 segments analyzed
21 min read4 days ago--We started with Java, 150,000 OS threads and roughly 600 GiB of RAM. We ended with readable sequential code in C++, 16 worker threads for terminal execution, memory usage reduced by more than 1,000x, and a benchmark client that could finally keep up with a distributed database.Press enter or click to view image in full sizeAt YDB, we build a fault-tolerant distributed database in C++. That means we also spend a lot of time building and running benchmarks.Benchmarks are not optional infrastructure. We use them to load-test the system, detect performance regressions, validate optimizations and new features, and compare YDB with other databases. The larger the database becomes, the more important the benchmark client becomes as well: it is surprisingly easy to saturate the machine generating the load long before the database itself is busy.That is exactly what happened when we tried to run TPC-C at scale. We wanted to benchmark the database. Instead, we benchmarked the client.This post tells the story of how that led us from OS threads to futures, from futures to callback hell, and finally to C++20 coroutines. Along the way, we will look under co_await, reconstruct a coroutine by hand, and discuss one of the most important practical questions in asynchronous C++:After a future or coroutine becomes ready, which thread executes the rest of your code?The code below is simplified and adapted for explanation, but it follows the architecture of our real TPC-C implementation.TPC-C as a Perfect Coroutine WorkloadTPC-C is an OLTP benchmark standardized in 1992. Despite its age, it is still widely used because it models a nontrivial transactional application rather than a stream of independent key-value operations.The benchmark represents a wholesale company with multiple warehouses. Each warehouse serves ten districts and contains roughly 100 MB of data. Each district has a terminal representing a user or employee interacting with the system.A terminal repeatedly performs one of several transactions: placing an order, making a payment, checking order status, processing delivery, or checking stock levels. The important part for our purposes is the execution model:A large run may have hundreds of thousands of terminals.A terminal is usually idle, simulating a human typing or thinking.When active, it executes a multi-step interactive transaction.A transaction contains roughly 5–10 database requests, with only a small amount of CPU work between them.In other words, each terminal is a mostly sequential workflow with many natural suspension points. It spends almost all its lifetime waiting: for a timer, the network, or the database.This is precisely the kind of workload where coroutines should shine.First We Benchmarked the ClientWe initially used BenchBase, the well-known JDBC benchmarking framework developed at Carnegie Mellon University under Andy Pavlo. It has a good architecture, supports multiple databases, and includes one of the few widely used implementations of TPC-C.The problem was not the benchmark logic. The problem was its execution model. TPC-C has ten terminals per warehouse. Therefore, a run with 15,000 warehouses has 150,000 terminals. In the original implementation we used, that meant 150,000 OS threads.For one of our runs, the client required approximately:150,000 OS threads;600 GiB of RAM;five client machines, each with 128 CPU cores and 512 GiB of RAM.The YDB cluster under test used only three machines of the same size.This ratio is difficult to justify. At larger cluster sizes, the load generator becomes a substantial part of the cloud bill. We estimated that a single experiment across several database configurations could cost around $10,000 in AWS, with much of the cost going to the client rather than the databases being tested.We first optimized the existing Java implementation. We replaced platform threads with Java virtual threads and immediately ran into a subtle deadlock related to their execution model. We also reduced memory consumption. For 15,000 warehouses, memory usage fell from roughly 600 GiB to about 90 GiB, and the benchmark could run on one machine with a reasonable number of platform threads.That was a major improvement, but it was still heavier than we wanted. We are C++ developers, YDB is written in C++, and we needed more control over memory, scheduling, and observability.
So we made the decision every programmer secretly enjoys making:We rewrote it.Press enter or click to view image in full sizeThe Ideal Code Is SynchronousLet us start with the most natural implementation of a TPC-C terminal:void RunTerminal(TInstant endTs) { while (Now() < endTs) { auto type = PickTransaction(); auto input = MakeInput(type); // Simulate the user entering the request. Sleep(KeyingTime(type)); auto result = RunTx(input); // Simulate the user reading the response. Sleep(ThinkTime(type)); Stats.Record(type, result); }}A simplified transaction is equally straightforward:TxResult RunTx(Input input) { auto tx = Db.BeginTransaction(); auto r1 = tx.Query(sql1).GetValueSync(); // Process r1 and prepare sql2. auto r2 = tx.Query(sql2).GetValueSync(); // Process r2. // Queries 3-9. auto r10 = tx.Query(sql10).GetValueSync(); // Process r10.
return tx.Commit().GetValueSync();}This code is excellent from the application developer’s point of view. It is linear, local, and easy to debug. The business logic is visible from top to bottom.The operating system also does a great deal of work for us. When a thread sleeps or waits for a database response, the kernel saves its execution context and schedules another thread. When the wait finishes and a CPU becomes available, the kernel restores the context and execution continues after the blocking call. The terminal does not need to know that it was suspended.Unfortunately, this model implies one thread per terminal. A few dozen threads are fine. A few thousand may already be uncomfortable. A hundred and fifty thousand is not a viable architecture.Each thread needs a stack and kernel resources. Context switching is relatively expensive. Page-level fragmentation adds more memory overhead. Thread creation and destruction also take a significant amount of time when there are a lot of threads already.The synchronous code is beautiful, but the execution model does not scale.Threads vs. Coroutines: A Small ExperimentTo make the difference visible, we wrote a simple microbenchmark.The machine had 32 physical cores on one NUMA node. We created N workers, either OS threads or stackless coroutines. Each worker performed approximately one microsecond of CPU work and then yielded. The test used one second of warm-up followed by five seconds of measurement.Up to 32 workers, both versions scaled similarly: there was a physical core for every worker. Above that point, their behavior diverged.The coroutine version maintained nearly constant aggregate throughput across thousands of workers. The thread version dropped immediately after oversubscription and became progressively worse as the number of threads increased. As shown in Figure 1, both approaches scaled similarly up to the number of physical cores, but only coroutines preserved throughput beyond that point.Press enter or click to view image in full sizeFigure 1. Aggregate throughput as concurrency increases. With a 1 μs work slice and 32 physical cores, coroutine throughput remains nearly flat after saturation, while thread throughput steadily declines.The wall-clock time was even more revealing. The benchmark was supposed to take approximately six seconds, including warm-up. The coroutine version stayed close to that value. With thousands of threads, however, the time spent creating, scheduling, and joining them grew sharply. Figure 2 shows that this overhead eventually dominates the benchmark itself.Press enter or click to view image in full sizeFigure 2. Total benchmark duration. Coroutine execution remains close to the expected six seconds, while the thread-based version becomes increasingly expensive once the worker count reaches the thousands.The exact numbers depend on the machine and implementation, but the shape is what matters:OS threads are a good unit of parallel execution. They are a poor unit for representing hundreds of thousands of mostly sleeping workflows.Press enter or click to view image in full sizeFutures Fix the Execution ModelThe obvious next step is to use the asynchronous database API.A query returns a future. The SDK eventually stores the result in the corresponding promise, and the future becomes ready. With a composable future implementation, we can attach continuations:auto future = tx.Query(sql1) .Apply([&](auto r1) { // Process r1. return tx.Query(sql2); });The standard std::future is not particularly helpful here because it does not provide continuation chains. Practical async code usually relies on a third-party future implementation such as YDB's TFuture, Folly futures, or another library with then/Apply/Subscribe-style operations.But there is an important subtle issue hidden in this code: which thread actually executes each part?With many future implementations, the continuation attached with Apply() is executed by the thread that makes the future ready.
In this case, that is an SDK thread:Future<TxResult> RunTx(Input input) { auto tx = db.BeginTransaction(); // Executed in the caller's thread. return tx.Query(sql1) .Apply([&](auto r1) { // Executed by the thread that fulfilled // the previous promise — typically an SDK thread. // Process r1. return tx.Query(sql2); }) // r3-r9, similarly SDK thread .Apply([&](auto r10) { // Process r10.