Ok, but does it scale? | Blog | Spacetime

This is probably the question I get asked most about Spacetime. It’s a simple enough question, and it certainly seems like it should have a simple answer. Scaling is a complex topic, and the devil is in the details, as it so often is. On the other hand, it’s also not so complex that we can’t understand scalability from first principles in a blog post. Let’s start by exploring scalability in general, and then let’s answer the question, “How does Spacetime scale?” If you want the TL;DR: There are three dimensions of scale: compute, storage, and networking. Horizontally scaling storage is relatively straightforward and shipping October 31st, 2026. However, not all networking and computation can be scaled horizontally. OLTP databases that claim general horizontal scalability often pay enormous overhead per transaction and perform extremely poorly when faced with contending transactions. Spacetime provides high performance under contention and provides tools to make it easy for you to scale your parallelizable OLTP workloads. Note NOTE: I talk about CockroachDB a lot in this article. CockroachDB is a rough stand-in for essentially all general purpose, horizontally scaling RDBMSs including Spanner and Aurora DSQL. Although I talk about some issues with these technologies, all of them are incredibly impressive feats of engineering. ScaleIntuitively, everyone has an idea of what it means to “scale”. It means to be able to do more. It means to keep up with demand. It means to handle a billion requests, or “infinite” requests, or an infinite amount of data, or an infinite number of customers, or the ability to grow your app at 10x or 100x year over year without needing to rewrite your software. In particular, I think that when most people say a system is scalable, they’re talking about whether or not it’s specifically horizontally scalable. Whereas vertical scalability means to do more with a single computer, horizontal scalability is the ability to do more with more computers. If doubling the number of computers lets us do roughly twice as much work, the computation scales horizontally. After all, a single computer can only be so big and fast, but in theory there’s no limit to the number of computers you can buy. There’s something very satisfying about that idea, so that’s the property that everyone looks for. Note NOTE: The “doing more with more computers” definition implies that all horizontally scalable systems must be distributed systems. However, it does NOT imply that the only purpose of distributed systems is horizontal scalability. For example, distributed state machine replication is designed to redundantly do the same computation on many computers for the purpose of reliability, not scale. More on this below in the Spacetime section. The question “Does it scale horizontally?” is underspecified. A better question is, “In what ways does it scale horizontally?” This is because there are actually three pretty independent dimensions of scalability: Compute: how many transactions you can process Storage: how much data you can store Networking: how many connections and how much bandwidth you can support Note To see what I’m talking about, let’s look at a few database systems that all expose broadly PostgreSQL-compatible interfaces, but have radically different architectures. For example: Postgres is for the most part a single-node database. Postgres does not scale compute, storage, or networking horizontally for you. You can of course scale Postgres horizontally by deploying many Postgres instances, but as far as the Postgres code is concerned, it’s largely unaware of those other instances. The one exception to this is read replicas which allow you to manually direct readers to a replica. This helps to scale both networking and compute, but it comes with caveats about read-after-write consistency and performance. Postgres itself has no notion of a cluster of primaries, cannot run transactions or queries across them, or route you to the appropriate one. You can of course write software to do this, and this is how people can and do scale Postgres horizontally, but how to do it is left as an exercise to the reader. Neon, which is a modified variant of Postgres, scales storage horizontally. Neon tables are backed by object storage with a local page caching structure to make data access fast and efficient. Page access latency can be higher for cache misses, but this architecture gives your Neon databases effectively infinite storage. Neon does not automatically scale compute and networking horizontally, however. Like Postgres, all write transactions must go through a single primary. However, being a Postgres variant, Neon also supports read replicas to horizontally scale compute and networking for read workloads, although this comes with similar consistency caveats. Horizontally scaling storage is a big win, even without automatic compute scaling. Even small web apps with not so many users can in principle use a lot of storage. CockroachDB (and Spanner), in principle, scales storage horizontally by spreading each table’s data around the cluster so that a portion of each table, called a “range”, is stored on each machine and typically replicated to two other machines. Provided that writers don’t all try to modify the same range, it can also scale networking horizontally. It has a symmetric architecture, meaning any node in the cluster can serve any SQL request, both reads and writes. Finally, for computation that parallelizes cleanly (e.g. analytics or writes to unrelated keys), it also scales horizontally. The node you connect to will compute the query plan and return the results, but both read and write operations will execute as part of a distributed transaction across the cluster based on the ranges. Can we find the Holy Grail?So if CockroachDB is able to scale in all three dimensions, it must be better than Postgres and Neon in all respects, right? Maybe it does something special with the CAP theorem[1] or atomic clocks? Unfortunately, there is no magic here. While CockroachDB is a modern marvel and does scale certain computations horizontally, not all computation can be scaled horizontally. Horizontal scalability is really a question of parallel computing: can we split the computation into pieces that different computers can do at the same time? The answer is often no, and the issue is with CockroachDB, you pay the enormous coordination cost of horizontal scalability even when data contention forces you to do updates one at a time. What CockroachDB gains in horizontal scalability, it loses in vertical scalability and then some. The ugly truth is that rather than doing more with more computers, horizontal scalability can often mean doing less with more computers: conceptually what one computer can do in 1 millisecond, 10 computers can do in 100 milliseconds. There are two main issues with CockroachDB’s approach to horizontal scalability: The data involved in each transaction is rarely co-located on a single machine. No transaction has exclusive access to a range, so every transaction pays distributed concurrency-control overhead, and conflicting transactions must wait, abort, or retry. The first problem is caused by spreading ownership of the data evenly around the cluster. If your data is spread around the cluster, you need to do network requests for essentially every transaction. Although sharding often sounds inconvenient, it can provide much better performance if most of your transactions operate within a single shard. Also note that Neon’s design does not suffer from this same issue for many workloads because it caches hot pages on the same machine. Note Spanner partially addresses this issue with “table-interleaving” which allows you to essentially tell Spanner to colocate related tables. This can dramatically improve performance in simple cases. CockroachDB supported table interleaving for several years but removed it in v21.2, judging the benefits too small to justify the complexity. The second problem is a general problem of parallelizing arbitrary computation: coordination under contention. Even vanilla Postgres runs into the same problem, just on a smaller timescale. Rather than coordinating transactions across a distributed system, it has to coordinate transactions across multiple cores. Postgres can run transactions on many CPU cores, but as soon as those transactions touch the same data, they need to spend time coordinating. The CPU itself has to coordinate raw memory access across L1, L2, and L3 cache.