From monolith to Lakebase to LTAP: rethinking the database from storage up

When I started my PhD at UC Berkeley 16 years ago, my advisor told me: "OLTP databases are a solved problem. They work. Focus on analytics." We were at the early innings of being able to collect far more data, structured and unstructured, and apply machine learning (which we now call “AI”). So I took the advice and joined my cofounders on the research project that became Apache Spark, and later on we started Databricks.As we built Databricks, we started using various databases out there, and we realized OLTP databases were far from a solved problem: they were clunky, difficult to scale, and incredibly fragile. We were frustrated enough at some point that we asked ourselves what an OLTP database would look like if we were to design it today. That question led to Lakebase, our serverless Postgres database.This post takes a deep dive into the Lakebase OLTP architecture. We start at the storage layer of a traditional monolithic database to see where the pain comes from, then we look at how Lakebase rearranges those same pieces into independent, externalized services. Finally, we turn to LTAP, where that same architecture lets transactions and analytics run on a single copy of the data, in real time, without the delays and extra cost of CDC or "mirroring.”The database as a monolithThe vast majority of databases running in the world today are monoliths. This includes MySQL, Postgres, classic Oracle. Lakebase is built on Postgres (as it happens, was also born at Berkeley), so we will be using Postgres as the primary example here, but most databases work similarly: You provision one machine that runs the database engine and the storage. In these database systems, there are two things on disk that matter the most: the write ahead log (WAL) and the data files.When you commit a transaction, the database does not immediately go and rewrite the data files. That would be slow, because the rows you are touching are scattered across the file in places that require random I/O. Instead, the database first appends a description of the change to the WAL, which is a sequential log on disk. A transaction is considered committed the moment that log entry is durably written. Only later, asynchronously, does the database go back and update the actual data files to reflect the change.

One simple way to think about this: the WAL exists to make writes fast (and safe), and the data files exist to make reads fast. The log lets you commit a transaction with a single sequential append instead of a scattering of random I/O. The data files let you answer a query by reading the current state directly, instead of replaying the entire history of the database from the beginning of time. (If you want to understand all the intricate details of this design, read the 69-page long ARIES paper. Be warned that this is one of the most complex papers in computer science.)As this design has become the foundation for virtually all databases out there, the monolithic architecture also creates a lot of challenges:Data loss from misconfiguration. A commit is only as durable as the disk flush behind it. If the database, the operating system, or the storage layer is configured such that a write to the WAL is acknowledged to the client before it has actually been flushed to durable media, then a commit can vanish in a power loss or kernel panic. These settings are subtle, easy to get wrong, and the failure is often silent. The operating system might even decide to lie to you about flushing!Data loss from node loss. Even with flushes configured correctly, the WAL and the data files live on one machine. If that machine's disk dies, the data on it dies too. Note that network attached storage or redundancy techniques like RAID-1/RAID-10 can improve durability but do not fundamentally solve this issue. If the storage mount dies, so does your data access.Scaling reads requires a physical clone. When one box can no longer serve your traffic, the standard answer is to add a read replica. But a read replica is a full physical copy of the entire database, streaming the WAL from the primary and replaying it. Provisioning one means copying the whole dataset and then catching up on the log. For a large database, that is not a quick operation and might even bring down the database.High availability also requires a physical clone. Surviving the loss of the primary means running at least one additional standby node, which is itself a complete physical copy of the database kept in sync from the WAL.

You pay for at least twice the infrastructure, you wait a long time to bring a standby online, and you have to set up synchronous replication to avoid losing any data when the primary goes down. (In practice, many recommend 3 or more nodes.)Analytics contend with your transactional traffic. A heavy analytical query runs against the same hardware resources as your latency-sensitive transactional workload. One large reporting query or one GDPR cleanup can degrade your main OLTP queries. You can run the analytical queries in a separate replica, but you end up paying for the replica and still don’t get optimal performance due to the row oriented nature of OLTP storage (analytics requires column-oriented storage for high performance).Almost every one of these problems traces back to the same root cause from the monolithic architecture: the WAL and the data files are stored inside a single machine. Durability is tied to that machine's disk. Scaling and availability require physically cloning that machine. Workloads interfere because they share that machine.Lakebase architectureIf you were to redesign an OLTP database today, you’d start with the components of the modern cloud: cheap and highly durable cloud object storage paired with elastic compute. This is the path the Neon team took on and the foundation of what became Lakebase.The core move is to make the Postgres compute instances stateless. We do this by externalizing the WAL and the data files on local disks into purpose-built, independently scalable services. The compute layer becomes a stateless Postgres engine that can be started, stopped, and replicated freely, because it no longer owns the data. Let’s see how these two storage services can work together to solve the aforementioned challenges without sacrificing performance.Scaling writes: WAL becomes SafeKeeperIn a monolith, a write is made durable by flushing it to the local disk. In a Lakebase, the WAL is externalized to a distributed storage service called the SafeKeeper. Instead of relying on disk flush for durability, a commit is made durable by replicating the log record across a quorum of SafeKeeper nodes using Paxos-based network replication. There is no longer a disk whose failure loses your data, and there is no longer a misconfigured flush quietly undermining your durability guarantee.

It’s natural to ask at this point: does moving commits from WAL on local disk to WAL on SafeKeeper increase the write latency due to the extra network hop? The answer is no. For any serious Postgres deployment that cares about durability and availability, you’d have to set up synchronous replication which requires the extra network hop, so externalizing the WAL into SafeKeeper does not incur additional overhead. As a matter of fact, due to how Postgres works internally, the combination of SafeKeeper and PageServer can lead to 5X higher write throughput and 2X lower read latency.Scaling reads: data files become PageServerThe data files move to another distributed storage service called the PageServer. The WAL is streamed from the SafeKeeper into the PageServer, and the PageServer asynchronously applies those changes to its version of the data, materializing pages into low-cost cloud object storage (the lake). You can think of the PageServer as a write through cache for the underlying object storage.This is similar to the WAL-then-data-files relationship from the monolith, except the two halves now live in separate, independently scalable services connected by the network instead of sitting on the same disk. When a page is requested from the PageServer, and if the PageServer does not yet have the latest version yet (keep in mind changes are written to the SafeKeeper first before making their way to the PageServer), the PageServer applies the logs from the SafeKeeper to reconstruct the latest state.A similar question: does moving data files from local disks to PageServer increase the read latency due to the extra network hop? The answer is also no for all practical purposes. The system is designed to isolate and minimize the latency impact through aggressive, multi-layered caching. To fetch a page, Postgres first looks up its buffer pool, which is in the node’s local memory. When the page is not present, it looks up a local disk cache. It only needs to go to the PageServer if there is a cache miss. Because a compute node can be configured with local memory and disk capacities identical to a monolithic setup, your local cache hit rate remains unchanged. For the vast majority of operations, read latency is indistinguishable from a monolith, but you gain the benefit of decoupled, virtually infinite storage.

What this unlocksOnce the WAL lives in the SafeKeeper and the data files live in the PageServer, a long list of capabilities that were hard or impossible in the monolith become natural consequences of the architecture. The following are already widely available as part of the Lakebase product on both Databricks and Neon:Still Postgres. This is real Postgres, so the wire protocol, SQL, drivers, and extensions all work as-is. Unlimited storage. Data lives in cloud object storage rather than on a provisioned local disk. You are no longer sizing a box to a capacity ceiling. Storage is, for practical purposes, infinite.Serverless, elastic compute. Because compute is stateless, it can scale up instantly under load and scale all the way down to zero when idle. You stop paying for a large machine to sit there waiting for traffic.Durable writes and zero data loss. A commit is durable once it is replicated across SafeKeeper nodes via Paxos, not when a single local disk claims to have flushed it. The loss of any individual node does not lose committed data.Simpler high availability. In the monolith, HA meant maintaining a second full physical clone, paying twice, and still risking data loss at cutover. Here, the durable state already lives in a replicated storage layer that is independent of any single compute instance. Failing over no longer means promoting a separate physical copy of the database and hoping the last segment of the log made it across.Instant branching, cloning, and recovery. This is my favorite. For code, creating a branch is a sub-second, fully isolated copy of the entire codebase, and we do it dozens of times a day without thinking about it. For a monolithic database, cloning means physically copying the whole dataset, which is slow, expensive, and risky to the production system. When the data lives in an externalized, versioned storage layer, a branch or a clone is a metadata operation rather than a physical copy. You can branch a large production database in seconds, run an experiment or a risky migration against the branch, and throw it away. Recovery to a point in time works the same way. The database finally moves as fast as your code.Separating compute from storage is not itself new. The previous post discussed the generation 2 cloud databases that had done this.