Object storage is all you need | Tigris Object Storage

ContextTigris makes object storage using a database engine we built on top of FoundationDB: a distributed key-value store. JP, the founder of Ampbase, uses a control plane on Tigris with no database underneath it, and today he's going over which database behaviors he had to build himself and what that cost.Thanks JP! Last time on the Ampbase blog I talked about all the database engines that we don't use and promised to follow up explaining what we actually do. We don't use a relational database. We use Tigris as the storage layer directly, and implement the few database behaviors we actually need on top of the two primitives it gives us. Yeah, yeah, I know; "we didn't need a database" is a catchy title that usually happens about eight (8) months before the inevitable next post being "how we tucked our tail between our legs and moved to Postgres". In practice, when you reach for a database engine you're actually reaching for four basic features: unique constraints, transactions, indices, and history tables. In order to use Tigris' global object storage as a database, we had to implement all of these primitives ourselves. Today I'm going to peel back the curtain and show you how those primitives work so you can understand what actually goes into your database engine of choice. What's actually stored Everything lives in two layers of buckets.

After a while we started adopting more and more features to our API with protobuf options so we can define validation alongside the schema definition among other things. Marshaling and unmarshaling all the JSON got more expensive than we thought, so we switched to using Protocol Buffers directly. Our database handles both formats so if records predate the protobuf migration, everything loads as expected. By using Protobuf, we eliminate the whole problem of managing a database layer: migrations, connections, schemas. The only downside is that Protobuf field names are forever, but to be fair it's about equally as painful to change column names in Postgres, MySQL, or SQLite. The naive way to create a bucket per customer would be to make a bucket per customer, all in the same $bigcloud account and create a new account every time you hit a quota limit. Or have one bucket with prefixes to get around the per-account bucket limit, and rely on complexity in the IAM policy to enforce isolation. All of this sounded rather dull, and Tigris has a Partner Integration Program for exactly this shape anyway.

As someone who has built a few platforms that managed databases in the past, this is the part that most people mess up. Beyond isolation, we need database-like behavior if object storage is truly going to replace our database. But how do you get database-like behavior with the simplicity of object storage? You leverage strong read-after-write consistency, conditional writes, and other primitives as the backbone of everything. Database-like behavior with the simplicity of object storage You can get all the important guarantees of a database from object storage. Don't believe me, and say I will wrest your Postgres from your cold dead hands? Please read on. All you need from your database (and in your object storage) is: Strong read-after-write consistency Conditional writes Uniqueness constraint Transactions Indices History tables We relied on Tigris for the strong read-after-write consistency and conditional writes off the shelf, but we implemented the other four ourselves. Strong read-after-write consistency Everyone expects strong, read-after-write consistency in their databases. Object storage didn't have a strong consistency model until about December 2020. If you want to learn more about how they added strong consistency to object storage, Werner Vogels has a great writeup that goes into the gorey innards. The main thing Tigris gives us is the fact that bucket data is global instead of just bucket names being global. This means that data is strongly consistent when both clients are in the same region, but once you cross regions it gets complicated. Global replication means eventual consistency, i.e. sometimes things can get out of whack while the system synchronizes changes. A core problem for us has been figuring out where we actually need strong consistency and where eventual consistency is good enough. Conditional writes Conditional writes are essentially compare-and-swap. Tigris supports HTTP preconditions on writes: If-None-Match: * writes only when the key doesn't exist, and If-Match: {etag} writes only when the object hasn't changed since you read it. Both evaluate against the object's latest state, within whatever consistency model your bucket's location type gives you. The choice of consistency model will become more important later. But the important thing here is that once you have compare-and-swap, you basically have the core primitive underlying every database. Uniqueness without UNIQUE Think about database indices as having two properties that make it more efficient to find data: precomputing lookups and ensuring the same data can't be stored twice. This is the difference between CREATE INDEX and CREATE UNIQUE INDEX. Most of the time you don't end up creating indices on your primary keys or UUIDs to make the lookup more efficient, you make them so you can't store the same user email address or unique identifier twice. In order to get the uniqueness property of indices in our database, we leverage a combination of content-aware storage for uniqueness and conditional writes. This lets us make sure things can't be stored twice. We attempt to create channels or users by passing the If-None-Match: * header in PutObject calls. This tells Tigris to reject the data if anything is already stored in that key. When two app instances try to write different data to the same place, Tigris decides which one wins and gives the loser an error which we handle and report back to the user: switch {case err == nil: return nilcase isPreconditionFailed(err): // Another writer created the slug concurrently. Re-read to determine // whether this is idempotent (same channelID) or a conflicting mapping. return s.handlePutConflict(ctx, slug, channelID, err)} Annoyingly this doesn't tell the client why they lost the race. In practice this may mean that another app instance wrote there first, a partial failure in a multi-region bucket write turned out a bit wonky, or a retry went wild and wasn't surfaced any other way. The only way to figure out what's going on is to read the data out of the database. Not doing this creates a really confusing scenario for the user where they can't create something because they already created it just now. It's the kind of problem that you only get in distributed systems. Something that doesn't make any sense when you say it out loud to the point that it's hard to handle because you lack the temporal relativity constructs to express it cleanly. Aren't computers great?