A Search-and-Inference Database from Scratch in Pure Zig — Antfly Research

My colleague Rowan summarized our ambitious goal for Antfly over a year ago: perfect search! This goal is silly, ambitious, unobtainable, and perfect for us. Hearing grandiose technologists talk about perfect search still strikes me as something out of an episode of Silicon Valley, but I love an impossible goal, and my inner tech hipster appreciates the irony. Aim for perfect search and you miss, but you miss somewhere interesting. So I'll walk you through why we did the thing you are never supposed to do with a new startup: we rewrote the product. Antfly v0.1 launched a document storage, full-text, vector, and graph indexing engine in Go. Antfly v0.2 launches the same engine in Zig, with zero dependencies (well, none that get to run the show, but more on that below). First principles# I don't hear people talking about first principles as often as I used to, but they're still important to me and to how I make decisions. When building the first version of Antfly, the idea was to fill a gap in the database market: a schema-ish, friendly query engine for indexing like Elasticsearch, closer in scale to Postgres than Iceberg, as easy to use as Mongo, and as easy to operate as Google's Spanner or Bigtable. (And I thought perfect search was too lofty... HA!) When building the first version I didn't have the same sort of software tools (cough cough Codex, Claude, Aider, Pi) available to assist in development, and the hard problems I wanted to tackle were the ones CockroachDB had similarly chosen Go for: distributed systems and concurrency. Rust can guarantee memory safety through lifetimes, but memory safety wasn't the hard part, and Go is infinitely more readable to me than Rust ever was. Plus, Go had the most battle-tested Raft implementation out there (etcd's), and I was only one person, working on weekends and on my Fridays, trying to build an Elasticsearch DSL on top of Bleve as well. Caring about the model, not the embeddings# Fast forward a little bit and embeddings started to become a thing. word2vec had shown that a vector could actually carry meaning, and the first practical embedding models meant an average dev could build Google-lite semantic search for their app. At work, our scale of vector storage was so small that a top-k could be an exhaustive search, but I read about all the fun algorithms behind Pinecone, Vertex AI Vector Search, Elastic, Mongo, CockroachDB, and pgvector after an old coworker rolled his eyes at the ridiculous Pinecone seed round. Pinecone might have been overvalued, but I think one of its most interesting ideas was overlooked: users could care about the model instead of having to care about the embeddings. It made me think about Postgres, and how for the most part a user can avoid knowing about B-trees and other indexing algorithms, or how in Bleve and Lucene you can avoid knowing about S2 indexes, finite state transducers, and so on. Vector databases and indexes, on the other hand, required you to know and care about HNSW, SPFresh, RaBitQ, and the rest. I decided to try my hand at implementing these algorithms from their papers and blog posts in the initial version of Antfly, and it was wildly successful. I was able to make a semantically searchable Wikipedia using my laptop, Antfly, and Ollama! Embedding generation was so slow that the database being a little bit slower was not a big concern for the initial implementation. Enter Zig# Concurrently with all this, I had tried my hand at implementing VSR, Protobuf, and an LSM in Zig a few years prior, after stumbling upon TigerBeetle. I really liked the readability of Zig, the concurrency primitives, the people implementing the language, and the interoperability with C. But Zig was too green for my weekend database project and lacked a lot of the heavy lifting: Raft, full-text indexing, a portable, battle-tested LSM, yada yada yada. I took a lot of the spirit of the TigerBeetle folks with me, though, and put myself to work incorporating VOPR testing with TLA+ trace validation (Rowan's post on formal verification with coding agents covers that), built on the new Go mock time and on prior art from etcd's Raft TLA+ spec and trace validation. Then std.Io started making a big splash across the technoverse when Zig decided to make some serious overhauls to the language. I had been reading the GitHub design threads on the subject for a while and thought it was all pretty cool from an engineering perspective. Antfly had just released v0.1 and I had a breath of air to start thinking about what came next. So I started to play around with Zig and the new std.Io work, trying to implement our LSM using the async I/O and std.Io.Evented machinery Zig had started to expose. At the same time, I wanted to see how far I could take the software factorization of our code, and it felt like Raft, TLA+ specs, clear traces, and language-agnostic tests were the perfect hill to climb. So I went to work directing traffic and building Raft, full-text indexing, an LSM, and... HTTP/2 (and our Raft transport, which was over HTTP/3 and QUIC). I had also rewritten most of our Go-based end-to-end tests in Python, both to make sure the coding agents couldn't "cheat" by reaching into the Go code and to make sure our Python SDK was solid. This turned out to be the perfect language-agnostic framework for transpiling to Zig too. Why rewrite# A conversation with James and another with Drew really solidified the idea for me when we talked about what we were optimizing for: asymmetric outcomes and reliability. Most startups land on their face, so the expensive bet, writing every high-performance dependency ourselves, is the one worth making. A user MUST know and trust that their database works, and a user wants that database to fly, not sprint (it needs to be astonishingly fast). People were already building all sorts of interesting projects on top of Antfly. To run it from any other programming language or in the browser, though, we'd need something like Rust or Zig to give the code a C-compatible, WASM- and WebGPU-compatible interface. I wanted Antfly to be the grand unified theory of databases: a machete for old-school use cases and traditional apps, and the perfect Swiss army knife for the AI and semantic use cases nobody has seen yet (perfect search didn't seem lofty enough anymore). Something a developer could embed in a sandboxed environment (laptop, unit tests, Lambda), run at average application workloads (PostgreSQL, Cockroach, Mongo), or run at analytic scale (data warehouse, serverless). And I wanted Antfly to be even more reliable in all of those environments while maintaining or exceeding the performance goals we had set for ourselves. A rewrite of everything from the ground up gives you a chance to build all your dependencies in a purpose-aware shape, baking in resource management, priority scheduling, and testing ideology from day one. So why Zig and not Rust?