Three ways to smuggle SQLite into Nix

The core of nixpkgs-multiverse, when you strip away the Nix API and the CLI, is an index. It is a map from (attribute, version) to the revision that shipped it as a JSON file.11There are actually a few other files that drive other features such as the statistics or “fast mode”, but they are all JSON as well. $ ls -lh index/ -rw-r--r--. 1 fmzakari fmzakari 7.5M Aug 19 13:57 history.json -rw-r--r--. 1 fmzakari fmzakari 5.3M Aug 19 13:57 versions.json As of 9cc0209, versions.json is 5.3 MiB and history.json is 7.5MiB covering 305,492 package versions across 31,904 packages and 1,534 revisions. The Nix API loads the JSON files lazily and are all read via builtins.fromJSON: index = builtins.fromJSON (builtins.readFile ./index/versions.json); I would like to enrich the data with even more information however it comes at a cost: mo’data, mo’problems. The goal of the project is to minimize the number of Nixpkgs that are downloaded. If we merely swap fetching huge Nixpkgs for huge JSON, it’s not a clear win. For now we have to be judicious about what we store in the JSON files and think of clever encoding schemes to make the data small and compact. If we were not constrained to the Nix builtins, we would leverage established technologies to efficiently encode our dataset that allow multiple query access patterns: databases! Let’s say we were not restricted to JSON, do we have any other options? §One lookup costs the whole file Why are large JSON files so problematic? builtins.fromJSON is eager. There is no lazy JSON in Nix, no streaming parse (i.e. “just give me this one key”). The moment you touch the result you have parsed all 5.3 MB and materialised all 305,492 values on the Nix heap. In the case of the multiverse, asking for one package costs the same as what asking for all of them. Note The lookup itself is not the problem. Nix attribute sets are a sorted array, so access is a binary search, not a scan. The cost is entirely in the JSON parse and in allocating the values and downloading a large file. If we want to do alternate questions over the index, we have to make sure we keep the answers efficiently stored to better match the access pattern. What we want is obvious. We want a way to efficiently encode the data and a declarative way to define queries: we want SQLite!22nixpkgs-multiverse already exports a SQLite database as a package to help others explore this data. $ sqlite3 index.db "SELECT version, rev FROM versions WHERE attr='hello'" 2.10|728 ... 0.01s, 4 MB Nix by default cannot do this. Unfortunately there is no builtins.sqlite, although I think there should be… Turns out though there are knobs we can touch or sources we can patch to get what we want anyways, albeit each one has a caveat. 😈 §One: builtins.exec I was surprised I did not know about this builtin, and it has been around since release 1.11.9 in April 2017. It is the ultimate escape hatch for a variety of use-cases when you simply can’t get them done with what’s available. builtins.exec takes a list of strings, runs the program, and parses its stdout as a Nix expression. It is gated behind a setting that makes it clear it’s unsafe. $ nix eval --option allow-unsafe-native-code-during-evaluation true \ --expr 'builtins.exec [ "/bin/sh" "-c" "echo 42" ]' 42 For integration, SQLite is perfectly capable of printing the Nix syntax. We never need a serialisation format in between as we make SQLite emit the attrset directly: let versionsOf = attr: builtins.exec [ "${sqlite}/bin/sqlite3" "-noheader" "-separator" "" "./index.db" '' SELECT '{' || group_concat( '"' || version || '" = ' || COALESCE(CAST(rev AS TEXT), 'null') || ';', ' ') || '}' FROM versions WHERE attr = '${attr}'; '' ]; in versionsOf "hello" $ nix eval --impure -f query.nix \ --option allow-unsafe-native-code-during-evaluation true { "2.10" = 728; "2.12" = 822; "2.12.1" = 1369; "2.12.2" = 1486; "2.12.3" = null; "2.7" = 0; "2.8" = 13; } The caveat is that every query is now a fork, an exec, a process image of SQLite, and a re-parse of the output through the Nix parser. If you do not plan to execute many queries that overhead is likely acceptable given the simplicity of the integration. §Two: builtins.importNative From researching builtins.exec, I stumbled upon builtins.importNative. It takes a path to a shared object and a symbol name, dlopens it, and calls that symbol. It landed in 1.8, December 2014.33The C++ field was originally called enableImportNative and was renamed to enableNativeCode for exec. The shared object must implement the following signature: extern "C" typedef void (*ValueInitializer)(EvalState & state, Value & v); We can define a new native function that returns the versions for our input: extern "C" void nix_sqlite_versions(EvalState & state, Value & v) { v.mkPrimOp(new PrimOp{ .name = "nix_sqlite_versions", .args = {"dbPath", "attr"}, .arity = 2, .impl = versions, }); } The implementation is ordinary C++ using the Nix API. Below is a snippet of the implementation, making sure to cache our sqlite3 handles to avoid the same startup penalty as builtins.exec: /* The whole point: the database handle outlives a single query, so the b-tree pages we touch stay warm for the rest of the evaluation. */ std::map<std::string, sqlite3 *> handles; void versions(EvalState & state, const PosIdx pos, Value ** args, Value & v) { std::string path(state.forceStringNoCtx(*args[0], pos, "...")); std::string attr(state.forceStringNoCtx(*args[1], pos, "...")); // cached across calls auto * db = openOnce(state, pos, path); sqlite3_stmt * stmt = nullptr; sqlite3_prepare_v2(db, "SELECT version, rev " "FROM versions " "WHERE attr = ?1", -1, &stmt, nullptr); sqlite3_bind_text(stmt, 1, attr.data(), attr.size(), SQLITE_TRANSIENT); /* ... collect rows ... */ /* Build the attrset directly. No text ever exists. */ auto bindings = state.buildBindings(rows.size()); for (auto & [version, rev] : rows) { auto & slot = bindings.alloc(state.symbols.create(version)); if (rev) slot.mkInt(*rev); else slot.mkNull(); } v.mkAttrs(bindings); } Using it looks like this: $ nix eval --impure \ --option allow-unsafe-native-code-during-evaluation true \ --expr '(builtins.importNative ./libnixsqlite.so "nix_sqlite_versions" ) "./index.db" "hello"' { "2.10" = 728; "2.12" = 822; "2.12.1" = 1369; "2.12.2" = 1486; "2.12.3" = null; "2.7" = 0; "2.8" = 13; } §Three: a giant Nix file This section was added after publishing based on an idea from rickynils. Nix is often described as resembling JSON and there is a very easy translation from JSON to Nix. What if instead of reading JSON we read the same contents but as a .nix file? Theoretically it should have no parser boundary, no fromJSON, and no serialisation format at all. The index becomes an expression the evaluator already knows how to read. The idea would be to leverage Nix’s laziness. Nix attribute set values are thunks, so in principle you should be able to import a very large expression, touch one attribute, and never pay for instantiating the rest. Transforming the index is a dozen lines of Python, and produces something very similar to the JSON: { revisionCount = 1534; attrs = { "2048-in-terminal" = { "2015-01-15" = 157; "2017-11-29" = 166; }; "2bwm" = { "0.2" = 166; }; "389-ds-base" = { "1.3.3.9" = 14; "1.3.5.15" = 100; "1.3.5.19" = 166; }; # ... 31,901 more }; } 6.0 MiB of Nix, against 5.3 MiB of JSON, holding identical data. $ nix eval --impure --expr '(import ./index.nix).attrs.hello' { "2.10" = 728; "2.12" = 822; "2.12.1" = 1369; "2.12.2" = 1486; "2.12.3" = null; "2.7" = 0; "2.8" = 13; } §Four: builtins.wasm Determinate Systems shipped another option in March of 2026: builtins.wasm, which calls a function inside a WebAssembly module.44Eelco gave a talk about this at SCALE 23x.  The motivation was similar to wanting to extend Nix surface area but avoid expanding builtins. Wasm is sandboxed and deterministic, so unlike the two builtins above, the goal is to provide a safe escape-hatch. WebAssembly is a binary instruction format for a stack-based virtual machine. The claim is that it is well suited for Nix because it has deterministic execution, which is a lot more restrained than a backdoor builtins.exec. §Writing a module A module needs to export memory, an initialiser called nix_wasm_init_v1, and the entry point. #![no_std] #![no_main] type ValueId = u32; #[panic_handler] fn panic(_: &core::panic::PanicInfo) -> ! { core::arch::wasm32::unreachable() } // Host functions supplied by the Nix evaluator. #[link(wasm_import_module = "env")] unsafe extern "C" { fn get_int(v: ValueId) -> i64; fn make_int(n: i64) -> ValueId; } #[unsafe(no_mangle)] pub extern "C" fn nix_wasm_init_v1() {} fn fib(n: i64) -> i64 { if n <= 1 { 1 } else { fib(n - 1) + fib(n - 2) } } #[unsafe(no_mangle)] pub extern "C" fn fib_entry(arg: ValueId) -> ValueId { unsafe { make_int(fib(get_int(arg))) } } Nixpkgs already includes the target for cross-compilation, so making one is pretty straightforward: pkgs.runCommand "nix-wasm-rust-fib" { nativeBuildInputs = [ pkgs.rustc pkgs.lld ]; src = ./modules.rs; } '' mkdir -p $out rustc --target wasm32-unknown-unknown --crate-type cdylib -O \ -o $out/modules.wasm $src '' $ nix eval --extra-experimental-features wasm-builtin \ --expr 'builtins.wasm { path = ./modules.wasm; function = "fib_entry"; } 30' 1346269 You call back into the evaluator through the Nix API functions, so a wasm module builds real Nix values, similar to builtins.importNative minus the footgun. §Can I haz SQLite? SQLite ships an official wasm build, so the pieces seem to be sitting right there and the gears in my mind began to turn. Initial attempts to try and load a SQLite database with the traditional Nix builtins were a bit of a failure as Nix strings cannot contain NULL bytes. $ nix eval --impure --expr 'builtins.stringLength (builtins.readFile ./index.db)' error: the contents of the file '/tmp/mvsql/index.db' cannot be represented as a Nix string