You can run git on object storage if you re-make packfiles | Tigris Object Storage

It sure seems that a bunch of companies are trying to ship a git product of some kind as of late. Wonder why that is. Either way, I’m building a Git server backed by object storage as an open-source project. It sounded simple enough to start: Git looks like a filesystem, so let’s use a filesystem as a translation layer on top of object storage to make Git speak object storage. This model worked… ok, I guess? But it didn’t work for real-world size repositories, so I needed a different approach. Git stores everything in Objects, so why not store those as objects in Tigris? Turns out Git packfiles and how they intersected with my (admittedly somewhat terrible) filesystem shim were the main reason why it was slow. I ended up having to invent my own packfile format with a columnar store that’s object storage native. This is the fruit of all of my performance analysis, metrics annotations, and more Texas-style distributed systems work than you can make your k8s cluster shake sticks at. This approach worked surprisingly well for production-sized repositories, so I’m sticking with this new Packfile format for now. It seems the least obtrusive change to make Git Objects feel like object storage Objects, without any client side changes. What is a Git? A miserable little pile of objects! When you make a commit, Git stores the changes you make as objects inside the .git (I’ll call this “dotgit” so I don’t have to write as many backticks) folder. Imagine Git as two things: a sea of objects and named references to individual objects. Each object is a content-addressed and compressed file. Here’s an example from a tiny git repository: $ mkdir ~/tmp/gitexample$ git init && git branch -m main$ echo "Hello, blog!" >> hello.txt$ git add .$ git commit -sm "chore: initial commit" This produces several objects on the disk like this: FIG 01A sea of objects, and a few names into it .git/objects/ refs/heads/main│├── 1c/7a26a901..ec7966 ─────▶ commit 1c7a26a│ │├── 8e/67afbb2e..857bd3 ─────▶ tree 8e67afb│ │ hello.txt└── 9c/c9867337..09fe26 ─────▶ blob 9cc9867"Hello, blog!"the filename is the sha1 of the bytes in the file, so the samecontent is always, everywhere, the very same object If you want to read the contents of an object, it’s compressed, so you have to use a fairly evil looking python oneliner to scoop out the tasty innards: $ file .git/objects/**/* | grep -v directory.git/objects/1c/7a26a901724b4ce766655ac387413fb9ec7966: zlib compressed data.git/objects/8e/67afbb2ee6bdcbb79061dfdfb93febce857bd3: zlib compressed data.git/objects/9c/c9867337c2ebae85ba2350f901e0bcc209fe26: zlib compressed data$ python3 -c "import sys, zlib; sys.stdout.buffer.write(zlib.decompress(sys.stdin.buffer.read()))" < .git/objects/9c/c9867337c2ebae85ba2350f901e0bcc209fe26blob 13Hello, blog! As you can see, the objects are just bare files. Let’s look at a Git repository of the Linux kernel and try to extract out an arbitrary commit. Everything should just be a billionty bare object files, right? It should be easy to find a single commit just by looking for the ID on the disk, right? If only reality were so simple: $ cd ~/Code/linux.git/$ tree objectsobjects├── info└── pack ├── pack-45986f41063f286029742ec12e2c2882b88c5786.idx ├── pack-45986f41063f286029742ec12e2c2882b88c5786.pack └── pack-45986f41063f286029742ec12e2c2882b88c5786.rev3 directories, 3 files Yeah, as I’m sure you guessed just putting everything into their own files won’t scale to something like the Linux kernel. I’m pretty sure you’d run into inode limits like everyone did in the era of fractal node_modules folders. AsideIf you’ve used Node for long enough to remember that, please go get a colonoscopy. Colon cancer is a real concern that too many people overlook for too long and takes too many lives too early. Git works around this by putting objects into packfiles, compressed bundles of objects that store them all in the same file. Here's an example of the packfile efficiency in my checkout of objgit: $ git count-objects -vcount: 756size: 3500in-pack: 448packs: 1size-pack: 321prune-packable: 0garbage: 0size-garbage: 0 If you ever need to “force” git to put bare objects into a packfile, you can run git gc: $ git gc[omitted for brevity]$ git count-objects -vcount: 0size: 0in-pack: 1203packs: 2size-pack: 848prune-packable: 0garbage: 0size-garbage: 0 AsideOne of the beautiful things about implementing Git on top of object storage like I am is that I’m using a platform where the object data is a sea of objects with named references to points in that sea stored in FoundationDB. This is a kind of divine recursion that I don’t really know how to describe the beauty of. As above, so below. Yo dawg, herd you like objects Here's the object count for a copy of the Linux kernel: xe@zohar:~/Code/linux.git$ git count-objects -vcount: 0size: 0in-pack: 11827138packs: 1size-pack: 3876775prune-packable: 0garbage: 0size-garbage: 0 This is eleven million objects, which at a very generous assumption of 10ms per GetObject call means that fetching each of them takes over an hour to fetch them all. The truth is there really aren’t 11M objects as individual files on the disk, they’re bundled into one big happy 3.4Gi packfile. Your typical git repo ends up accumulating them as it makes sense to break them up. My local copy of the Tigris blog has 4 packfiles and 290-ish bare objects. So you’d be thinking, “Oh, if git has packfiles, then why is the rest of this post a thing?” Well, like many things in distributed systems it’s complicated. Packfiles are difficult because they’re designed with local storage and/or mmap in mind. Git constantly writes packfiles to disk and then re-reads them. Filesystem reads in that case are 10 nanoseconds at most (the filesystem cache helps so much here) but doing any network roundtrip is 10 milliseconds at minimum. It’s at least a million times slower because of how reality works. noteOne of the things that /usr/bin/git does that makes integrating it into object storage difficult is the unix-y idiom of writing to a file and then immediately reading back from that file to calculate the hash. In object storage you can’t GetObject something that hasn’t finished a PutObject call. I worked around this previously by writing to the disk and then doing it that way, but the experience kinda sucked in practice. Messin' with Packfiles Each packfile has an index that describes what’s in it. Here’s a view of the index of the packfile made out of that trivial Git repo from earlier uppost: $ git gc # force objects into a packfile$ git verify-pack -v .git/objects/pack/pack-3971f5085c23c38be00e517ed0c64ca7df19b746.idx1c7a26a901724b4ce766655ac387413fb9ec7966 commit 526 366 129cc9867337c2ebae85ba2350f901e0bcc209fe26 blob 13 22 3788e67afbb2ee6bdcbb79061dfdfb93febce857bd3 tree 37 48 400non delta: 3 objects.git/objects/pack/pack-3971f5085c23c38be00e517ed0c64ca7df19b746.pack: ok The commit points to the tree whose file “hello.txt” points to the blob and, blob’s your uncle, you have a repo. Git uses these binary indices to let it know where to look and how far it needs to seek into the packfile to know where to go to get things. FIG 02Eleven million objects, one packfile, one index $ git count-objects -v .git/objects/pack/ count: 0 └── pack-45986f41..c5786.pack 3.7 GiB in-pack: 11827138 packs: 1 eleven million loose files would be size-pack: 3876775 eleven million inodes. so: one file. .idx .pack ┌───────────────────────┐ ┌──────────────────────────────────────┐ │ 0002ff4c.. 0x0000c │ │███ ██ ████ █ ███████ ██ █ ████ ██████│ │ 0031ab90.. 0x0a13f │ └──────────────────────────────────────┘ │ 1c7a26a9..