There's no reason for software to be slow anymore

The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance. In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences1, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality. And Michael Malis has noted There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to write a JIT compiler. This is the thesis behind pgrust. Databases historically were the hardest piece of software to build and were limited because of that. Now, with AI, we can be more ambitious about the type of software we build. Optimizing for a class of workload Let's try this out with FRE, the regex engine we built in the last post. Recall that it was created by having an agent loop for a month on improving regex engine performance with access to the rebar regex benchmark suite. This resulted in FRE being heavily overfit to rebar until we warned our agent that we had a holdout benchmark, which caused the agent to generalize the optimizations enough that performance was ok-ish on our holdout. There's no particular reason to use a "software factory" regex engine that doesn't beat a well-tested regex engine on holdout benchmarks, but one notable thing about FRE was that the native AOT compiled version did quite well at longer searches. We noted that, it stands to reason that one could run the native code compiler in another thread while ripgrep was running its normal matcher and then cut over to the native code when it finished compiling and generally get better performance. Of course this will generally result in worse performance for short queries as we lose a thread to compilation, but I care a lot more about how long ripgrep takes when it runs for many seconds or minutes than when it runs for a few seconds, so I'm ok with that tradeoff. In the same way we could build a regex engine in a few minutes of human time, we can also just try this experiment in a few minutes of human time. I typed a few sentences and an agent went and did the work to allow this to happen (which would be a decent chunk of code surgery for a human) and it ran the benchmark on actual ripgrep queries that come from my codex history. For longer queries, we see a 2x-4x performance improvement here for a few very simple queries. But most queries are more complex, and when we run on representative holdout queries, for queries where AOT should be enabled2, we get about a 7% speedup. Not an earth shattering result, but also not a bad outcome for spending a few minutes typing to codex (and it's still doing more optimization and will presumably speed things up further). Build an index? This is arguably a silly thing to do, since if we're repeatedly searching for text on a computer, the obvious thing to do to speed that up isn't to write a native code compiler for regex matching, it's to create an index. But the point here is just that this kind of technical work, which used to take a fair amount of time and expertise, can just be done trivially now. And if we wanted to build a text index, it just so happens that I worked on BitFunnel, the Bing search index that was specialized for constant/fast text ingestion that won Best Paper Award at SIGIR, so I can think of a few experiments to try if we're going to build a fast local index of our entire machine (the projects I've seen seem to be intended to index your code directories, but what really kills my machine performance is when codex decides to run ripgrep against huge temporary directories with a ton of generated files and then expands to looking at my whole machine when it misses, so I'd want an index of my entire disk and not just of the code for some projects). If I were working at an AI lab and had access to things like SOTA models running on Cerebras chips or other accelerators that greatly increase tok/s and therefore load/demand for search, I might actually survey the existing indexers to see if they're fast enough or if I'd want to build something custom myself. While the open source version of BitFunnel "only" contains a bytecode interpreter and one JIT, the Bing version contains multiple JIT compilers. A project that did that level of optimization used to be a major undertaking, but "I could do that in a weekend" is now actually true for some of these kinds of projects. With my lowly $200/mo account, I think a somewhat faster ripgrep plus any off-the-shelf index is fine, so maybe this fast-ingesting whole-machine index project can be left as an "exercise for the reader (who works at an AI lab)". Optimizations are cheap The drastic reduction in the cost of optimizations has been true going back to November 2025 and maybe even somewhat before then with public models (and I'm sure before that still with what folks at AI labs had access to). For an example from the GPT-5.1 or 5.2 days, with no knowledge of game AIs, I tried building an Azul AI. This ended up being the strongest AI in the world for the game by a pretty large margin. From reading the thesis that describes the 2nd strongest AI, I think my AI is probably a bit better on the "AI" side of things, but the main place it wins is on optimization. For example, that other AI is single-threaded and my AI is multi-threaded. Since I have a native code version as well as a heinous wasm shared memory + javascript version, and two different search architectures for two different versions, which "require" completely different multi-threading algorithms (minimax for a very small and fast net and MCTS for a larger net), this would've been a fairly large undertaking if done by hand. And, because I let an LLM pick the multi-threading algorithm based on its own (incorrect) reasoning a couple times before spending 30 minutes reading about multi-threading algorithms for game AIs myself, I ended up re-writing (having codex re-write) the multi-threading algorithm multiple times. There's a bunch of standard stuff it makes sense to do to debug and verify a multithreading algorithm for something like this, like implementing replay from debug logs that can reproduce bugs despite the algorithm being nondetermistic. Doing that alone would've probably been days to a week of work had I done it by hand, but it's exactly the kind of thing an agent can trivially do in a loop (just have it try to replay logs and insert logging for non-determinism every time you don't get a perfect replay). A lot of the tedium it used to take to get a tricky optimization like this working is gone. This also applies to a lot of other tricky optimizations. From having written CPU microcode, done CPU verification, worked on optimizing a search engine index, etc., I have a lot of experience looking at optimizations and thinking "hmm, this would increase performance by 2%, but it's going to take N person-days to verify that this tricky optimization works" and making a call to go ahead or not based on whether or not it's worth the time to get the optimization working. Now that this N has dropped by a tremendous factor (variable but, in terms of human time, frequently 1000x / 10000x / 1000000x), the number of these kinds of optimizations it makes sense to do goes way up. The same goes for optimizations that you aren't sure will work out. I used to sometimes look at an optimization that I wasn't sure would speed things up and think "this will take M hours to implement to the point where we have a good enough measurement to guess at the performance impact". Many more of those optimizations make sense to try out now. Going back to the game AI case, at least for the AI I tried, it seems like you gain about 100 Elo for every doubling in speed (more than in chess, I suspect because draws are very rare). Just adding multithreading alone is enough to wipe the floor with an otherwise comparable AI on a large machine. If you stack in 10-20 more optimizations that seem too annoying for most people to do by hand, the difference in strength is tremendous and it's not really reasonable to try to keep up with a hand-written AI3.