Senior SWE-Bench, napkin math, and winter tires

We're going to look at three different kinds of benchmarks, one set of calculations for baseline numbers for performance "napkin math" estimates, one set of AI model evals, and one on car tires. To build my intuition for things, I like thinking about them before seeing the explanation, so these are presented with the benchmark information first and the explanation later in case you want to think about your answer before seeing my thoughts. 29. A friend of mine is reviewing performance orders of magnitude to prep for computer performance interviews and found that https://github.com/sirupsen/napkin-math (5.4k stars) was the top hit. The README's tables include: Napkin Math performance estimates Operation Latency Throughput 1 MiB 1 GiB Sequential Memory R/W (64 bytes)0.5 ns ├ Single Thread20 GiB/s50 μs50 ms ├ Threaded200 GiB/s5 μs5 ms Network Same-Zone10 GiB/s100 μs100 ms ├ Inside VPC10 GiB/s100 μs100 ms ├ Outside VPC3 GiB/s300 μs300 ms Hashing, not crypto-safe (64 bytes)10 ns5 GiB/s200 μs200 ms Random Memory R/W (64 bytes)20 ns3 GiB/s300 μs300 ms Fast Serialization [8] [9] †N/A1 GiB/s1 ms1s Fast Deserialization [8] [9] †N/A1 GiB/s1 ms1s System Call300 nsN/AN/AN/A Hashing, crypto-safe (64 bytes)100 ns1 GiB/s1 ms1s Sequential SSD read (8 KiB)1 μs8 GiB/s100 μs100 ms Context Switch [1] [2]10 μsN/AN/AN/A Sequential SSD write, -fsync (8KiB)2 μs3 GiB/s300 μs300 ms TCP Echo Server (32 KiB)50 μs500 MiB/s2 ms2s Random SSD Read (8 KiB)100 μs70 MiB/s15 ms15s Decompression [11]N/A1 GiB/s1 ms1s Compression [11]N/A500 MiB/s2 ms2s Sorting (64-bit integers)N/A500 MiB/s2 ms2s Proxy: Envoy/ProxySQL/Nginx/HAProxy50 μs??? Network within same region250 μs2 GiB/s500 μs500 ms Premium network within zone/VPC250 μs25 GiB/s50 μs40 ms Sequential SSD write, +fsync (8KiB)300 μs30 MiB/s30 ms30s {MySQL, Memcached, Redis, ..} Query500 μs??? Serialization [8] [9] †N/A100 MiB/s10 ms10s Deserialization [8] [9] †N/A100 MiB/s10 ms10s Sequential HDD Read (8 KiB)10 ms250 MiB/s2 ms2s Random HDD Read (8 KiB)10 ms0.7 MiB/s2 s30m Blob Storage GET, if-not-match 30430 ms Blob Storage GET, 1 conn (128KiB)80 ms100 MiB/s10 ms10s Blob Storage GET, n conn (offsets)80 msNW limit Blob Storage LIST100 ms Blob Storage PUT, 1 conn (128KiB)200 ms100 MiB/s10 ms10s Blob Storage PUT, n conn (multipart)200 msNW limit10 ms10s Network between regions [6]Varies25 MiB/s40 ms40s Network NA Central <-> East25 ms25 MiB/s40 ms40s Network NA Central <-> West40 ms25 MiB/s40 ms40s Network NA East <-> West60 ms25 MiB/s40 ms40s Network EU West <-> NA East80 ms25 MiB/s40 ms40s Network EU West <-> NA Central100 ms25 MiB/s40 ms40s Network NA West <-> Singapore180 ms25 MiB/s40 ms40s Network EU West <-> Singapore160 ms25 MiB/s40 ms40s What's wrong with this benchmark? 30. I keep seeing people reference DeepSWE and Senior SWE-Bench to "prove" that their favorite model is better than other people's favorite models or just as generally good benchmarks, such as in What's wrong with these benchmarks? 31. People frequently say that winter tires are superior to all-season tires in cold weather. For example, on googling "all season tires during winter cold" (no quotes), the Google AI summary leads with All-season tires lose traction and stiffen in freezing winter temperatures. Their rubber compounds are designed for warmer weather and become hard below 7°C (45°F), leading to significantly longer braking distances and reduced grip ... The rubber in all-season tires cannot maintain pliability in sub-zero temperatures, causing them to perform more like hard plastic on snow and ice. Given that there are a lot of internet comments in the training data, this is a reasonable comment, in that I frequently see variations on this comment on discussions of which tires one should use. What's wrong with this benchmark? 29. Napkin math numbers Random memory access latency The thing that immediately jumped out to my friend (Jamie) as odd was random memory R/W listed as 20ns, since random memory R/W is implied to be a real DRAM read (as opposed to a cache hit), which he felt this should be around 100ns for an order of magnitude estimate. As we were chatting about this, he noted that the README uses the term "latency" for some things that aren't really latencies. Then, when he pulled up the code for random memory read latency, he found the following (if you want another exercise, consider what's wrong with the following code before reading the explanation below): while test.i < test.vec.len() { let random_index = test.order[test.i]; black_box(test.vec[random_index]); test.i += 1; } Jamie noted that there's no data dependency between the loop iterations, so the memory reads here happen in parallel. Since the alleged latency number is determined by finding the average time for an access, this is incorrect because the CPU can have multiple loads in flight at once. If you wanted to measure latency this way, you'd have to introduce a dependence between loads, to prevent overlapping accesses (we discussed a related topic in exercise 19, covered in part 4 of this series). Random SSD read I agree with all of Jamie's comments, although I didn't really flag the use of the term latency myself because maybe it's shorthand for latency in some cases and something a bit latency-like in other cases (such as reciprocal throughput), which makes the table simpler. What first jumped out to me, besides the memory latency number, was some of the other numbers. For example, random SSD read is listed as 100 us / 70 MB/s. You can get much faster (as well as much slower) SSDs. For example, if you have a fast (but non-exotic, e.g., non Optane) device, you might see latencies below 40us, e.g., the Kioxia CD9P-R was measured at ~30 us here. Other than for some trivial scripts, I haven't worked on anything where I care about disk performance, so I don't have an intuition for what numbers someone would want to have in mind1, but I also wonder if having a single number for random read latency and throughput is less useful than it is for DRAM accesses. Whenever I've looked at disk benchmarks, it seems like there's a huge range of results based on read size, queue depth, and number of jobs (e.g., see the previous link on the Kioxia CD9P-R). Of course, there are analogous factors that influence DRAM latency and bandwidth, but it seems like you're much more often in a regime where knowing one or two numbers is helpful when thinking about memory accesses. Since I don't know anything about disk performance, I asked Peter Geoghegan, who's done work on Postgres disk performance; he concurred and also wrote some additional comments on the complexity of disk performance below If we look at the code for this SSD random read number, it feels off to me in the same way that the random memory read code felt off to Jamie. It generates offsets with for i in 0..(buffer.len() / page_size) { pages.push((i * page_size + 1) as u64); } and then does 8 KiB reads (offsets are shuffled to create random reads). Some things that don't feel right about this are: The +1 makes every read unaligned. With a 4KiB page size, this makes one read touch 3 pages Different offsets can overlap the same pages, causing seemingly unintended reads from page cache Depending on the page size, reads can extend past the end of the file and cause a panic The "buffer.len() / page_size" construction seems to be intended to keep accesses in bounds, but this is independent of the access length. If we want to be lazy and not think about exact offsets, consider some huge access length like 4 GiB (the buffer size is 8 GiB). That will surely overflow. If we want to be more precise, the overflow case will be more like a 4KiB page with 8KiB access length, but the same idea applies. The very last offset is going to be SIZE - 4096 + 1. This gives us 4095 bytes we can access, but we try to access 8192 bytes. Because the benchmark only runs for 5 seconds, it may or may not actually try to read past EOF and fail, but there's a bug here regardless of whether or not it randomly fails on any given run. Sequential SSD read Just looking at the code, a lot of it doesn't feel quite right to me. For example, consider the code that's used to generate the sequential 8 KiB SSD read, which is said to have 1us latency and 8 GiB/s throughput. Like I said, I haven't worked on any problems where disk performance matters, so I don't have an intuition on whether or not numbers like this are plausible, but the code feels off to me. The code takes a 1 GiB file, flushing it, and then re-reading it repeatedly, so we'll have one uncached read followed by cached reads. It seems like the intent is to measure uncached reads here, but if the intent is to measure cached reads, the code isn't doing that either (this appears to be an issue for some of the other numbers as well, such as 3 GiB/s of fsync'd reads). One could argue that it's realistic to have an uncached read followed by cached reads, but it's not clear what someone who's using the aggregate number of 1 uncached read followed by N cached reads should do with the number when they don't have the exact same workload; N isn't stated in an obvious way, so they wouldn't even know if they have the same workload. Since I have no idea what the numbers should be here, maybe we can look up some numbers. The measurement was said to be done on a c4-standard-48-lssd. Google's docs for that instance claim that the maximum throughput for all 8 attached disks is 5000 MiB/s (from Google's table, this scales per number of attached disks and is 625 MiB/s per disk). From the very little I know of disk benchmarks, it seems like peak throughput numbers are generally done when using larger reads, so 8 GiB/s seems excessive and the feeling that something is off from the code seems to be right. And if we look at other numbers, it seems like the broader point that having a few single numbers for specific read sizes isn't representative of disk performance in general. Representativeness But the idea