Client-side semantic search for your static site

Listen to this article insteadYour browser does not support the audio elementEight years ago I added client-side search to this blog with Lunr.js. It creates an inverted index at build time, ships it as JSON, and matches strings in your browser. No server-side engine required. It has worked fine ever since, in the sense that it finds a post if you type a word that is actually in it.Earlier this year I wrote a semantic search engine in ±250 lines of Python (the kind that lets you find the “London Beer Flood” when you search for “alcoholic beverage disaster in England,” because it understands that beer is alcoholic and a flood is a disaster). That one needs a machine with sentence-transformers installed and a few hundred megabytes of PyTorch; to serve something like that in a production server requires beefy machines with expensive RAM. Not something you run in a browser tab.So this blog had keyword search that runs anywhere and understands nothing, but no semantic search that understands things but can’t run anywhere near a static site. This post is about closing that gap: semantic search that runs entirely in your browser, with no server and no API, where the entire model is essentially a 4 MB lookup table. You can try all three of the models I benchmarked, further down, running live on your own hardware (I can’t afford GPU clusters).Doing this surfaced a couple things, chief among which that the keyword search had a bug for a couple of years. TUrns out that doing proper evals is important.The obvious approach costs 23 megabytesThe models that power the Python post (sentence-transformers like all-MiniLM-L6-v2) can, in fact, run in a browser. Transformers.js will download an ONNX build of one and run it on WebAssembly. I benchmarked it, and on my laptop, the quantized model plus its runtime is 23.45 MB over the wire, takes about two seconds to load and become available, and then embeds a query in ±18 ms.Twenty-three megabytes is a bit rich to embed a search box query, for a blog that has 14 posts. That is roughly a dozen high-resolution photos worth of bytes to download so we can have fancy search.

It works, and for some applications it is completely worth it, but it is not something I want to inflict on someone who clicked through to read about bloom filters, especially not on a mobile connection.The thing is, you do not need a transformer to embed a search query. You need a vector that’s good enough, and there is a much cheaper way to get one.A static embedding model is a lookup tableThe trick is a family of models called model2vec (the specific ones are named “potion”). They are distilled from a real sentence-transformer, but the result is not a neural network. It is a table.Here is the model’s entire forward pass. Not a simplification — the whole thing, from the library’s source:ids = tokenize(text) # split into subword token ids rows = embedding[ids] # look up one vector per token vector = rows.mean(axis=0) # average them vector = vector / norm(vector) # normalise to unit length That is it. Tokenize, look up a row per token, average, normalize. There is no attention, no layers, no inference. “Running the model” is a handful of array lookups and an average. For potion-base-8M, the table is 29,528 tokens by 256 dimensions of float32 (about 30 MB) and it reaches 81% of MiniLM’s retrieval quality while being, definitionally, a dictionary lookup.Thirty megabytes is still too much, but a table is a much friendlier thing to shrink than a transformer.Building the index: chunking, and a cache that never earns its keepEmbeddings are generated at build time. On my laptop, whenever I run hugo, a Python script walks through the posts, strips out the front matter and the code blocks, splits each post into overlapping 600-character-ish chunks, embeds every chunk, and writes the vectors to a file the browser downloads when a user clicks the search input box.The chunking matters more than I expected, because of that averaging step. A static embedding is the mean of its token vectors, and if you average an entire 15,000-character post into one vector, you get something that points at “generic English prose about software” and not much else.

The rare, distinctive words (terms like pydub or mmh3) get drowned out by the hundreds of ordinary words around them. Chopping the post into smaller chunks helps keep those signals sharp. I’ll come back to this, because it turns out to be the key to why some models beat others.I also built an embedding cache, keyed on a hash of each chunk’s text, so that re-embedding only touches chunks that changed. This turned out to be a waste of time and tokens. Embedding every chunk of this blog is a few hundred lookups and an average; it takes about ten milliseconds. I built a cache to speed up an operation that is already pretty instantaneous (turns out 14 blog posts is not a lot of data). It would matter more for the MiniLM model, where embedding is a real neural network forward pass, but I’m not shipping that one. So the cache sits there, correct and pointless, and I’ve left it in as a reminder that overengineering is easy.Shrinking the table: the model’s stopword list is hiding in plain sightThe table is 30 MB because it is float32. Quantizing it to int8 makes it a quarter of the size. The catch is that you cannot just clip everything to the same scale, and understanding why was one of my favorite things I learned building this.Look at what the row magnitudes actually are. If you sort every token in potion-base-8M by the length of its vector, the shortest vectors (i.e. the ones closest to zero) are:a . , - ) the to and of in And the longest are:turkmenistan seychelles guantanamo hemingway vanuatu This is not a coincidence and it isn’t noise either. The model’s stopword list is its row magnitudes. When you average token vectors together, a word with a tiny vector barely moves the result, and a word with a big vector dominates it. The model has learned, with no stopword list and no special-casing, that a word like “the” should contribute almost nothing and a word like “guantanamo” should contribute a lot. It’s kinda beautiful, and it’s sitting right there in the geometry.

Which is why quantization needs a per row scale: each token gets its own float32 multiplier, so that the relative magnitudes survive being crushed into int8. However, turns out that it didn’t matter much anyway.I measured it; I quantized the real table with a single global scale and checked how much it actually degraded the query vectors. The answer was: almost nothing. Aggregate cosine similarity against the original stayed at 0.9998. A global scale zeroes out exactly two rows in the entire 29,528-token vocabulary (. and a) which are precisely the two tokens the model had already decided contribute pretty much nothing. The mechanism is real; it just doesn’t matter for this particular model.I kept the per-row scales anyway, because they cost 118 KB out of 4 MB and they’re correct, but more like “cheap insurance” than “load-bearing”. The whole int8 table, per-row scales and all, reproduces the original float32 model to a cosine of 0.999958. Good enough to ship a lookup table in the browser rather than a model.WordPiece in eighty lines, and its three gotchasThe browser has the token table, but it still has to turn your typed query into token ids the same way we did in Python, so we can look up the right rows. That means reimplementing the BERT WordPiece tokenizer in JavaScript. It’s about eighty lines, and it has three gotchas that could each silently poison query vector:No [CLS]/[SEP]. BERT tokenizers normally wrap your text in special marker tokens. The tokenizer.json config even has a section describing how. model2vec doesn’t use it, but calls the tokenizer with add_special_tokens=False instead. Add the markers and you’re averaging in two vectors that shouldn’t be there.Unknown tokens are deleted, not embedded. If a word isn’t in the vocabulary, model2vec drops it from the sequence entirely rather than substituting an [UNK] vector. So a query made entirely of gibberish produces an empty token list and a zero vector, and you have to handle that instead of dividing by zero. (

In practice this is nearly unreachable; with a 29,528-token vocabulary, every single character is in the vocabulary, so the only way to trigger it is a word longer than 100 characters.)"strip_accents": null means accents are stripped. This one is a little nasty. The config says strip_accents is null, which reads like “off.” But in HuggingFace’s tokenizer library, a null value inherits from the lowercase setting, which is on. So café becomes cafe. If we’d copy the config literally, every accented query drifts.Some frustrations and a bunch of Claude generated test strings ( pydub, café naïve, C++ vs C#, 日本語のみ) later, I’m reasonably confident the JS tokenizer matches the BERT WordPiece tokenizer.Search is a few hundred dot productsWith the query embedded, search is almost anticlimactic. There are a few hundred chunk vectors (one per chunk of every post). Computing the cosine similarity against all of them is a few hundred dot products of 128 numbers each, tens of thousands of multiply-adds, which a browser on a reasonably modern machine does in a fraction of a millisecond. Then we group the chunks by post, take each post’s best-scoring chunk, and sort. On my machine (M1 MacBook Air) the entire query (tokenize, embed, score every chunk, rank) takes about 0.4 milliseconds.In a production setting, this is where you reach for approximate-nearest-neighbour indexes (HNSW and friends), and if you have a bazillion documents you should. With a few hundred, an ANN index would be slower than the brute-force loop and much larger on disk. It’s worth saying out loud because “vector search” in this case isn’t a complex database system,but just a for loop.There’s one wrinkle worth knowing if you build one of these. The document vectors are stored as int8. Cosine similarity is invariant to a positive scale, and the document matrix uses one global scale, so the browser can dot a float32 query straight against the raw int8 bytes and get the right ranking without ever un-quantizing them. But int8 × int8 in JavaScript overflows silently; a dot product whose true value is three million comes back as -64, no error, no warning.