14× faster embeddings: how we rebuilt the ONNX path in Manticore

When we shipped Auto Embeddings — the feature that turns any text column into a vector automatically, with no separate model service to run — the most common piece of feedback was about speed. The previous path went through SentenceTransformers on top of Candle , Hugging Face's pure-Rust ML inference runtime, and it left a lot of CPU on the floor: most workloads sat in the low-double-digits of docs/sec no matter how we fed them, and concurrent calls serialised on a single model session.So we spent a few weeks rebuilding how Manticore runs ONNX models. The new ONNX Runtime backend shipped in Manticore Search 27.1.5 . ONNX (Open Neural Network Exchange) is the portable model format that most of the popular open-source embedding models — MiniLM, BGE, E5, and friends — already publish. The result is a backend that's ~14× faster on average than the previous SentenceTransformers/Candle path on the same hardware (average cheap 16 cores / 32 threads server), same model, same weights, averaged over the full threads × batch workload grid — and that advantage holds whether you run 1 client thread or 32. The old path stayed in the 5–11 docs/sec range across the entire grid; the new one lives in the 70–230 docs/sec band.This post is the engineering log: what we tried, what surprised us, what we threw away, and what the final design looks like.TL;DR~14× faster on average than the previous SentenceTransformers/Candle path, averaged across the full threads × batch workload grid (1 / 2 / 4 / 8 / 16 / 32 threads × batch sizes 1…128) on the same box (16 cores / 32 threads), same model, same weights.Released in Manticore Search 27.1.5 , the new ONNX path is now the default fast path for any HuggingFace model that ships an .onnx file.On all-MiniLM-L12-v2, the old Candle path sat at 5–11 docs/sec across every configuration we tried.

The new ONNX path lands in the 70–230 docs/sec range — the same ~14× margin holds whether you run 1 client thread or 32.Single-insert latency on our test box: ~14 ms with a single client, ~56 ms under 8-way concurrent load — both well below the 200+ ms Candle was hitting.Want maximum bulk ingest throughput? Use a high batch size (32–128) on a single client thread. The new backend parallelises inside the call, so client-side fan-out just piles coordination overhead on top — peak on our box was 233 docs/sec at 1 thread + batch=64.The two changes that mattered most: turning intra_op_spinning off, and giving up on batching documents inside the worker.No user-facing API changes. A table that already points at an ONNX-capable MODEL_NAME picks up the new path automatically. Switching an existing table to a different model isn't a one-liner — Manticore doesn't allow altering MODEL_NAME on a FLOAT_VECTOR field in place — but you don't have to recreate the whole table either: you can add a new column with the new model alongside, rebuild its embeddings, and drop the old one.Why this mattersWith auto-embeddings, the database itself runs the model on every INSERT. That means embedding speed is INSERT speed — your ingest throughput is whatever the embedding step can sustain.The old SentenceTransformers/Candle path left performance on the table. Concurrency hit lock contention, batched calls plateaued because of padding overhead, and between calls the runtime parked threads in ways that prevented the next call from picking up where the previous one left off. The headline symptom was simple: top would show the box well under full load no matter what you threw at it.

// ~10% faster activation, no quality loss .commit_from_file(&onnx_path)?; Most of these are uncontroversial, "of course you turn that on" knobs. One is not: intra_op_spinning(false). We'll come back to it — it's the single biggest win in the whole branch, and it's not really an ORT setting so much as a load-shape decision.The concurrency model — the part most readers will find newIf you give a Rust developer "make ONNX go fast" with no other constraints, they reach for one of two patterns. We tried both. They are both wrong for this workload.Pattern 1: a single shared Session behind a Mutex (a Mutex is a lock that lets only one thread touch the session at a time). Easy to reason about, easy to get right. Throughput collapses under concurrency because every caller serialises on the lock. Fine for a CLI tool, awful for a database serving many concurrent INSERTs.Pattern 2: a session pool, one Session per CPU. No more lock contention, but cold-start time multiplies, RAM use multiplies, and small inputs pay a dispatch cost just to land on a session. We had a working version of this in a development branch and it never quite delivered.The thing that unlocked the design is something most Rust ONNX wrappers get wrong: on Linux and macOS, ORT's C Run() API is thread-safe. You can share one Session across many concurrent callers without any locking. The C++ side already serialises what needs serialising; the Rust API just hides it behind borrow-checker rules that do not match what the underlying library actually allows.So we wrap the session in a small platform-aware type:#[cfg(not(target_os = "windows"))] struct SessionWrapper { inner: std::cell::UnsafeCell<ort::session::Session>, }

#[cfg(not(target_os = "windows"))] unsafe impl Sync for SessionWrapper {} #[cfg(not(target_os = "windows"))] unsafe impl Send for SessionWrapper {}

impl SessionWrapper { fn with_session<R>(&self, f: impl FnOnce(&mut Session) -> R) -> R { f(unsafe { &mut *self.inner.get() }) } } Yes, this is unsafe.

We're taking the borrow checker out of the loop because the underlying library is documented to be safe under the access pattern we're using. It's a deliberate unsafe with a one-line justification, not a foot-gun.On Windows, ORT's threading model has known issues, so we serialise Run() with a Mutex. Importantly, the lock is held for the entire closure, not just the call to run() — that's what fixed the race we saw on Windows where one thread's SessionOutputs were still being read while another thread had already started a new run(). Closure-scoped locking, not call-scoped.Adaptive parallelism — the wrong turns we tookThis is the part of the work that took the longest, because every textbook says "to make ONNX fast, batch your inputs". So our first attempts followed the textbook.We tokenized chunks of 8, 16, 32 documents at a time, padded them to max_len, and ran a single forward pass per worker thread. The throughput numbers came back lower than processing the same texts one-by-one through the same session. We ran it again. Same result. We spent a while trying to disprove it before accepting it. The reverted commit 980b24b "Revert: perf(model): batch inference in worker threads" is the moment we stopped fighting and rebuilt around what the profiler kept telling us.Two things were behind the surprise.The padding tax. A batch of mixed-length texts pads every row up to the longest row. The model then does work proportional to batch_size * max_len * hidden_dim, regardless of how much real content is in the batch. Real text inputs are highly variable in length: a typical batch of 8 random sentences might have one 60-token outlier and seven 8-token rows. The model spends most of its cycles multiplying padding tokens against attention weights. With one-doc batches, the model only does work proportional to that doc's actual token count. Per-document, "no batching" is cheaper than "batching" once the variance in input length is realistic.Spinning. ORT's intra-op thread pool defaults to spinning between dispatches — threads burn CPU in a tight loop waiting for the next chunk of work. With one big batch per session call this is invisible: the thread is always busy with real work.

With many concurrent small calls, it becomes a disaster: every worker's intra-op pool is pinned at 100% CPU between calls, and there's no CPU left for anything else. We saw exactly this pattern in top: every core at 100%, throughput lower than spinning-off. This sounds wrong until you remember the rest of the system needs CPU time too — the tokenizer, the HNSW build, the rest of searchd. Flipping with_intra_op_spinning(false) on was a one-line change that immediately raised throughput and dropped CPU usage at the same time.So the final shape is the opposite of the textbook recipe:One shared session, no pool.One document per inference call, no batching inside the worker.Many concurrent callers, scaled to CPU count.No spinning between calls — yield the CPU like a polite citizen.fn predict_pipelined(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>, _> { let bs = batch_size();

// Small input — single tokenize + infer, no thread overhead. // This is the path a 1-doc INSERT takes. if texts.len() <= bs { return Self::tokenize_and_infer(&self.session, &self.tokenizer, texts, ...); }

// Large input — split across workers, each running 1-doc-at-a-time // through the SHARED session. This deliberately mimics the // many-concurrent-callers pattern that ORT is happiest with. let num_workers = (texts.len() / bs).min(available_cpus()).max(1); let docs_per_worker = texts.len().div_ceil(num_workers);

std::thread::scope(|s| { for worker_texts in texts.chunks(docs_per_worker) { s.spawn(move || { for text in worker_texts { Self::tokenize_and_infer(&session, &tokenizer, std::slice::from_ref(text), ...)?; } Ok(()) }); } }); // ... } The two-branch design is on purpose. A 1-row INSERT comes in with texts.len() == 1, which is <= bs, so it takes the fast path with zero thread spawning, zero channel sends, zero coordination overhead.