Building a Korean ambiguity solver fast enough to skip the GPU: 7,300 words/sec
Pangram verdict · v3.3
We believe that this document is primarily human-written, with a small amount of AI-assisted content detected
AI likelihood · overall
HumanArticle text · 1,735 words · 6 segments analyzed
TL;DR. I needed to resolve Korean lemma ambiguity over entire books, fast, and assumed I'd have to buy a GPU server for it. Turns out a 14M-param KoELECTRA-small, quantized to int8 and run through a hand-rolled, pure-Rust inference crate (zero new deps), does ~7,300 disambiguations/second on one 16-core CPU. So I never bought the GPU. For people who don't know what Kimchi Reader is: it's an immersion tool for learning Korean, and the core problem I'm solving is lemmatization, finding the dictionary base form of a word as it appears in real text. Korean makes this genuinely hard. It's agglutinative and heavily conjugated, so a single surface form can be the result of many different stems plus stacked particles and endings. My core lemmatizer is written in Rust with a rule-based strategy: it explores every valid way to decompose a word and reports back every possible lemma. It's reliable, and it's fast. Easily north of 100k words/s multithreaded. The one downside: sometimes more than one decomposition is valid, and I end up with ambiguity.
In 음악을 들어요, the 들어요 could reduce to 듣다 (to listen) or 들다 (to hold), two candidate lemmas for one surface form. A human picks the right one instantly from context; a rule engine cannot. From the user's perspective this is fine. My users are smart enough to pick the right lemma from context, and I've already saved them the work by narrowing it down to a couple of options. The popup shows both; they move on. The problem is the stats. They suffer from ambiguity at every level, and I have to come up with some different options on how to deal with ambiguity when it happens. This is the main potential gain this model would get us, and that would be crazy good.
Before any modeling, two decisions framed the whole search space:
It has to be absurdly fast. We can't just solve it on-the-fly. Comprehension stats mean resolving every word in an entire book/movie/novel, ahead of time. A model that's accurate but slow is useless to me here. The model only ever suggests, on top of the deterministic rule engine. I never wanted to replace the rule-based lemmatizer, only to add a layer that picks among the candidates it already produced. This is the design decision that pays off later: the model is handed a closed set of real candidates and forced to choose one. It can't hallucinate a lemma that doesn't exist, and since it's only a suggestion, sub-100% accuracy is fine by design.
In 2023 a friend pointed me at course.fast.ai, Jeremy Howard's course. At the time I just wanted to learn and have fun. That's where I got into ML at all, and ran my first finetune. Excellent course, by the way. I'd still recommend it to anyone in 2026.
2023, me taking the fastai course and classifying gigachads. 41.7% confident, apparently.
By 2024 I'd started dreaming, on and off, about a model that could solve ambiguity for my lemmatizer. This is the earliest screenshot I can find of the idea.
Me, 2024, long before any of this worked.
In 2025 I finally started seriously messing around to see what was possible. Real open models had landed; Gemma 3 was out. I don't fully recall how I got there (probably brainstorming with a chatbot), but I started framing disambiguation as a seq2seq task. It felt elegant: seq2seq is just translation. One sentence in language A, the same sentence out in language B. Except here the "languages" were Korean → lemmatized Korean. Then I'd reconcile the output against my lemmatizer wherever they matched.
Disambiguation framed as a seq2seq translation task: Korean sentence in, lemmatized Korean out.
KOREAN 음악을 들어요
LEMMATIZED 음악 듣다
seq2seq Gemma 3 1B
Same shape as translation: Korean in, lemmatized Korean out.
I knew an LLM would be slow for this, I wasn't deluding myself. But I just wanted a starting point, LLMs were the easiest one to reach for, and I wanted to see how far I could push it. The plan was a classic distillation pipeline:
Hand-build a small golden dataset. Finetune a big teacher (Gemma 3 27B) on it. Use that teacher to generate ~8M synthetic seq2seq sentences. Finetune the small Gemma 3 1B on the synthetic data.
I rented GPUs on vast.ai (awesome service for quick experiments, I love it), built the pipeline, and tried a few model families, Qwen too. Gemma 3 gave me the least headache and the best results at the time. I don't have the exact numbers saved, but I was missing roughly one to two orders of magnitude on both accuracy and speed versus what I needed.
Even with batched offline inference on vLLM on a 4090, it solved about 1,500 sentences/s.
RTX 4090, Gemma 3 1B, batch-512 offline inference. Honestly? Not bad at all. But this is a small feature of my product, and it implied a ~$500/month GPU server, for which a real production workload would never hit the ideal batched-offline throughput anyway. And the accuracy wasn't there yet either. So I decided to conclude that experiment there. I did ship something out of it, though: a quantized ONNX build of the Gemma 1B running on CPU, behind an experimental flag, at a sluggish ~1 to 2 seconds per sentence. That was the first version my users ever touched.
Around then I started learning about embeddings, and how you can find similar things via context (i.e. attention). I thought: why not reframe it entirely? Precompute a vector for every definition in the dictionary, then at runtime parse a sentence once, embed it, and pick the candidate whose definition vector is closest.
Attempt 2: the rule lemmatizer narrows the whole dictionary to 2 candidates; encode the sentence once and cosine-match the target's vector against only those 2 precomputed definition vectors.
음악을 들어요
1 forward pass
context vector
cosine
ENTIRE DICT · PRECOMPUTED DEFS
rule lemmatizer keeps 2
듣다
0.82
들다
0.31
The rule lemmatizer narrows the whole dictionary to 2 candidates. Encode the sentence once, then cosine-match the target's vector against only those 2 precomputed definition vectors. A closed set, so there's always exactly one winner.
I love this approach because I only ever compared against the two candidates my rule lemmatizer had already given me. A closed set, always one winner, no way to invent a lemma, unlike attempt #1. And so I built it but it got worse accuracy and worse speed than attempt #1, so I didn't push it far.
I gotta admit this one I only tested quickly and got immediately discouraged by the inference speed and the difficulty to finetune those model (skill issue mostly tbh).
Trying random off-the-shelf embedding models for a baseline. About 75% of ambiguities solved. It wasn't a total loss, though. What I learned there is exactly what later shipped (2026) as semantic search for the dictionary, hanja and grammar pages. No reranking, just raw cosine distance, lol.
Note: at the time I had this idea, Gemma 3 270M wasn't out yet (it dropped shortly after, while I was mid-build). After the first two attempts I still liked #1 the most. I just wanted it faster and runnable on cheaper GPUs. So, clueless me: why can't I just train my own model from scratch? Let's try and see. I skipped pretraining entirely and threw the 8M synthetic seq2seq samples from attempt #1 straight at it. And while I was at it, a character-level tokenizer felt more intuitive than BPE for this. I had a lot of fun here because I decided to be a serious boy and do both training and inference in Rust, with burn-rs, which looked very cool. I reimplemented the Gemma 3 architecture in burn-rs (gist) with help from some LLMs (reminder: no agentic coding existed yet, only web chat UIs, and that was pain).
The training dashboard. Loss dropping over a run. And it actually worked quite well! Against my ~200-sample handwritten test set, after many attempts, I got a 15M-param model with the Gemma 3 architecture to beat Gemma 3 1B on this task. I also realized I could do speculative decoding. It was now trivial to make a second, smaller draft model. Lots of ideas. But I was tired, I needed a break, and I needed to ship actual product features instead of toying around forever. So I shipped the 15M model for that tiny feature, finally retiring the sluggish Gemma 1B ONNX build I'd been running on CPU since attempt #1.
It took ~100ms to parse one or two sentences. Fine for a flag, nowhere near the "whole book" bar.
Early 2026 I started using Claude, my first AI subscription after the GitHub Copilot autocomplete era, slightly before Opus 4.6 shipped. Long story short: fully converted. I started vibe-coding more and more. The economics shifted; code got cheaper, so you can experiment a lot, much faster. I was still busy building features, but around early April I got distracted. Instead of doing what I should have been doing (marketing), I told myself I'd give the ambiguity problem one more try, 1 to 2 weeks max (spoiler: it was two months), just to test the water. Now that I had Claude, maybe things would go differently? And yeah, damn, it went differently. "Jarvis, organize all the documents and write a proper problem-statement .md with every edge case in full"
Your browser does not support the video tag. Actually it was our boy Claude. That was step one. Having taken a few runs at this problem already, I knew what I wanted more precisely. So I dumped everything and had Claude write a complete problem statement: every sub-problem, like detecting names in text (another hard one), and handling subwords like 금-도끼 ("golden axe"), which my lemmatizer splits into two separate vocabulary words. Then I asked what approaches it would consider. At some point it suggested this, for any BERT-like model:
BERT input layout: sentence with the target word marked, followed by the candidate lemmas, one of which the model must pick.
[CLS] 음악을 [TGT] 들어요 [/TGT] [SEP] 듣다 [CAND] 들다 [CAND]
sentence, target marked cand0 cand1
My first reaction was huh, interesting, but aren't BERT models ancient? But testing was cheap, so I had Claude build a minimal version to check whether it was viable on speed and accuracy. And it was the best of every world I'd tried.