GitHub - Carloscodix/qapla: A char-level transformer trained from scratch on an $8 ESP32-S3. Not inference: the chip runs the full training loop, with backprop written by hand in C.

The model is born somewhere else — a GPU, a datacenter — trained on its data, quantized, and only then loaded onto the chip so it can run it. The brain is cooked outside and served on-chip. We asked ourselves a different question: what happens when the model can't be born outside? What happens when it can't come pre-trained — because the data it needs to learn doesn't exist until the device is in place — can't carry that data with it, and has no internet to download it?

But… if we're going to get creative, picture this: a sensor bolted to a piece of farm machinery in the middle of a field, which has to learn the normal vibration of that specific machine (different from any other machine in the world) to spot when something's going wrong and schedule maintenance before it breaks (and no, you won't catch me out: I said there's no internet, but… LoRa? ;). Or picture a sensor in a plot of land that learns how that particular soil dries out — its earth, its sun, its drainage — and predicts when it'll need watering, before the plants suffer. In those cases (and others we can think of) the data didn't exist until the device was installed: nobody could have pre-trained it. The chip has to learn on the fly, alone, right where it stands.

We don't have a piece of farm machinery lying around to experiment with. So, to put the real learning ability of an ESP32 to the test, we designed the experiment around the only thing we did have: language.

How far can an eight-buck chip get, learning a language from scratch, with nobody's help? What does training on a micro actually mean? Training inside an eight-buck micro imposes rules of the game a datacenter doesn't have. And those rules dictate everything else: Once again, memory rules. An ESP32-S3 has a few MB of RAM/PSRAM, not gigabytes. That caps the size of the model: we're talking hundreds of thousands of parameters here, not millions. A model that "fits" and trains on the chip is, by necessity, small. The small model rules the task. A model this size can learn the structure of a language (how words are built, its phonotactics, some grammar), but not the deep semantics of an entire tongue. The task has to be cut to the model's size. And the corpus rules the result. With a small model you don't need gigabytes of text. You need a corpus that is compact, clean and structured. And, if you're going to publish it, you need it free of copyright headaches. Put the three constraints together — small model, narrow task, clean and compact corpus — and the question becomes concrete: which language ticks all of those at once?