dlab Open Source Week: Frontier AI on Your Own Hardware — Tim Dettmers
Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,680 words · 8 segments analyzed
In one of my classes I asked the question I was afraid to ask but I just needed the answer to: “Who is afraid of not getting a job after graduating?” About eighty percent of the 150 people in the room raised their hands.
That is roughly 120 students answering, in one motion, that they do not believe there is a place for them in the future.The other story arrives by email. PhD students who cannot wait to graduate, because they want to join a frontier lab and they have concluded that research in academia is meaningless. They are counting the years until they can leave.I believe both stories are wrong, and wrong for the same reason. They assume the future of research belongs to whoever has the most GPUs. I think the opposite is true. Academia is probably about to have a renaissance, and the most exciting work of the next decade will happen in university labs — not in spite of their limited resources, but because of them.This week is our argument for that claim, and we are making it in code rather than in prose.This post has six parts: why a lab like ours now publishes ecosystems instead of papers; what is actually in this open-source week; why the pessimism I keep running into is mistaken; what to let go of, and what to hold on to; what research will look like once you have let go of it; and why the renaissance happens in academia. The unit of research is no longer the paperSomething changed in the last year, and most of us have not updated our habits to match it.With agents, research per projects have become easy and quick. Work that used to take a year of engineering and experimentation now takes weeks, sometimes days. Here is the part that took me longer to see: when every individual project becomes easy, piecemeal work stops being good research. A paper here, a paper there, each one self-contained, each one asking the reader to stitch the pieces together themselves — that is a format from a world where every piece was expensive.The difficulty did not disappear. It moved. It is no longer hard to publish a paper. It is hard to publish a coherent ecosystem.The unit of research is the ecosystem.That is what Open Source Week is for. When my students and I started, we set out to build components that build on each other rather than merely coexist, so that each piece makes the next one more useful.
My lab and I believe in using our academic freedom to bring the best AI tools to everyone for free. Something that you can do uniquely at universities. Concretely, that meant building open systems, making models cheaper to run locally, making local models stronger, building local systems that replicate frontier performance in deep and autonomous research, and creating new methods for for building domain-specific reinforcement learning environments.All of it sits at the intersection of three things: inference-serving frameworks, agent harnesses and work, and the combination of the two into autonomous research systems.
And all of it has to be easy to use, because open source that only experienced researchers can run is not open source. Accessibility has two halves — the resources you need and the expertise you need — and only one of them is fixed by hardware. A couple of GPUs, or a MacBook, can be enough. The expertise requirement is a design problem, and you solve it by abstracting away every technical detail the user does not need to think about. That is where most of our effort went, and it is most visible in the agent harness.I am not going to give away everything before the open-source week starts, so here is what I can tell you now.If you ask me what a small lab can do today, wee will show you three things: frontier autonomous research, the most efficient test-time scaling I know of, and auto-compaction that is far more efficient than what Claude Code or Codex implement.Start with the harness, because it is what makes everything else usable.You have probably heard about agent sessions that run for hours, days, or even weeks. For most people, and especially for anyone who has never worked with agents, it is a mystery how that is achieved. You point our harness at a repository — an inference framework with CUDA kernels, say — and you tell it to optimize the kernels. Then you leave. It keeps improving them through the parts where progress is slow and the work is frustrating, and it keeps going until you come back. No feedback will be provided along the way, so the agent has to figure things out on its own whenever it is unclear or unsure.That is what we did with the Mac and Metal implementations of our inference framework. One command set the agent loose on the kernels.
What came back was quantized inference of a Qwen 3.6 35B-A3B model at 450 tokens per second, with high-quality output at 1.5 bits per weight.
A half-precision model needs sixteen bits for every weight; at 1.5 bits, the same model runs in about a tenth of the memory, and it runs fast enough to feel like a local process rather than a remote service.Then there is the theme in the title of this post. What happens when the models that used to be out of reach fit on the hardware you already own?Qwen 3.8 at 27 billion parameters has been the popular local model. Our framework lets you run its larger sibling, Qwen 3.8 Flash Next at 125 billion parameters, on a single 24 GB GPU — the card in a normal desktop machine. With AMD Strix, an NVIDIA DGX Spark, or a MacBook with 128 GB of memory, you can run DeepSeek V4.1 — a 550B model. You will not have to manage context length either: compression and context handling are automatic, and inference stays fast even at long contexts.Then there is the part I am most excited about.We combined these pieces and pushed further into autonomous research, and on the way we built a new information retrieval technique with a precision I have not seen before. The system beats deep research systems from frontier labs, and it produces better autonomous research results than Sakana AI’s system or Google’s ScientistOne. It runs entirely locally, with no internet access at all.Using it is simple. Let me give you the experiment I ran.I asked the agent to find a problem worth working on in the domain of bioinformatics — because I do not know much about it — and the criteria were specific. Progress had to be fast. The evaluation had to be cheap enough to run on the hardware we already had. And it had to be a fresh problem, with active research published in the last four weeks, so that we would be working on something the field has not settled. The agent came back with three problems. We took the first, and within about two hours it had established a new lower bound on heuristic methods, developed and tested the best heuristic method in the literature, moved closer to expensive methods trained with AI models, and found issues in the data sources that everyone uses to evaluate this problem. We did not reach state of the art on the overall problem. Still: two hours of work on a machine in my lab produced four results, and one of them questions the evaluation data the whole area depends on.The system is not a demo that we trot out for blog posts. My students use it every day.
Before it lived inside the harness, it lived in a Slack bot, and it was flaky enough that the bot would go down at times. I did not have an email system that alerts me to the Slack bot going offline, but I had the next best thing: my students often wrote me “Tim, there is something with the slack bot and it does not work anymore. Can you help?” In a collaborative setting I used it after recording a meeting: it generated research questions from the recording, evaluated the ideas discussed against the literature, and sorted the promising directions from the unpromising ones. Then created a google doc and sent it to the students. I did that for two meetings. The students liked it, but it was cumbersome since it had a manual component of me copy pasting two pieces in the pipeline, so I stopped. For the next two meetings I did not use it — and then the students asked me with anticipation if we can again use the system because they found it to be so useful to make sense of their research.That is the only evaluation of a research tool I trust: people ask for it after you stop giving it to them.The last piece is the one we use the most and talk about the least. How do you keep an agent working after the conversation would have ended?Our answer is an auto-compaction technique called CliffCompaction. We have used it in the lab for months, and I, for one, want to never run an agent without it. It is considerably more powerful than the auto-compaction in Claude Code or Codex. Sessions with it run for millions of tokens, and some of mine have run past a hundred million. It also cuts overall cost by about fifty percent. One of our partners deployed it inside their company and measured a forty-five percent reduction in their total AI budget — nearly half of what they spend on AI, gone, without giving anything up. On KernelBench it reaches state of the art, beating methods far more complicated than ours, AlphaEvolve-style approaches and hierarchical memory systems among them, by a wide margin. We will published a strong version.
We already parts of the next one autocompaction technique, and it is better.It moves both sides of the cost/capability trade-off at once: sessions that run longer, and a bill that runs smaller.