GitHub - cactus-compute/cactus-hybrid: On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff. Copy-paste quickstarts for Cactus, Transformers, llama.cpp and MLX.
Pangram verdict · v3.3
We believe that this document is a mix of AI-generated, and human-written content
AI likelihood · overall
MixedArticle text · 615 words · 4 segments analyzed
A small, on-device model is fast and private, but sometimes wrong. At Cactus we post-train models to know when they are wrong: we ship probes inside the checkpoint that score every answer with a confidence between 0 and 1, returned as structured data (never parsed out of the answer text). Answer on-device when confidence is high; you can re-route to a bigger model when it's low: if confidence < 0.85: answer = ask_a_bigger_model(prompt) We start the rollout with Gemma 4 E2B Hybrid, all builds live in the Cactus Hybrid collection on Hugging Face. Gemma 4 E2B hybrid, the smallest Gemma model, matches Gemini 3.1 Flash-Lite on most benchmarks by routing only 15–35% of queries to the Gemini 3.1 Flash-Lite and running the remnant itself.
Benchmark Handoff to match Flash-Lite (FP16) At 4-bit At 3-bit
ChartQA 15–20% 25–30% 40–50%
MMBench 30–35% 40–45% 50–55%
LibriSpeech 25–30% 35–40% 55–65%
GigaSpeech 30–35% 40–45% 50–55%
MMAU 30–35% 35–40% 50–55%
MMLU-Pro 45–55% ~90% n/a
N/B: Quantisation quality is measured on Cactus Quants which performs well at uniform quantization. Developers are encouraged to benchmark for Unsloth, GGUF, and MLX quantization independently.
Cactus # pip install cactus-compute import json from cactus.bindings.cactus import cactus_complete, cactus_init from cactus.cli.download import download_bundle
lm = cactus_init(str(download_bundle("Cactus-Compute/gemma-4-E2B-it"))) result = cactus_complete( lm, [{"role": "user", "content": "What is the capital of France?"}],
json.dumps({"max_tokens": 512, "auto_handoff": False}), None, lambda *_: None, ) print(result["response"].strip()) print("confidence:", result["confidence"]) MLX # pip install mlx-lm import re from mlx_lm import load, generate
model, tokenizer = load( "Cactus-Compute/gemma-4-e2b-it-hybrid-mlx", tokenizer_config={"trust_remote_code": True}, )
messages = [{"role": "user", "content": "What is the capital of France?"}] answer = generate( model, tokenizer, prompt=tokenizer.apply_chat_template(messages, add_generation_prompt=True), max_tokens=512, ) # the checkpoint reasons before answering; keep only the final answer answer = re.split(r"<\|?channel\|?>", answer)[-1] answer = re.sub(r"^(thought|final)\b\s*", "", answer).strip() print(answer) print("confidence:", model.last_confidence) Transformers # pip install "transformers>=5.5.4,<5.6" torch (5.14+ segfaults on this checkpoint) import torch from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Cactus-Compute/gemma-4-e2b-it-hybrid"
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="auto").to(device)
messages = [{"role": "user", "content": "What is the capital of France?"}] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, return_tensors="pt", return_dict=True ).to(device) out = model.generate(**inputs, return_confidence=True, max_new_tokens=512)
print(tokenizer.decode(out.sequences[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) print("confidence:", out.confidence) Load the model with an explicit .to(device), not device_map="auto": the probe scores generations outside the module forward() path, so weights that accelerate offloads (left on the meta device) crash the confidence read. llama.cpp llama.cpp is C++, so the probe is a patch you compile into the engine (see patches/llama.cpp/). Build the patched server once: git clone https://github.com/cactus-compute/cactus-hybrid && cd cactus-hybrid ./patches/llama.cpp/install.sh && rehash Then serve and query it like any llama-server — the response carries a top-level confidence field: llama-server -hf Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF:Q4_K_M --jinja curl -s http://localhost:8080/v1/chat/completions \ -d '{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":512}' \ | jq '{answer: .choices[0].message.content, confidence}' Routing Quality (AUROC) Gemma 4 E2B Hybrid AUROC measures how well the the separates wrong answers from right ones (higher = better, 0.5 is
random, 1.0 is perfect):
Hold-out Modality Cactus Hybrid Token Entropy
MMLU text MCQ 0.770 0.697
MMLU-Pro text MCQ 0.771 0.692
ARC-Easy text MCQ 0.888 0.655
ARC-Challenge text MCQ 0.834 0.646
GSM8K (3-shot) text gen 0.782 0.731
MMBench-EN-Dev vision MCQ 0.840 0.435
ChartQA vision QA 0.779 0.615
DocVQA vision QA 0.781 0.512
MMAU audio MCQ 0.789 0.517
GigaSpeech audio 0.876 0.343
Earnings-22 audio 0.839 0.323
LibriSpeech audio 0.822 0.427
Mean
0.814 0.549
The strongest result: the probe was trained on zero audio data, yet achieves 0.79–0.88 AUROC on four audio benchmarks (two transcription, one audio MCQ, one out-of-domain transcription). This rules out surface-level explanations, the probe is reading a modality-independent correctness signal from the hidden state, not memorizing patterns from training data.
MIT-licensed. Gemma model use is subject to the Gemma terms.