GitHub - drumih/turbo-fieldfare: Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

TurboFieldfare

Gemma 4 26B-A4B inference in about 2 GB of RAM A custom Swift + Metal runtime for any Apple Silicon Mac, even the 8 GB ones.

Quick start · Local server · Benchmarks · Contribute results · How it works · Experiments · References

Memory got expensive. So I gave a 26-billion-parameter model a ~2 GB budget. TurboFieldfare runs the instruction-tuned Gemma 4 26B-A4B without loading the entire 14.3 GB model into memory. It keeps the shared 1.35 GB core and FP16 KV cache in memory, then streams only the experts needed for each token from SSD. This is what lets the model run on Macs with 8 GB of RAM. The runtime, streaming installer, CLI, and native Mac app are written in Swift and Metal. TurboFieldfare is model-specific rather than a wrapper around MLX or llama.cpp. The curated experiment record summarizes 103 measured results across kernels, caching, I/O, prefill, and decode. Try it git clone https://github.com/drumih/turbo-fieldfare.git cd turbo-fieldfare swift build -c release .build/release/TurboFieldfareMac On the first run, Swift Package Manager downloads and builds the Swift packages required by the tokenizer. The complete release build includes the foreground Mac app and its sibling decode-service executable. When the app opens, choose Download and let TurboFieldfare fetch and repack the pinned model (about 15 GB). Once it is ready, choose Load Model, type your prompt, and press Generate. At a glance

Metric Value

Model Gemma 4 26B-A4B IT, 26B total parameters, about 3.88B active per token

Weights MLX affine 4-bit, group 64; 8-bit router; 4-bit shared and routed experts

Memory ~2 GB of weights and 4K KV cache

Storage About 14.3 GB for the installed text-only model

Hardware Apple Silicon Mac; 8 GB of RAM

Platform macOS 26, Metal 4, Swift 6.2

M2 measured decode 5.1-6.3 tok/s on an 8 GB M2 MacBook Air

M5 measured decode 31-35 tok/s on a 24 GB M5 Pro

The measured result is a reference point, not a performance ceiling. Prompt length, generated length, page-cache state, and hardware all affect throughput. To help measure another Apple Silicon Mac, follow the community benchmark guide. Using TurboFieldfare TurboFieldfare provides a native Mac app, a command-line interface, and an experimental loopback OpenAI-compatible server. They use the same .gturbo model directory, but only one model-owning product should run at a time. The Swift package exposes six products:

Product Purpose

TurboFieldfare Swift library containing the runtime and Metal kernels

TurboFieldfareMac Native Mac app for installation and generation

TurboFieldfareDecodeService One-shot local model and Metal owner used by the Mac app

TurboFieldfareCLI Command-line instruction chat and raw completion

TurboFieldfareServer Loopback OpenAI-compatible Chat Completions server

TurboFieldfareRepack Streaming model installer and install verifier

Requirements

An Apple Silicon Mac; the validated target is an 8 GB M2 MacBook Air macOS 26 with Metal 4 Xcode 26 and Swift 6.2 or newer Enough free storage for the ~14.3 GB model installation An internet connection for the first model install

The package is arm64-only. Older macOS and Metal versions are not supported. Prompting the model The Mac app treats what you type as an instruction and handles Gemma's chat formatting automatically. Just describe the task and include any context the model needs. Generation defaults to temperature 0.2, Top-K 64, and Top-P 0.95. Set temperature to 0 for deterministic greedy output. The model can still repeat itself or give incorrect answers, so check important results. TurboFieldfare is text-only. The app and CLI support user and model messages plus optional system guidance; they do not expose or execute tools. The loopback server accepts function-tool declarations and returns model-produced tool calls for the client to authorize and execute. Images, audio, and video are not supported. Mac app Clone the repository, then run the app from its root: swift build -c release .build/release/TurboFieldfareMac Build the complete package so the app and its sibling decode service are both available. When launched from this checkout, the app stores the model in scratch/gemma4.gturbo. Install the model On first launch, the app checks the available storage and shows the download and installed sizes. Choose Download to begin. The installer never materializes the full source checkpoint. It streams the required byte ranges from the pinned Hugging Face revision and repacks them directly into the .gturbo layout as they arrive. This avoids a second full checkpoint on disk and keeps scratch memory bounded. The first installation transfers about 15 GB through bounded Hugging Face range requests. Network speed and Hugging Face response times vary, so it can take a while. The completed .gturbo installation occupies about 14.3 GB and is accepted only after its manifest and file hashes have been validated. Installation does not load the model into memory. Load and generate After installation:

Choose Load Model. Enter a prompt in the composer. Choose Generate, or press Command+Return. Use the stop button or Escape to end generation early.

The status bar shows generation progress, decode speed, and memory use. Use the right pane to configure sampling, context length, expert-cache slots, and runtime options. See Runtime controls for details and defaults. Command-line interface The CLI uses an existing .gturbo installation. If you installed the model through the Mac app, it is already available at scratch/gemma4.gturbo. Otherwise, install it from the command line: swift run -c release TurboFieldfareRepack \ --output scratch/gemma4.gturbo \ --overwrite Continue a cancelled or interrupted download: swift run -c release TurboFieldfareRepack \ --output scratch/gemma4.gturbo \ --overwrite \ --resume Remove saved download state: swift run -c release TurboFieldfareRepack \ --discard-partial \ --output scratch/gemma4.gturbo The runtime accepts only a completed .gturbo directory with a final manifest.json. Verify an existing installation without loading the model: swift run -c release TurboFieldfareRepack \ --verify-install \ --input-gturbo scratch/gemma4.gturbo Instruction chat Put chat messages in a JSON array and pass it with --messages-file: [ {"role": "user", "content": "Explain why chunked prefill reduces time to first token while keeping memory bounded."} ] swift run -c release TurboFieldfareCLI \ --model scratch/gemma4.gturbo \ --messages-file messages.json This formats messages in the same way as the Mac app. The CLI response limit is set with --max-new, which defaults to 1,024 tokens. The Mac app can generate until the selected context window is full. Raw completion --prompt is available for raw completion and reproducible comparisons. It passes the text directly to the model without chat formatting. Use --messages-file for instruction-response conversations. swift run -c release TurboFieldfareCLI \ --model scratch/gemma4.gturbo \ --prompt "The capital of France is" \ --max-new 64 \ --temperature 0 This example deliberately requests a short greedy completion. Common generation options include --max-context, --temperature, --top-k, --top-p, --repetition-penalty, --seed, and repeatable --stop strings. The public CLI uses production runtime defaults. Run the following command for the complete option list: swift run -c release TurboFieldfareCLI --help Generated text goes to standard output. Timing statistics go to standard error; add --quiet to suppress that footer in scripts. Local OpenAI-compatible server Build the server and point it at an installed model: swift build -c release --product TurboFieldfareServer .build/release/TurboFieldfareServer \ --model scratch/gemma4.gturbo It listens on http://127.0.0.1:8080/v1 and supports Chat Completions, streaming, function tools, and single-prefix prompt reuse. The client must authorize and run every tool call. Keep the server on loopback; it has no remote authentication or TLS. See Local server for a test request, Python and OpenCode setup, prompt reuse, tool handling, and the supported API subset. Test and contribute Run the public test suite serially: Scripts/test.sh Before starting a model run, close memory-heavy apps and check memory_pressure -Q. If it reports little free memory, postpone the run. Run only one TurboFieldfare app, decode service, CLI, server, test, or other local-model process at a time. To contribute a comparable performance result, follow the community benchmark guide. How the inference engine works At each transformer layer, Metal computes attention and the router from resident weights. The CPU uses the router's top-8 expert IDs to plan against the layer's 16-slot LFU cache, then fills misses with bounded parallel pread calls into Metal-visible buffers. Metal computes the resident shared-expert branch while those reads run, then combines the shared and routed outputs. Prompt prefill uses chunks of up to 128 tokens so one fetched expert can serve multiple rows. Generation repeats the routed layer loop one token at a time. The installer applies the same bounded-memory rule: it repacks remote ranges directly into .gturbo without staging a full shard or tensor. For a visual introduction to the model architecture, see Maarten Grootendorst's A Visual Guide to Gemma 4. System design explains the .gturbo layout, memory ownership, prefill, router handoff, cb1/io/cb2 phases, Metal kernels, and correctness invariants. Status and scope TurboFieldfare currently includes: