cua/blog/gpu-passthrough-macos-vms.md at main · trycua/cua

Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with llama.cpp Published on August 11, 2026 by Francesco Bonacci and Johnny Franks If you've been following Cua from the start, you may remember that it began with a Show HN launch for Lume, our macOS virtualization stack. A macOS guest running through Apple's Virtualization.framework uses a virtual GPU backed by the host's Apple GPU. In our stock Tahoe VM, that device reported a conservative Metal capability profile. Applications use those answers to select kernels and rendering paths, which left llama.cpp running much slower GPU code. We built a small, process-scoped compatibility layer that changes selected capability answers for one guest process, allowing llama.cpp to select newer Metal kernels. This is the first result from our broader effort to connect Lume's virtualization foundation to the local computer-use environments behind Cua Driver and the infrastructure behind Cua Cloud and Fleets. We're releasing this work today as a research release under the same permissive license as Lume and Cua, so others can reproduce the results and help map which Apple Silicon chips, macOS releases, and Metal workloads benefit. On an M1 Ultra, TinyLlama 1.1B running through llama.cpp processed prompts 11.08× faster and generated tokens 16.36× faster than the same workload in the same stock VM. Prompt processing reached 98% of our bare-metal result. The source, build scripts, capability probe, and raw benchmark logs are included so you can inspect and reproduce the result. We repeated the experiment with Google's Gemma 4 12B QAT Q4_0, a 6.98 GB model released this year. The same layer improved prompt processing 7.20× and token generation 14.54×. The unlocked VM reached 99.59% of bare-metal prompt speed and 94.82% of bare-metal generation speed. We then tested Meta's official Muse Glimmer 30B Q4_K-M GGUF in a 64 GiB guest. Through llama.cpp b10359, the unlocked VM processed a 512-token prompt 7.55× faster and generated 128 tokens 8.87× faster than the stock guest. This was a text-only llama.cpp test; it did not use Ollama, a multimodal projector, or a drafter. The same capability gap has surfaced in other Virtualization.framework frontends. Tart, another macOS virtualization CLI, has an open “No GPU passthrough in macOS guest?” issue covering graphics and LLM performance inside macOS guests. The cap inside a macOS VM Apple's Virtualization.framework presents a macOS guest with a virtual graphics device. The guest submits Metal work through a purpose-built GPU driver, and Apple's host stack executes it on the physical GPU. This arrangement is paravirtualization, where the host keeps control of the hardware and the guest uses a virtualization-aware device. This differs from other virtualization stacks built on QEMU and KVM, which can use a different architecture. On x86 Linux hosts, VFIO can assign a compatible physical PCI device or hardware function to a VM through an IOMMU, giving the guest direct access to that device. This is the model usually meant by GPU passthrough. In our stock Tahoe VM, the paravirtualized device reported roughly an Apple 5-era family, 32 KB of maximum threadgroup memory, and SIMD-group matrix support as unavailable. Modern Metal software uses those answers to select kernels, so llama.cpp took a slower path even though the device could execute newer kernels. Apple documents GPU capability through GPU families and feature tables and recommends querying the device at runtime. That makes the reported capability boundary consequential: applications are doing exactly what the platform tells them to do. The solution: a process-scoped Metal capability shim We built a small Metal capability shim (a compatibility layer inserted between an application and an API) that runs inside one guest process. It intercepts selected Metal capability queries and changes the answers returned to that process. Metal applications use those answers to select kernels, so returning the tested Apple-family and threadgroup-memory values lets llama.cpp choose its newer GPU paths. For our tested profile, the shim: answers supportsFamily: through Apple family 9 (1009); and raises the reported maximum threadgroup memory from 32 KB to 64 KB. That was enough for the tested llama.cpp build to select newer SIMD-group reduction, SIMD-group matrix, and bfloat16 paths: Capability Stock guest Tested profile supportsFamily:1009 false true SIMD-group matrix off on SIMD-group reduction off on bfloat16 off on Maximum threadgroup memory 32 KB 64 KB The tested profile changes two reported values: Apple-family answers and the threadgroup-memory limit. Common, Mac, Metal, and working-set-size values keep their stock settings during the benchmark. We removed the original research hook's private feature-profile hook, clock and timing interposition, mesh substitution, ray-tracing override, argument-layout guard, and pipeline-compilation fallback. Its source is small enough to audit, and malformed or missing configuration keeps the process on its stock capability path. The workload stays on Apple's Virtualization.framework graphics path and executes on the host's Apple GPU. The capability changes are scoped to the injected guest process. Physical GPU assignment, raw PCI or VFIO passthrough, and kernel changes sit outside this mechanism. A reported family describes the paths covered by our tests; each additional Metal API requires separate validation. The shim unlocks Metal capabilities on Apple's existing virtual GPU path. VM users often encounter the broader limitation under the name “GPU passthrough.” Fresh result from the minimal artifact We tested on one Apple M1 Ultra with a 48-core GPU and macOS 26.6.1. The guest was the current public Tahoe Cua image (macOS 26.5.2, 8 vCPU, and 16 GiB) running in Lume 0.5.1. All three runs used the official llama.cpp b10167 release and the same TinyLlama 1.1B Chat Q4_K_M model. The command was: llama-bench -m tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \ -p 512 -n 128 -r 10 -t 8 -ngl -1 -o json Values below are medians of the ten samples emitted for each benchmark row: Workload Bare-metal host Stock guest Unlocked guest Guest speedup Unlocked / host Prompt processing, 512 tokens 4,871.99 tok/s 431.86 tok/s 4,786.70 tok/s 11.08× 98.25% Token generation, 128 tokens 286.71 tok/s 12.63 tok/s 206.60 tok/s 16.36× 72.06% Prompt processing nearly reached the host result. Generation reached 72.06% of host speed, leaving a measurable VM gap. The gain depends on the host GPU, guest version, application, and workload shape. The TinyLlama raw results and environment record include the exact image digest, model and binary hashes, commands, JSON output, stderr, and checksums. These release-candidate results certify the reduced shim used in this post. A current 12B model TinyLlama makes a useful controlled benchmark because it runs quickly and exposes the Metal path clearly. We also wanted a larger model that developers might choose today, so we ran Google's official Gemma 4 12B instruction-tuned QAT Q4_0 GGUF through the same llama.cpp binary. The host, VM, shim, benchmark shape, and ten-sample method stayed the same. We disabled speculative decoding and left the multimodal projector unloaded, keeping the comparison on the same Metal inference path: Workload Bare-metal host Stock guest Unlocked guest Guest speedup Unlocked / host Prompt processing, 512 tokens 517.88 tok/s 71.66 tok/s 515.76 tok/s 7.20× 99.59% Token generation, 128 tokens 52.38 tok/s 3.41 tok/s 49.67 tok/s 14.54× 94.82% The Gemma 4 evidence pins Google's model revision and SHA-256 alongside the final raw samples. We discarded and reran a preliminary stock series after detecting another host compute workload. The retained stock, unlocked, and bare-metal files come from the same uncontended window and show tight sample ranges. A 30B text model in a 64 GiB guest Muse Glimmer let us test the same capability path with a larger model. We used Meta's official 16.76 GB Q4_K-M GGUF, raised the Tahoe guest to 64 GiB, and updated llama.cpp to b10359. Prompt processing and generation ran as separate fresh processes with eight threads and full GPU offload: Workload Stock guest Unlocked guest Guest speedup Stock range Unlocked range Prompt processing, 512 tokens 25.8328 tok/s 194.971 tok/s 7.55× 25.7641-26.0987 194.565-195.331 Token generation, 128 tokens 2.37551 tok/s 21.0823 tok/s 8.87× 2.14729-2.41391 21.0729-21.0954 These values are medians of three llama-bench samples. The built-in same-process warmup ran before each row and is excluded from the samples. All four processes exited successfully. Before and after every arm, the guest reported 98% free memory, zero swap, and zero compressor use. Stock stderr reported Apple family 5 with the newer SIMD-group and bfloat paths disabled; unlocked stderr reported Apple family 9 with those paths enabled. The host was shared with another VM that showed intermittent CPU activity, so the Muse Glimmer evidence preserves that boundary. The pp512 samples were tight, and the stock tg128 median agreed within 5.6% of an earlier independent run. The public evidence includes the official model revision and SHA-256, llama.cpp and shim hashes, path-sanitized raw JSON, capability logs, exact arguments, telemetry summaries, and checksums. This result applies to the text-only GGUF through llama.cpp. It should not be read as Ollama throughput or as a result for Muse Glimmer's multimodal and speculative-decoding components. We also tested MLX-LM 0.31.3 with mlx-community/Llama-3.2-3B-Instruct-4bit on MLX 0.32.0. Performance stayed flat because MLX-LM was already fast in the stock VM: Workload Stock guest Unlocked guest Ratio Prompt processing, 512 tokens 1,656.55 tok/s 1,665.47 tok/s 1.005× Token generation, 128 tokens 172.09 tok/s 170.86 tok/s 0.993×