Etched Sohu vs NVIDIA: Transformer ASIC vs GPU (2026)

Etched Sohu is a transformer-only ASIC, and Etched AI claims one 8-chip Sohu server delivers 500,000 tokens per second on Llama 70B, roughly 62,500 tokens/sec per chip. For comparison, a single H100 SXM5 achieves around 700 tokens/sec at batch 1 with vLLM. That per-chip advantage is real in the sense that it reflects Sohu's architecture: the chip hard-codes transformer attention directly into silicon as fixed-function logic rather than as software running on a programmable compute unit. The implied tradeoff is the entire story here. Sohu is a bet that transformer attention is the dominant AI architecture for the next several years, and that the workload is stable enough to justify giving up all programmability.For teams evaluating inference hardware right now, the practical question is not whether Sohu is fast. It is whether the architectural constraints, supply risk, and toolchain migration cost are acceptable for your specific workload. This post covers the architecture in detail, compares Sohu against H100, B200, and the Groq 3 LPU, and gives a framework for deciding when the ASIC bet pays off. For a currently available ASIC comparison, see the SambaNova SN40L vs H200 and B200 guide, which covers the RDU architecture and live cost-per-token math.Update: Etched Exited Stealth on June 30, 2026Etched formally came out of stealth on June 30, 2026, and the announcement moved the story from "interesting claim" to "funded, contracted roadmap." The concrete facts: working A0 silicon demonstrated, a rack-scale 8-chip Sohu inference system unveiled, roughly $800M raised across four rounds (including a reported $500M round at a $5B valuation), over $1B in signed customer contracts, and first racks scheduled to ship in summer 2026. What has not changed: no independent third-party benchmarks, no public pricing, and no self-serve way to rent a Sohu today. Everything below, including the cost-per-token framework, still applies; the difference is that the availability question now has a date attached instead of a shrug.What Is Etched AI and the Sohu ChipEtched AI is a chip startup founded in 2022, having raised roughly $800 million across four rounds, including a reported $500M round at a $5B valuation. Their first product is the Sohu chip, a transformer-only ASIC designed for autoregressive language model inference. The name and funding are real. Since the June 30, 2026 stealth exit, Etched has shown working A0 silicon and booked over $1B in signed contracts, with first racks slated for summer 2026. It is still not something you can buy or rent today, and no independent benchmarks exist yet.The core architectural claim is that by implementing transformer attention as fixed-function silicon rather than as programmable matrix multiply instructions, Sohu can achieve throughput figures that no GPU can match for the same workload class. NVIDIA GPUs are programmable compute units that run CUDA kernels written in software. Groq's LPU is a dataflow processor with a custom compiler. Sohu takes a different position: there is no programmability layer at all. The chip does one thing, and it does it by being physically built for that one thing.This is different from other custom inference chips in an important way. The Groq 3 LPU has a compiler that can, in principle, be extended. AMD GPUs support ROCm. Tenstorrent ships TT-Metal as open-source software. Sohu's architecture does not have a software abstraction layer in the same sense. If transformer attention changes in the next generation of models, the chip cannot adapt. For hyperscaler-built ASICs (Trainium 3, Maia 200, Meta MTIA), see the hyperscaler custom AI chip roundup.Sohu Architecture: The Transformer-Only BetFixed-Function Transformer UnitsTransformer attention requires three core operations at each layer: computing query, key, and value projections; running multi-head attention across the KV cache; and passing results through a feed-forward network. On a GPU, each of these is a CUDA kernel that can be swapped out for a different implementation. PagedAttention, FlashAttention-2, and FlashAttention-3 are all software optimizations that improve how attention is computed on general-purpose hardware.On Sohu, these operations are hardwired as static circuits. The chip does not have a general matrix multiply unit that runs attention software. It has physical circuits that implement the attention computation directly. This removes all of the overhead from kernel launch latency, memory allocation, and scheduler decisions. It also means the chip cannot run any computation that does not map to transformer attention. There is no way to compile a convolution, an SSM scan, or a diffusion U-Net step onto Sohu because there are no programmable units to target.On-Chip Memory DesignSohu's throughput advantage comes primarily from memory bandwidth. The bottleneck for autoregressive decode on GPUs is KV cache reads: each new token requires reading the full KV cache from HBM. The H100 SXM5 has 80 GB of HBM3 with 3.35 TB/s of bandwidth. Every token generation step is bounded by how fast the model can read those KV cache values.According to Etched's published materials and industry reporting, Sohu uses 144GB of HBM3E per chip with approximately 1.8x the memory bandwidth of an H100 SXM5. Our HBM3e vs HBM4 vs HBM4e inference guide covers what that same HBM3E generation delivers on GPUs, so you can see how much of Sohu's bandwidth edge comes from the memory type versus the fixed-function architecture. That gives Sohu more memory capacity than an H100 (80GB) at roughly 1.8x the H100's 3.35 TB/s bandwidth, using the same HBM architecture that GPUs use rather than replacing it with on-chip SRAM. The Groq 3 LPU takes a fundamentally different route: 500 MB of on-chip SRAM per chip achieves 150 TB/s bandwidth but with very limited context window capacity. Sohu's throughput advantage over GPUs comes from architectural specialization of transformer attention patterns built on top of standard HBM3E, not from a SRAM-based design like Groq. Multi-chip configurations are still required for large model weights since even 144GB fills quickly with 70B+ parameter models at full precision.What Sohu Cannot DoThis is the section that matters most for most teams.Vision and multimodal models: any model with a visual encoder (LLaVA, Qwen-VL, LLama 3.2 Vision) cannot run on Sohu because the encoder uses convolutional or attention operations outside the pure transformer patternDiffusion models: image generation (Stable Diffusion, Flux) and video generation (Wan 2.1, CogVideoX) require U-Net convolutions that are not transformer attentionMoE with dynamic expert routing: DeepSeek V4, Mixtral, and Qwen3-235B-A22B use sparse expert selection at each token, which requires irregular memory access patterns that fixed-function transformer circuits cannot accommodateSSM and Mamba architectures: state space models replace attention with a scan operation, which is computationally different from transformer attentionTraining and fine-tuning: Sohu has no backward pass implementation; it is inference-onlyFuture architectures: any model architecture that does not conform to dense transformer attention requires new hardwareDeepSeek V4 and Qwen3-235B-A22B are two of the most widely deployed open-weight models as of April 2026. Both are MoE architectures. Both are incompatible with Sohu. This is not a niche edge case. It means a significant fraction of current production inference workloads cannot run on Sohu at all.Etched's Claimed Numbers vs Realistic NVIDIA BaselineEtched's 500,000 tokens/sec figure for Llama 70B is for an 8-chip server, from their own published materials, and has not been independently verified. The conditions are important: this appears to be measured at or near batch size 1, where the fixed-function attention circuits can operate at peak efficiency without the batching overhead that GPUs exploit to amortize their lower per-token bandwidth.At higher batch sizes, the picture changes. GPU throughput scales well with batching because the compute units stay busy across multiple requests. Fixed-function attention circuits do not benefit from batching in the same way; their advantage is decode throughput per request, not aggregate throughput across many requests.ModelSohu per chip (claimed by Etched)H100 SXM5 (vLLM)B200 SXM6 (vLLM)NotesLlama 70B at batch 1~62,500 tok/s~700 tok/s~1,200 tok/sSohu figure derived from 8-chip server claim of 500k tok/sLlama 70B at batch 32TBD~5,000 tok/s~9,000 tok/sLlama 70B at batch 256TBD~45,000 tok/s~80,000 tok/sSohu figures are per chip, derived from Etched's published 8-chip server claim of 500,000 tok/s on Llama 70B; not independently verified. NVIDIA figures reflect vLLM with FlashAttention-2 on a single chip. Run your own benchmarks before making procurement decisions.The batch size gap is critical for any team running a serving API with concurrent requests. At batch 256, a single H100 can generate 45,000 tokens per second. Etched's 500k tok/s figure is for an 8-chip server at batch 1, which normalizes to roughly 62,500 tok/s per chip. A fair comparison requires matching conditions. Etched has not published batch 32 or batch 256 figures.Sohu vs B200 and B300: Throughput and Cost for Pure Transformer InferenceFor teams with pure transformer workloads, the cost-per-million-token comparison matters more than raw throughput. The formula is: Cost per 1M tokens = ($/hr) / (tokens/sec × 3,600) × 1,000,000ChipOn-demand ($/hr)Spot ($/hr)Llama 70B tok/s (batch 32)Cost/1M tokens (on-demand)H100 SXM5 (Spheron)$3.38$1.46~5,000~$0.188B200 SXM6 (Spheron)$7.50$2.74~9,000~$0.231B300 SXM6 (Spheron)$10.21$5.81~16,000 (est.)~$0.177 (est.)Sohu (Etched, est.)Not available~62,500/chip at batch 1, unverified (8-chip server: ~500k)Not calculablePricing fluctuates based on GPU availability. The prices above are based on 02 Aug 2026 and may have changed. Check current GPU pricing → for live rates.Sohu's