Introducing SOB: A Multi-Source Structured Output Benchmark for LLMs

copy markdownLLMs are increasingly deployed to produce structured data from unstructured and semi-structured sources, parsing invoices, medical records, meeting transcripts, and converting PDFs to database rows.For deterministic output, the next step in a workflow reads a specific key and expects a specific type. A hallucinated invoice_total or an array ordered incorrectly because of inaccurate date values silently breaks downstream systems. Yet existing benchmarks either check schema compliance alone or evaluate value correctness within a single source domain.Top 5 at a glanceA side-by-side look at the top 5 models across all seven metrics. The structural metrics (JSON Pass, Path Recall, Structure Coverage, Type Safety) cluster near the ceiling for every model, while Value Accuracy and Perfect Response separate them.The problem with current structured output benchmarksMost benchmarks collapse "structured output quality" into a single number: does the response parse, and does it validate against the schema? That's necessary, not sufficient.Problem in current benchmarksWhat it missesSchema compliance as the only metricA model can emit perfectly valid JSON with wrong values and score 100%Single-source inputs (text only)Real systems extract from OCR, screenshots, meeting audio, and PDFs, not just clean textNo difficulty weightingMedium and hard schemas are scored identically, hiding which models actually handle nested structureNo separation of parse / structure / value errorsYou can't tell if a model failed at JSON, at the schema, or at the factsReasoning / chain-of-thought blended inResults measure reasoning + extraction together, not the extraction capability itselfReferences to existing benchmarks: JSONSchemaBench | StructEval | DeepJSONEval | LLMStructBenchHow SOB worksSOB evaluates structured output across three modalities using the same scoring harness. The goal is to isolate the extraction capability from every other ability a model has.Three sources, one scoring pipelineModalitySource datasetEval recordsTextHotpotQA context passages5,000ImageolmOCR-bench documents209AudioAMI Meeting Corpus conversations115Every record is paired with a JSON Schema and a ground-truth answer that was verified against the source context through human authoring with an LLM cross-check, so a missing or hallucinated value is unambiguously wrong.To isolate the structured-output capability from vision and ASR quality, image and audio records are converted to text-normalized context before scoring.

Models see the same modality-stripped context, and the differences that remain are attributable to how they handle schemas, nesting, and value grounding under different content distributions.Seven metrics, not oneSOB reports seven metrics per record so you can see exactly where a model fails:MetricWhat it measuresValue AccuracyExact leaf-value match against the verified ground truth (primary)JSON Pass RateThe response is parseable JSONType SafetyAll leaf values match the declared JSON Schema typesStructure CoverageThe response includes the required object/array structurePath RecallAll required JSON paths (keys) are presentFaithfulnessValues are grounded in the source context, not hallucinatedPerfect ResponseEvery leaf value is exactly correct for the full recordValue Accuracy is the metric that matters for production. It's the share of fields a downstream system can trust without a human review step.Scoring gatesTwo gates prevent inflated scores from schema-only wins: Hardening gate: If JSON parse fails, downstream semantic metrics are zeroed for that record. Coverage gate: Value Accuracy is only credited on fields the model actually returned, with missing paths counting as wrong. Schemas are tagged between easy, medium or hard. The final leaderboard is schema-complexity-weighted (easy = 1.0, medium = 2.0, hard = 3.0) so hard schemas contribute more to the ranking than medium ones.

Without that, the top cluster looks identical.Value AccuracyThe metric production systems care about. Note how tightly the top cluster sits compared to the overall leaderboard spread.FaithfulnessHow often values are grounded in context instead of hallucinated.JSON Pass RateAlmost every modern model clears 95%+ in the unified leaderboard. This is why a pass-rate-only benchmark can't separate them anymore.Path RecallWhether all required keys appear in the output.Structure CoverageWhether nested objects and arrays are present with the correct shape.Type SafetyWhether leaf values respect the declared JSON Schema types (no strings where numbers are expected).Perfect Response RateThe fraction of records where every single leaf value is exactly right. This is the hardest metric and collapses to roughly half even for the best models.The JSON-pass vs Value-Accuracy gapThe single most important view: most models clear 95%+ on JSON Pass, but Value Accuracy sits 15 to 30 points lower. That gap is the space where structured output benchmarks have been lying to us.The gap column is the headline. Every model on this list passes JSON parsing 97%+ of the time, but actual leaf-value extraction drops by 17 to 26 points. Qwen3.5-35B has the tightest gap (16.8) and the highest Value Accuracy on the list, while Schematron-8B passes JSON 98.7% of the time but lands the lowest Value Accuracy at 73.1% — a 25.6 point fall.ModelJSON PassValue AccuracyGapGPT-5.499.3%79.8%19.5 ppNemotron-3-Nano-30B98.7%74.7%24.0 ppSchematron-8B98.7%73.1%25.6