If You Think You Can Do Real-World Text-to-SQL – Communications of the ACM

Any text-to-SQL benchmark should address the difficulties of real-world data stores.  Posted Jul 20 2026

Credit: Shutterstock AI

Data warehouses are often subject to a preponderance of ad-hoc queries. They came into existence in the 1990s to support historical data stores of customer-facing data, especially sales data. At that time, it was widely reported that they paid for themselves based on better stock rotation in less than six months. For example, if pet rocks are “out” and Barbie dolls are “in,” then discount the pet rocks and move them up front; tie up the manufacturer of Barbie dolls with a big order so competitors cannot get merchandise.

Although there are repeated queries (for example, “produce the weekly sales report”), much of the traffic is ad hoc in nature, including the above example. Historically, data warehouses were fronted by business intelligence tools such as Business Objects, Power BI, and Tableau. This allowed many queries to be coded by business users using a simple graphical interface. Obviously, one would like to query data warehouses and other large, structured data stores in natural language to further empower non-programmers. It would have the added benefit of not requiring familiarity with a bespoke user interface. With the advent of large language models (LLMs) and agentic AI, this becomes a real possibility, and there are public benchmarks with leaderboards that show promising results.

On Spider 1.0,1 the leaderboard had accuracy in the 80+% range, while on Bird-SQL,2 accuracy was above 90%. However, both benchmarks have the following issues:

The data is in “the pile,” so LLMs can train on it. It is widely known that an LLM can only find data it has seen before. Almost all data warehouses we know about are behind serious enterprise access controls and are not publicly available.

Real data warehouses exhibit “schema rot.” Inevitably, the warehouse begins life with a well-structured schema.

However, the schema of a typical warehouse is updated as business conditions change, the company buys or sells business units, and regulations change. Often schema changes occur multiple times a year. It would be wonderful if enterprises spent the time to construct a well-structured replacement schema. Invariably this is not the case, and schemas “rot” over time. The result after a few such changes is non-intuitive table and column names. For example, there may be six different columns in various tables labelled “salary.” Such columns may have overlapping non-documented semantics; one salary column may be for “net after taxes and commissions,” while another one may be for “gross salary.” Materialized views may be used to speed up popular queries. These characteristics give an LLM multiple ways to solve a query, and to generate headaches.

Real data warehouses often have idiosyncratic data. For example, in the MIT administration data warehouse containing information on students, faculty, building, classes, etc., there are mentions of “J-term” (a one-month term in January between fall and spring semesters) and building numbers (MIT buildings are numbered, not named—Stata Center is officially Building 32). Such bespoke information is difficult for an LLM.

Real data warehouse queries are more complex than the student-generated ones in Spider and Bird-SQL. They rarely have just one join and often two or three.

Spider 2.03 has addressed the last issue, but not the first three. The leaderboard of agentic-AI solutions for this benchmark is in the 90+% range.

In our opinion, real data warehouses have all of the issues mentioned above. Hence, any text-to-SQL benchmark should address the difficulties of real-world data stores. Those that do not are of academic interest only and do not reflect reality.

Beaver Benchmark

For the last two years, we have been trying LLM-based text-to-SQL on real workloads from real data warehouses. We started with the above-mentioned MIT data warehouse, an Oracle data warehouse of 1,400+ tables that is queried by many units at MIT (including CSAIL). Typically, business users pose a question to IT personnel who translate the request to SQL, often with clarifying dialog between the two people.

We got real queries from the Oracle log and generated the corresponding natural language with the assistance of real users. This generated a collection of (natural language, gold SQL) pairs. Later we applied the same methodology to three other data warehouses. With permission, we anonymized and extended the queries and the data to produce the Beaver benchmark.4 One should clearly note the four issues noted above are all present in this benchmark. As such, we do not expect to get the same high accuracy as Spider/Bird. More details on the benchmark are published online.5

On this benchmark, a pure LLM generated an accuracy score of zero. Adding RAG, prompt engineering, and agentic AI raised accuracy to the 10+% range. See Jindal6 for results on this sort of model. To get a higher score, we gave the LLM the specific tables in the from clause of the gold SQL and the actual join clauses when there were multiple tables. That generated accuracy in the 30+% range. A leaderboard for results on Beaver is available online.7 Note that this is 50+% lower in accuracy that the public benchmarks and is the difference between “the technology has promise” and “the technology does not work.”

We would welcome systems and ideas on how to make LLM/agentic AI technology work much better on real-world business applications, and we have made Beaver publicly available to assist in this process. So have at it! Our own ideas on how to do better are contained in the Rubicon system,8 so take a look.

A simple conclusion can be drawn from this discussion. Good results using traditional LLM/agentic AI technology are likely only on data warehouses with clean schemas, simple queries, and little idiosyncratic data. Otherwise, try Rubicon or your good idea.

References

1. https://yale-lily.github.io/spider

2. https://bird-bench.github.io/

3. https://spider2-sql.github.io

4. https://beaverbench.github.io/

5. https://beaverbench.github.io/#overview

6. https://arxiv.org/pdf/2602.08320

7. https://beaverbench.github.io/#leaderboard

8.

https://arxiv.org/abs/2604.21413

Michael Stonebraker is a professor of Computer Science at the Massachusetts Institute of Technology, where he specializes in database management. For his contributions, he was awarded the ACM A.M. Turing Award in 2014.

Peter Baile Chen is a Ph.D. student at the the Computer Science and Artificial Intelligence Laboratory of the Massachusetts Institute of Technology, where he works with Michael Stonebraker at the intersection of AI and data systems.

Submit an Article to CACM CACM welcomes unsolicited submissions on topics of relevance and value to the computing community.

You Just Read If You Think You Can Do Real-World Text-to-SQL © 2026 Copyright held by the owner/author(s).