By Hansraj H. Pabbati
Principal AI Engineer, Palo Alto Networks | Formerly Oracle Cloud Infrastructure & Amazon Web Services

Key Takeaways

• A modern model can look highly capable on conventional text-to-SQL benchmarks and still struggle on enterprise-grade database tasks. Spider 2.0 reports GPT-4o at 10.1% on its real-world enterprise workflow benchmark versus 86.6% on Spider 1.0
• This gap is not a model capability issue — it stems from missing business context. Models simply never learn what your organization means by terms such as “active customer” or “revenue.”
• Switching to a larger, more powerful model will not bridge this gap. It raises the ceiling of what is achievable, but it does not teach the model your internal business definitions.
• A well-constructed semantic layer — built on clear business definitions, synonym mapping, and validated query patterns — is what genuinely closes the gap, not the size of the model.
• When assessing a natural-language-to-data vendor, focus your questions on schema ingestion, how they handle stale data, whether the glossary is editable, and their evaluation approach — not their Spider score.

In 2023, Spider — a widely referenced academic benchmark for text-to-SQL tasks — became the go-to standard for one of the most eagerly awaited capabilities in enterprise software: could a large language model take a plain-English question and convert it into a functional database query? By late 2025, every capable foundation model was routinely hitting scores above 85% on Spider. The headlines practically wrote themselves: “AI Reaches Human-Expert Levels on Database Queries.””Goodbye to the Data Analyst.”

And then the same models encountered production data. Accuracy collapsed to 10–20%.

That collapse is not a bug. It is the most important number in enterprise AI today — and it explains why 90 percent of natural-language-to-data pilots quietly die after the demo. The key architectural problem here, then, is understanding why the gap exists — and what closes it. The answer is not what most teams expect, as I learned after two years of building these systems at Oracle’s Generative AI organization.

What the gap actually is

Spider runs against tidy, well-documented databases. Schemas are small. Column names are descriptive. Questions are written by people who already have knowledge about the data behind them.

The real enterprise schema looks nothing like this. It has hundreds of tables. Column names are abbreviations of abbreviations by legacy natural selection; a vestige from someone who designed this system in 2009 that no one has renamed / rewritten ever since. The same concepts go by three different names in three different silos of the business. And the real users’ questions are so full business jargon that never heard by the database.

You may ask a simple question like: “Show me revenue of active customers in the previous quarter.”

Now to a business user this is an obvious question. To the DB it’s a mine field. The column “Active customer” is definitely not. In the systems I was dealing with, it was an abstract idea — a composite of account status, subscription state, and a time window that differed by business unit. Revenue could map to three separate tables, depending on if one is referring to recognized revenue or billed revenue or contracted revenue. To understand what “last quarter” means, you need to know that it is the current date, what month or quarter that falls in according to the fiscal calendar and more importantly which date field on which table is meant to be authoritative.

The SQL a frontier model generates in this situation will be produced with great confidence. The query will compile. It will run. The number it yields will be incorrect — and the model will have no cue that something has gone astray; because it never possessed enough within your business context to worry.

Why the obvious solution fails

Most engineering teams will instinctively want to grab for a bigger model. Two years after seeing teams attempt this, the answer is fairly uniform: It does not work.

A more capable model increases the ceiling of possibility. It does not inform the model about what constitutes an “active customer” in your organization. A model trained on the entire public internet has absorbed more SQL than any human who walks the earth. It has never read a single piece of your internal documentation. That gap, however, isn’t about intelligence — no amount of additional pretraining can close it. It is about context — business context that no one but you has inside your own organization.

The Semantic Layer: The Real Lever

Now the deep-learning companies that close the 85-to-15 gap aren’t necessarily those with the best models. These are those with the most elaborate semantic layer.

For the systems I created at Oracle, we had a paradigm shift that happened where we stopped thinking about it like it was a model problem and started treating it as you would treat a context problem. In the beginning, we were heavily reliant on table and column metadata — literally, raw schema — to contextualize model queries. We noticed that ambiguous business questions were, by far, the largest source of errors. So because the model has no understanding of SQL, it was not failing at learning how to write SQL queries. It was not working because it had no idea what we meant.

We enhanced the semantic layer with three major contributions.

First, explicit business definitions. For each grey area — active customer, revenues, last date of usage — we documented clear plain language definitions and made them discoverable. The model learned what “active” meant from your data up to October 2023 so it stopped guessing.

Second, synonym mapping. Even users go for different words for the same concept: “sales,” “revenue,” “bookings,” ARR. If there is no synonym mapping available, the model matches whatever word the user uses to the most similarly named column in the schema, which often ends up being incorrect. With synonym mapping in place, all of these route to the appropriate native metric.

Third, canonical query examples and validated join patterns. For analytical questions recurring in nature we gave tested query templates, not as parameters to mimic but more like grounding examples for the model to reason through. In particular, it was effective on complex joins where there are multiple plausible-but-wrong paths through the schema.

We ran the same evaluation set after adding these three layers. The improvement on these ambiguous questions was drastic — and it had nothing to do with the model. We had not changed the model. We minimized the amount of business logic that was supposed to be deduced by the model.

The insight that reframed everything

The observation that altered my understanding of this problem is straightforward: Once a model can reasonably generate SQL queries, the higher-level (and remaining accuracy gap) is basically not a model problem but context and semantics.

This has a tangible consequence, one that most teams overlook — a better model is more beneficial when your semantic layer is thick instead of thin. Putting in the work on your semantic layer —business definitions, synonym dictionaries, validated query patterns kept up to date as your schema shifts will give you the kinds of returns that make a demo come alive in production.

The work is unglamorous. Curating a glossary. Another form you can do is write evaluation sets on data that are yours. Before: Listening for scheme changes, and re-indexing. Embeddings refreshed on a defined cadence. Implementing permission checks at retrieval layer so that data that should not be visible to this particular user does get displayed. None of this is seen in a keynote. All of this is what turns a system into something a Fortune 500 bank will let loose on production queries.

Vendors hawking natural-language access to data: 4 questions for you

This will help answer the question: Ask how the system learns your schema. Incorrect answer: “It reads from the database.” A production system will have a clear ingestion process for schema, example values and narrative (descriptions written by humans) — and a way to keep that ingestion fresh.

Imagine that your schema changed? It is also incorrect, “it re-reads it”. The correct answer consists of event-driven re-indexing and a staleness SLA telling the system when its view of your data is stale.

In other words, where does the business language reside? An editable glossary should be presented to you. The vendor is either covering something up or they have not yet built a solution if you cannot edit it.

How is accuracy measured? If the answer is ‘we test on Spider,’ walk away. If the answer is “we use a hand-curated evaluation set on your own data and we report groundedness rate, query success rate, false-answer rate,” keep listening.

The real moment we are in

No new model is going to be the headline of the next few years in enterprise AI. It will be the long, not-so-glamorous process of rebuilding this data layer that lies underneath the model. To the outside observer, it will appear that these companies just cracked AI. They won’t have figured anything out. They will have given up on finding a cleverer system and started doing the hard, long-lasting work of training an AI in what their business really is.

The 85-to-15 gap will not close by sitting back. In the organizations that view enterprise AI as a data discipline before a model discipline, it will close, customer by customer, semantic layer by semantic layer. Meanwhile everyone else will continue to upgrade their model, seeing how the demo dies on real data and coming to all the wrong conclusions.

It works. What needs to be done is simply in a place where most people have not been looking.