heron
A text-to-SQL benchmark that runs on a database shaped like production: one multi-tenant SaaS schema, 211 FK-linked tables across 14 domains, millions of deterministically seeded rows, and 100 questions whose gold SQL is scored by execution-equality.
I built it because I needed a number I was allowed to publish. PromptQuery claimed to be good at production-scale schemas; the existing benchmarks all run on five- to seven-table toy databases, so nothing in the field could confirm or refute that. This page is how heron was made, and why you should believe what it says — including about my own tool.
The gap it fills
The field is strong and heron stands on its shoulders — it is not the first hard schema, the first to isolate table retrieval, or the first Postgres benchmark. What is genuinely empty is the intersection: a single coherent FK-linked multi-tenant Postgres schema of ~211 tables, rebuildable on a laptop from a deterministic generator with no cloud account and no private data, with schema-retrieval-at-scale measured as a first-class axis.
| Benchmark | Tables | Dialect | Reproducible locally | One schema |
|---|---|---|---|---|
| Spider 1.0 | ~5 | SQLite | yes | no — 200 dbs |
| BIRD | ~7 | SQLite / MySQL / PG | yes | no — 95 dbs |
| Spider 2.0 | ~53 | BigQuery / Snowflake | no — cloud account | no |
| LiveSQLBench-Large | ~54 | PostgreSQL | yes | no — 18 dbs |
| BEAVER | ~101 | Oracle / MySQL | no — private warehouses | no |
| heron | ~211 | PostgreSQL 16 (local) | yes — dump + seed | yes — 14 domains |
Sizes and citations for each row are in the repo's RELATED-WORK.md. The differentiator is verifiable reproducibility, not difficulty.
How it was built
Seven decisions, each one recorded in DECISIONS.md at the time it was made — with the rule that reversing one means amending the file in the same change and saying why.
Put it outside the tool it measures
The first decision was the one that mattered most, and it cost the project its easiest shortcut. A benchmark shipped inside the repo of the tool that tops it gets dismissed on sight — of course prq wins its own benchmark. So heron is a standalone repository that imports nothing from PromptQuery. prq is one adapter in harness/adapters/, shelling out to the installed CLI exactly like Vanna, MAC-SQL and DIN-SQL do. Spider, BIRD and Spider 2.0 are all independent of any product; matching that was the price of being taken seriously.
Choose a domain that sprawls honestly
Multi-tenant B2B SaaS commerce. Auth, tenancy, catalog, pricing, inventory, orders, billing, subscriptions, CRM, support, marketing, analytics, audit, comms, ops — fourteen modules that reach 200+ FK-linked tables without a single padding table invented to inflate the count. It is also instantly legible: everyone understands orders and refunds, which matters when a reviewer has to trust the gold SQL.
Write the schema as a design contract
211 tables across 14 schemas, with a written CONVENTIONS.md the DDL has to obey — naming, tenant_id placement, soft-delete columns, nullable FKs, currency handling. The traps are deliberate: tenant isolation, soft-deleted rows, nullable joins, mixed currencies, time bucketing. Production data is messy and skewed, so the benchmark's is too.
Generate the data, don't ship it
A single seeded RNG in seed/generate.py produces every row, so a given (scale, seed) pair is byte-reproducible on any machine. Four scale factors: tiny (~0.2M rows, CI smoke), small (4.4M rows, the eval scale, 233 MB compressed), bench (87.8M rows, 4.6 GB), large (~430M rows). No cloud account, no private warehouse, no signup — make up && make seed on a laptop. That constraint is the whole differentiator: reproducibility, not a difficulty boast.
Author 100 questions, and prove each gold answer
The most damaging failure mode in published benchmarks isn't the metric — it's wrong gold answers. A 2026 audit found annotation errors in 52.8% of BIRD Mini-Dev and 66.1% of Spider 2.0-Snow items, and fixing the gold reordered the leaderboard. heron owns its generator and writes its own gold, so every gold query is executed at authoring time against a fresh database, its result inspected, its referenced tables parsed with sqlglot and checked against the hand-labelled gold_tables set. Each question is tagged on two orthogonal axes — SQL shape (single / join / multi-join / analytical) and retrieval difficulty (named / 1-hop / 2-hop+ / lexical-gap).
Build the harness around execution, not string match
Scoring runs the SQL. A prediction is correct only if its result set equals gold's under a hardened comparator: multiset semantics, order-insensitive unless gold has an ORDER BY, column-permutation tolerant, float tolerance 1e-6, NULL == NULL, and an empty result only ever matches an empty gold. Around that sit VES (efficiency), Soft-F1 (partial credit), Set-Recall@k (the signature retrieval axis), an exact OpenAI billed-token meter, and a transparent 0–100 Grade weighting EX at 0.45.
Hand the leaderboard to CI
Nobody edits leaderboard.svg / .json / .csv — not even me. A submission is a folder with the adapter, the results and the metadata; a workflow re-runs its SQL and regenerates the standings. Which is also how PromptQuery's own row moved from 0.2.x to 0.3.0: same flow, same verification, no privileged path.
Scale factors — measured, not estimated
| Scale | Use | Rows | On disk | Compressed dump |
|---|---|---|---|---|
| tiny | CI smoke | ~0.2M | ~90 MB | < 10 MB |
| small | the eval scale | 4.4M | 1.2 GB | 233 MB |
| bench | "millions of rows" artifact | 87.8M | 22 GB | 4.6 GB |
| large | retrieval under load | ~430M | ~110 GB | several GB |
Every volume comes from one seeded RNG, so (scale, seed) is byte-deterministic on any machine.
Why the numbers hold up
A benchmark is only worth the weakest link between its claim and its evidence. These are the eight places heron closes that gap — and the ones a skeptic should attack first.
Numbers are recomputed, never accepted
verify-submission spins up Postgres 16, loads the schema, seeds the gold database at (small, 42), audits it, then re-executes every pred_sql in the submission and recomputes EX@1, VES, Soft-F1, Set-Recall, errors and timing. The check fails if the run is incomplete, uses a non-gpt-4o model, contains non-SELECT SQL, or claims an accuracy its own SQL can't reproduce. It runs with a read-only token, so it works identically on fork PRs.
The rejection is published, not deleted
leaderboard.json carries a rejected list. LangChain sits in it: claimed EX@1 16.0, but the submitted SQL only reproduced 3.0 on re-run — a gap past the 2-point tolerance, so it never reached the board. That entry is the load-bearing proof that verification is real. A benchmark that has never rejected anything hasn't demonstrated it can.
Gold passes an audit before anything is published
make audit is a release gate: every gold executes without error; every gold returns non-empty unless the question is explicitly an empty-set probe; gold results stay structurally stable across seeds; gold_tables exactly equals the parsed table set; no duplicate questions; a second author must be able to reconstruct the gold from the English alone, or the question is reworded or split out as ambiguous.
The comparator is strict on data, lenient on shape
Natural language doesn't pin column order or names, so those are permuted before comparison. Everything about the data itself is strict: duplicate rows must match in multiplicity, order is enforced when the question asked for an order, strings are case-sensitive and untrimmed, and a blank answer is never a free pass. Correct paraphrases pass; wrong-but-similar SQL does not.
Coincidental correctness is designed out
A query like WHERE status='paid' can match gold on one database state by accident. Because heron owns the generator, the methodology defines the headline as multi-state EX@1 — correct on all N=3 independently seeded states of the same schema, or not correct at all. EX@k with the unbiased pass@k estimator is reported separately as a capability ceiling, never as the headline: it answers could the system do this, not does it reliably.
One model, one machine, one run
Every tool on the board generates SQL with the same gpt-4o at temperature 0 — a fine-tuned or local model breaks the control and is refused. Token counts come from the billed response.usage, not an estimate. Timing warms the cache once, then keeps the minimum of five repetitions, and every tool is timed on the same box in the same run, so efficiency is a fair relative comparison even though it is hardware-dependent in absolute terms.
Every number ships with its conditions
No bare percentage is ever published. Scale, model, temperature, EX@1 vs EX@k, single- vs multi-state, and n travel with the number, and aggregates are always shown next to their per-bucket breakdown — which is how you can see that prq's 72% is 82.8% on single-table questions and 20% on window functions.
Losing results stay in the repo
Unfavourable runs are committed on purpose, including ones where heron's own namesake tool doesn't top a column. Reproducibility is the moat: every number traces to a committed question, a committed gold query, and a database anyone can rebuild deterministically. Run the harness yourself and you should get the same answer — that is the only claim that matters.
The standings
Five tools, 100 questions each, all generating SQL with the same gpt-4o at temperature 0 against the same seeded database. The only difference between them is how each one ingests 211 tables and decides which handful to use.
| # | Tool | Approach | Grade | EX@1 | VES | Set-Recall | Tokens/q | Cost | Errors |
|---|---|---|---|---|---|---|---|---|---|
| 1 | PromptQuery0.3.0 | retrieval | 68.7 | 72.0 | 72.0 | 99.0 | 4,689 | $1.21 | 0 |
| 2 | Vanna0.7.9 | RAG | 61.9 | 46.0 | 46.4 | 79.0 | 2,000 | $0.53 | 18 |
| 3 | raw-gpt-4o | full-schema dump | 56.8 | 55.0 | 55.4 | n/a | 15,314 | $3.87 | 10 |
| 4 | MAC-SQL | multi-agent | 48.2 | 49.0 | 49.0 | 90.0 | 15,656 | $4.00 | 3 |
| 5 | DIN-SQL | decomposed | 47.9 | 52.0 | 52.8 | 90.0 | 16,343 | $4.21 | 6 |
Conditions: model openai/gpt-4o, temperature 0, EX@1, n = 100, small scale (4.4M rows), seed 42. Grade weights EX 0.45 · efficiency 0.20 · token economy 0.15 · reliability 0.10 · latency 0.10. Cost is the exact OpenAI-billed total for the run. A sixth submission — LangChain — was rejected for claiming EX@1 16.0 when its own SQL reproduced 3.0.
What it does not claim
Overclaiming is the fastest way to make a benchmark worthless. These limits are stated in the repo and they are stated here.
It is one benchmark on one schema in one domain. A tool that tops heron has proven it handles a sprawling multi-tenant SaaS Postgres schema — not that it handles yours.
The published run is a single seeded state. The multi-state N=3 protocol is specified in the methodology and is what future headline numbers are held to; the 0.3.0 results on the board were scored single-state, and that condition is stated wherever the number appears.
Contamination-resistant, not contamination-immune. The public split's generation seed is disclosed, and a held-out seed is reserved for a future hidden split — that is a resistance claim, not a permanent one.
The board is young. Five tools, all adapters I wrote myself, all verified by the same CI. External submissions are the point of the contribution flow, and none have landed yet.
Efficiency is hardware-relative. VES compares tools against each other within one run on one machine; it is not an absolute cross-run constant, which is why it never gates correctness.
Run it yourself
The fastest way to check any of this is not to read the page. Four commands rebuild the database and reproduce the gold baseline at 100% EX; the fifth reproduces a tool's row.
make up # Postgres 16 in Docker
make schema # 211 tables, 14 modules
make seed SCALE=small SEED=42 # 4.4M deterministic rows
make bench ADAPTER=gold # sanity: 100% EX
make bench ADAPTER=promptquery MODEL=openai/gpt-4oBack to PromptQuery or the rest of the site.