A fixed, stratified 150-question subset of BIRD Mini-Dev (SQLite, with evidence; seed 13; split 50 train / 50 val / 50 test). Execution accuracy (EX) uses BIRD's official set comparison against the gold SQL on the same database file. Costs are tokens × published paid list prices — the project itself runs at $0 on free tiers.
Ablation
Updated 2026-10-08
Config
What's on
n
EX
95% CI
LLM calls / q
Tokens / q
List price / q
p50 / p95
C0Baseline
One call, full schema, evidence included
50
48.0%
34.0% – 62.0%
1.0
2,244
$0.0007
929.5 ms / 1.6 s
Execution accuracy with 95% bootstrap CI
Accuracy vs list-price cost per 1,000 questions
EX by difficulty
C0
EX by database · C0
Failures, left in.
Questions the best configuration got wrong, with the gold and predicted SQL side by side.