Skip to content
DATAPILOT
Sign in

BIRD Mini-Dev · execution accuracy

What each agent buys — and what it costs.

A fixed, stratified 150-question subset of BIRD Mini-Dev (SQLite, with evidence; seed 13; split 50 train / 50 val / 50 test). Execution accuracy (EX) uses BIRD's official set comparison against the gold SQL on the same database file. Costs are tokens × published paid list prices — the project itself runs at $0 on free tiers.

Ablation

Updated 2026-10-08

ConfigWhat's onnEX95% CILLM calls / qTokens / qList price / qp50 / p95
C0BaselineOne call, full schema, evidence included5048.0%34.0% – 62.0%1.02,244$0.0007929.5 ms / 1.6 s
Execution accuracy with 95% bootstrap CI
Accuracy vs list-price cost per 1,000 questions
EX by difficulty

C0

EX by database · C0

Failures, left in.

Questions the best configuration got wrong, with the gold and predicted SQL side by side.