Method and game data
This is an illustrative simulation, not a model benchmark. Its fixed costs and pass patterns demonstrate a tradeoff. They do not establish current model quality, prices or production savings.
What is the source?
The scorecard is retained from the original Model Golf game design. The underlying evaluation artifacts, ticket texts, reference labels and per-ticket costs have not been verified. We therefore do not present these values as measured results. Model names are labels in this game, not recommendations.
Download the exact game inputs (JSON). The download includes all 17 setups, their ten-position pass patterns, costs, field definitions and assumptions. It reproduces game scores; it is not an evaluation dataset.
How the score works
- The main model receives all ten ticket positions. Each
1in its pass pattern clears that position; each0leaves it for a backup. - The backup receives only the remaining positions. Its added cost is
listed cost × remaining positions ÷ 10. Repeat for a second backup, if selected. - A shot clears the course only if all ten positions pass. Lower total cost wins; fewer swings break ties. Par is the fixed game baseline of $1.5132 per 1,000 tickets.
For example, Mistral Nemo clears nine positions at $0.00371. Ministral clears the remaining position and adds $0.02031 × 1/10 = $0.002031. The total is $0.005741 per 1,000 tickets, about 264 times below the game baseline. These are simulated figures, not an observed routed workload.
What this leaves out
The game knows every wrong answer in advance. In production, checking whether a category is correct requires reliable labels or another validated detection method. Valid JSON alone does not establish correctness.
Backup costs use a per-ticket average, not the actual token cost of the specific failed ticket. Checking, routing, retries, outages, latency and traffic variation are excluded. Ten fixed positions do not establish general model accuracy.
The four courses restrict model choices. The two-vendor course does not test outage recovery. Leaderboard opponents are samples; your best round is stored only in your browser.
For your own application
Compare candidates on representative requests and checks, inspect failures, and measure the complete workflow before drawing conclusions about quality or savings.