Can an expensive model actually be cheaper?

01A token price can't tell you the price of a job

An agent might make several model calls before returning an answer. It can search for a record, read it, do a calculation, and ask for more context. Each step can bring another inference bill. Two models can reach the same answer through very different sequences of calls.

For a simple illustration, imagine a model that charges twice the rate but uses a quarter as many tokens for the same correct result. It costs half as much to do the job. Choosing by token price alone would have picked the more expensive option.

A failed attempt still costs money. Suppose 100 attempts cost $2, but only 80 produce a usable result. You paid two cents per attempt and 2.5 cents per success. Use the second figure when budgeting for completed work.

Artificial Analysis puts cost alongside quality in its September Intelligence Index report. GPT-6 Astra at max reasoning and Claude Fable 5.1 at max with fallback both scored 53, but their average costs per benchmark task were $3.26 and $7.63. Its metric averages across all tasks. We wanted to measure the cost of a correct business result, with failed attempts included in the bill.

02How we ran the benchmark with libfx

We used libfx, Vercel's agent runtime, with its native backend in a TypeScript harness. We changed the model ID while keeping the instructions and tool interface fixed. Libfx ran the loop: call the model, execute its requested tools, feed back the results, and continue until the agent finished.

Each model could list, search, and read the task's records, use a calculator, and submit a structured result. The model chose which tools to call. We wanted to measure the cost of those choices as well as the final answer.

We wrote 30 original synthetic tasks, split evenly across three workflows:

Every model got three fresh attempts at each task, giving us 90 attempts per model and 990 in the comparison. We included complications such as partial payments and conflicting records. These are invented business records, not private customer data.

Each attempt started with a new agent and task state. We gave every model the same request and time limits. The model could inspect the records, but it could not see the answer key or get feedback from the grader.

After the agent finished, deterministic graders checked its submitted result. A good explanation couldn't rescue a wrong amount or a missing decision. Timeouts and incomplete submissions counted as failures too.

We sent inference through Vercel AI Gateway and matched each request to its generation bill. Every included request has a reconciled charge, and failed attempts keep their costs. The benchmark download contains the libfx harness, task records, graders, and complete traces. You can rebuild the tables from saved results without making another model call.

03What happened on one invoice task

An accounts-payable inbox contains two emails with the same invoice number. The first says $5,000. The second says $4,500 and explains that it replaces the original after a pricing error.

We asked the agent to retain both records, identify the corrected version, calculate the $500 difference, and cite the source emails. Returning a plausible total wasn't enough to pass.

On their first attempts, GPT-5.6 Sol and Qwen 3.8 Max submitted identical correct results. Their traces were different:

Sol cost about 0.89 cents; Qwen cost about 1.42 cents. Both did the job. With the traces, we can inspect where they took different steps instead of guessing from the final answer.

This is one attempt from each model, so it illustrates the work rather than establishing a ranking. The results below include every attempt. Read the two traces.

04What the 990 attempts showed

DeepSeek V4 Flash had the lowest observed cost per successful task: $0.00028, passing 89 of its 90 attempts. Qwen 3.8 Flash passed all 90 at $0.00084 per success, making it the least expensive model with a perfect score on this set.

GPT-5.6 Sol, Gemini 3.8 Flash, and GPT-6 Astra also passed all 90 attempts. Astra cost $0.03959 per success. For these tasks, several models reached the same observed success count at very different prices.

DeepSeek V4 Flash also had the cheapest advertised token basket. In this study, cheap tokens did translate into cheap completed work. We could establish that because we checked the answers.

The price side uses the same basket for every model: 1,000 input and 1,000 output tokens. The task side uses the bills for the inference each model consumed. Those bills reflect the rates we received, including caching and provider offers. Pricing details and the full results are in the study appendix.

For another view of cost, we kept the 81 task-and-trial positions that every model passed. This compares bills for shared successes. Since we selected those runs after grading, it cannot measure reliability. The full 90-attempt results remain the main comparison. See the matched costs.

05Keep failed attempts in the bill

Across the comparison, 979 of 990 attempts passed. The inference bill was $11.82081, including $0.14744 spent on failures. We kept those charges in the cost-per-success calculation.

Every model passed between 87 and 90 attempts. That leaves little room for an expensive model to recover its extra inference cost through a higher success rate. A harder task set could produce a different answer.

A failed run also has consequences beyond its inference charge. A wrong refund could cost much more than the tokens used to recommend it. Our benchmark measures inference economics; your production decision may also need review costs and the consequences of acting on a mistake.

06Try it on a workflow you already run

Choose one recurring job in your app and collect examples that represent the work. Include the cases people usually have to untangle: a partial payment, a missing field, an exception to the policy.

Write down what a correct result requires. Run your current model and a few alternatives against the same examples, then compare success rate and cost per successful task. Keep the traces for the expensive attempts and the wrong answers. They can show you whether the problem is the model's output, an unnecessary lookup, or the workflow itself.

The decision can be specific: keep a larger model for cases it handles better, or move routine jobs to a cheaper one after checking the results. You have evidence for where the extra spend is worth it.

All task data is synthetic. The main comparison includes 11 models with reconciled bills; Muse Spark's unresolved charge excludes it from this comparison. Its original records remain in the source download. The study documents model settings, cohorts, billing, and limitations.

Code and complete source results · Result tables · Matched comparison · Readable traces