4 September 2026 · Benchmark · Vasiliy Radostev

The workhorse tier: ten fast, low-cost models measured on production workloads

Most model comparisons focus on the frontier. This one does not. It looks at the fast, inexpensive models that carry agentic automation and high-volume routine work, where the deciding factors are usually cost and latency rather than peak capability.

The capability floor has risen steadily. Models that now sit at the cheap end of the market would have counted as frontier systems a few generations ago, and for a large class of production work they are sufficient. The open question is no longer whether they are good enough, but which of them to choose.

We measured ten such models against three production workloads, scoring quality, latency and cost, then deriving the two ratios that tend to govern a deployment decision: quality per unit of cost, and quality per unit of latency. The intended reader is someone selecting a model for work that requires an LLM but does not require the most capable one available.

The workloads

Three workloads drawn from live traffic in production applications. All three are structured-output tasks of the kind that fills an agentic pipeline, rather than open-ended conversation or reasoning.

1. Recommendation normalization. Convert a product recommendation into a strict JSON record with a canonical brand name, a rank, a justification and a conditional URL. The difficulty lies in schema discipline and house style, not in generation.
2. Event extraction. Read a scraped listing page and return the events it contains as a JSON array, filtered by date, returning an empty array where the page contains none. Long and noisy input, short and precise output.
3. Company summarization. Produce a short descriptive summary of a company from enrichment data, covering market, customers and positioning.

Method

Each workload was frozen as a fixed set of 20 production traces. All ten candidates replayed the same six cases twice, producing 12 scored samples per model per workload and 120 per workload, on identical inputs. Responses were graded against a rubric derived from the workload's own production behaviour by an independent judge model (Claude Sonnet 5), which was not itself a candidate.

Cost and latency are expressed relative to the other models on the same workload, so 0.40x means the model cost 40 percent of what a typical model in this field cost for the same work. The reference point is the geometric mean of all ten candidates on that workload, recomputed per workload and then aggregated, again geometrically. No incumbent or baseline model enters the comparison: every model here is scored against its peers. Quality is an absolute score between 0 and 1 and is averaged arithmetically.

A response that does not conform to the workload's required output schema is scored zero. That is the right treatment for a deployment decision, since an unparseable response is of no use whatever its content, but it means a zero in this study can indicate either a wrong answer or an unusable format. Where the distinction matters below, we say which occurred.

Results

Model	Cost	Quality	Latency	Quality / cost	Quality / latency
GPT-5.6 Luna	0.81x	0.944	0.62x	1.16	1.52
Grok 4.1 Fast Reasoning	1.65x	0.903	1.43x	0.55	0.63
DeepSeek V4 Flash	0.59x	0.865	2.76x	1.47	0.31
MiniMax M3	1.29x	0.722	0.96x	0.56	0.75
Kimi K2.5	2.22x	0.708	0.94x	0.32	0.75
Gemma 4 31B	0.40x	0.622	1.19x	1.55	0.52
Gemma 4 26B A4B	0.46x	0.611	0.62x	1.32	0.99
Nemotron 3 Super	1.03x	0.583	0.92x	0.57	0.63
Grok 4.1 Fast	1.20x	0.569	0.55x	0.48	1.04
Claude 3 Haiku	1.95x	0.486	1.21x	0.25	0.40

Values below 1.00x are cheaper or faster than the field average for the same work. All ten models were measured on all three workloads.

Findings

Highest quality: GPT-5.6 Luna, at 0.944. It was joint second-fastest at 0.62x, behind Grok 4.1 Fast at 0.55x, and cost below the field average at 0.81x. It is the least conditional recommendation here: strong on all three measures at once, and the only model that completed every workload without a formatting failure.

Best cost efficiency: Gemma 4 31B, at 1.55 quality per unit of cost. It is the least expensive model measured, at 0.40x the field average, with mid-field quality of 0.622. DeepSeek V4 Flash follows closely at 1.47, and the gap between them is small enough that the ordering would not survive a change of method.

That headline needs an immediate qualification. Gemma's 0.622 is an average over three workloads, one of which it could not perform at all: on recommendation normalization both Gemma variants failed all twelve cases because their output did not conform to the required response schema. Generation succeeded every time; the responses simply could not be consumed. Read Gemma's cost efficiency as applying to the two workloads it completed, not as a general result. On those two it scored 0.917 and 0.833, which is competitive.

The more useful distinction is what each trades. Gemma is cheapest where it works. DeepSeek is better (0.865), completed all three workloads, and costs about 50 percent more.

Also of note

DeepSeek V4 Flash, at 0.865 quality and 0.59x cost. Second on quality per unit of cost at 1.47, just behind Gemma 4 31B, and the strongest of any model that completed all three workloads. Undercut by latency: 2.76x the field average, the slowest model measured. That is immaterial for batch and asynchronous work and disqualifying for anything interactive.

Gemma 4 26B A4B, at 0.611 quality, 0.46x cost and 0.62x latency. Second-cheapest, joint second-fastest, and third in cost efficiency at 1.32. It carries the same schema-conformance failure as the 31B on recommendation normalization, so the same qualification applies.

The two Grok 4.1 Fast variants. These share a token price, and the aggregate scores diverge sharply: 0.903 for the reasoning variant against 0.569 for non-reasoning. The per-workload figures show the gap is not general. On company summarization the two tied exactly at 0.833; on recommendation normalization reasoning led modestly, 0.875 to 0.792. The aggregate difference comes almost entirely from event extraction, where the non-reasoning variant scored 0.083, close to total failure, while the reasoning variant scored 1.000. The reasoning variant is the safer choice, but on the evidence here that is because it avoids a catastrophic failure on one task rather than because it is broadly better.

Cost behaves differently from what a shared token price implies. Reasoning emits more tokens, so per request it cost 3.25x more on recommendation normalization and 1.39x more on company summarization. On event extraction it cost 0.58x, less than non-reasoning, because the failing variant produced long unusable output.

One negative result is worth stating directly. Claude 3 Haiku placed last at 0.486 quality, and was the second most expensive model in the field at 1.95x. An older small model is not necessarily an inexpensive one, and on structured work of this kind the distance to the current generation is considerable.

Obtaining these models

All ten were called through Vercel AI Gateway, which is why ten models from eight labs could be run against a single credential and billing surface. Aggregators such as Vercel, Fireworks, OpenRouter and Bedrock are the practical route for a comparison of this kind, and increasingly for production, since they allow a team to move between models without a new contract or a new integration.

Going directly to a lab (OpenAI, Anthropic, Google, DeepSeek) typically offers earlier access to new releases, occasionally better rates at volume, and a direct support relationship, at the cost of one integration per provider. Aggregator and direct pricing are not always identical, and we have observed the same model priced differently between a lab's own API and a gateway in both directions.

Self-hosting

Several of these models are open-weight, including DeepSeek V4 Flash and both Gemma variants, so renting GPUs and serving them directly is a viable alternative. Whether it is economic reduces to weekly request volume.

The measured absolute cost per request on the gateway, taken from the runs above:

On these workloads a low-cost hosted model runs between $0.0001 and $0.0005 per request. Gemma 4 31B averaged $0.000255 across the three. The break-even against self-hosting is:

requests per week to break even  =  weekly cost of the GPU instance
                                   ----------------------------------
                                   hosted cost per request

A 31B model fits comfortably on a single 48GB card. As of early September 2026 an L40S-class instance rents between $0.48 and $2.20 per hour depending on provider tier, with a median near $1.54, and an A100 80GB starts around $1.09. Dedicated GPU clouds run at roughly half hyperscaler rates. Running one card continuously:

So the threshold sits somewhere between roughly 300,000 and 2 million requests per week, depending on the card and the deal, before self-hosting wins on infrastructure alone. That figure ignores engineering time, idle capacity outside peak hours, autoscaling and on-call, all of which push the real threshold higher.

Self-hosting these models is usually a decision about data residency, latency control or vendor independence. It becomes a cost decision only at substantial volume.

Rates checked against provider listings on 4 September 2026 and against the Vercel AI Gateway model catalogue the same day; GPU rates in particular move quickly.

Limitations

• Twelve samples per model per workload. Sufficient to separate 0.9 from 0.5, not to separate 0.62 from 0.61. Small differences should be treated as noise.
• Three workloads, all structured output. The results say nothing about prose generation, conversation or tool use.
• Cost and latency are relative to the field on each workload, not absolute. A model at 0.40x is cheap compared to these nine peers, not cheap in general.
• A single judge model. A different judge would shift the scores, though we would expect the ordering to be broadly stable.
• Four of the ten models earned at least one zero for schema conformance rather than for answer quality: both Gemma variants (12 of 12 cases), Nemotron 3 Super (4) and Kimi K2.5 (3), all on recommendation normalization. Their quality averages understate them as answer generators and correctly state them as drop-in components for that workload.
• Measurement depth is not uniform. Eight models are scored on the screening sample throughout. DeepSeek V4 Flash on event extraction and Gemma 4 31B on company summarization cleared the cost threshold, advanced to a deeper evaluation, and are reported on that larger sample for those workloads. Both are flattered slightly as a result; on screening depth alone Gemma 4 31B scores 0.583 rather than 0.622, which would place it below Gemma 4 26B A4B.

Questions or suggestions: vasiliy@psl.com