Frontier models for a software factory: where the additional cost is justified
This article is for engineers who run coding agents against their own repositories and who pay the inference bill.
Several frontier models have been released since the summer, and each was presented as an advance on agentic software work: GPT-6 Astra, Claude Opus 5.5 and Fable 5.1, Grok 4.7 and Kimi K3. For a team that already runs a fleet of coding agents, the practical question is whether that fleet should move onto one of them.
We measured the models on six workloads that are typical of a software factory: design, coding, code review, CI triage and various pull request operations. We evaluated the performance of the models on each workload individually, using traces from real software factories.
Frontier models only for the most challenging tasks
LLMs are still not perfect software designers, and design tasks need the best models. Frontier models are clearly ahead, but all of them have room for improvement. Among the frontier models, GPT-6 Astra, Grok 4.7 and Opus 5.5 were close to one another at the top. Kimi K3 and Fable 5.1 were noticeably behind. Even the best design anticipated only about half of what human reviewers expected.
For the other five workloads, a high-efficiency model is the better choice. The top performers are GPT-6 Luna and GLM 5.3 Flash, which were practically as good as the frontier models at a fraction of the cost. If you are using Anthropic models, Claude Sonnet 5.5 is a great choice.
Design work
Frontier models are clearly ahead, at 0.46 against 0.26 on average, and on none of the eight design tasks did the best high-efficiency model match the best frontier model. GPT-6.1 Sol, which is not a frontier model, scored as high as Astra at a quarter of the price.
Every model did well what was directly asked. The key difference was in how the models performed on implicit requirements. A good design also covers the cases that nobody wrote down: what happens when an input is missing, what a later step will do with the result, and which data a number is based on. The stronger models covered these cases more often.
Getting the design right is important, because it avoids building the wrong thing, so it is worth paying extra for a quality design. Astra wrote a typical design for about 31 cents and GLM 5.3 Flash for under one cent, but the savings in human work are worth the extra price. Grok 4.7 matched Opus 5.5 at about a third of the price, but took a median of four minutes per design, against about a minute and a half for Opus 5.5 and one minute for Astra.
Coding, code review, CI triage, working with pull requests
On these five workloads, twelve of the thirteen candidates met between 0.92 and 1.00 of the checks on average. Most of the difference between the models is in price and speed, not in quality. For the same task, GPT-6 Luna costs about 3 percent of what Sonnet 5.5 costs, and Fable 5.1 costs about five times as much. GLM 5.3 Flash completed a step in about 4 seconds, and Kimi K2.7 Code took about 15. GPT-6.1 Sol met 1.00 of the checks at 61 percent of Sonnet's cost. Opus 5.5 scored 0.95 and Sonnet 5.5 scored 0.96, so the larger model's additional cost buys no measurable quality. Grok 4.1 Fast Reasoning, at 0.86, was the exception, and its provider failed between 25 and 88 percent of its requests.
Code review and commit and pull request separated the models most. On code review, Gemini 3.1 Pro met 0.70 of its checks, while Sol and Kimi K2.7 Code met 0.99. On commit and pull request, Sol, Astra, Luna and Grok 4.7 met 0.99 or more, while Kimi K3 and GLM 5.3 Flash met 0.90. The Anthropic models scored lower on this workload, Opus 5.5 at 0.86 and Sonnet 5.5 at 0.87, but they were graded by a different judge, as the note under the table explains.
| model | coding | code review | CI triage | commit and PR | merge | design | cost vs Sonnet 5.5 | latency |
|---|---|---|---|---|---|---|---|---|
| Fable 5.1 | 0.95 | 0.95 | 0.99 | 0.91 | 0.93 | 0.41 | 4.93x | 11.4 s |
| Opus 5.5 | 0.96 | 0.94 | 1.00 | 0.86 | 0.98 | 0.49 | 1.99x | 8.8 s |
| Sonnet 5.5 | 0.98 | 0.98 | 1.00 | 0.87 | 0.96 | 0.46 | 1.00x | 7.3 s |
| Gemini 3.1 Pro | 0.97 | 0.70 | 1.00 | 0.94 | 0.98 | 0.09 | 0.72x | 7.1 s |
| Gemini 3.8 Flash | 1.00 | 0.90 | 1.00 | 0.95 | 0.97 | 0.17 | 0.26x | 4.8 s |
| Kimi K3 | 0.99 | 0.88 | 1.00 | 0.90 | 1.00 | 0.38 | 0.95x | 5.3 s |
| Kimi K2.7 Code | 0.98 | 0.99 | 1.00 | 0.95 | 0.99 | 0.18 | 0.29x | 15.3 s |
| GPT-6 Astra | 1.00 | 0.91 | 1.00 | 1.00 | 1.00 | 0.52 | 3.04x | 6.0 s |
| GPT-6.1 Sol | 1.00 | 0.99 | 1.00 | 1.00 | 1.00 | 0.55 | 0.61x | 6.2 s |
| GPT-6 Luna | 0.96 | 0.97 | 1.00 | 0.99 | 1.00 | 0.28 | 0.03x | 5.3 s |
| Grok 4.7 | 0.92 | 0.91 | 0.99 | 0.99 | 0.96 | 0.51 | 0.67x | 8.4 s |
| Grok 4.1 Fast | 0.51 | 0.85 | 0.94 | 1.00 | 1.00 | 0.22 | 0.06x | 13.0 s |
| GLM 5.3 | 0.96 | 0.87 | 1.00 | 0.94 | 0.98 | 0.43 | 0.44x | 8.4 s |
| GLM 5.3 Flash | 0.99 | 0.94 | 1.00 | 0.90 | 1.00 | 0.32 | 0.05x | 3.8 s |
Each score is the share of checks a model met. Cost and latency are averages over the five agent workloads, relative to Sonnet 5.5. The cost of a design is shown on the design chart, relative to Opus 5.5. On the five agent workloads, GPT-6.1 Sol graded the Anthropic models and Claude Sonnet 5.5 graded all other models. Sol grades more strictly, so the Anthropic rows are best compared with each other. The design column is the mean of both judges.
Method
Each workload was evaluated by replaying sample traces. For the five agent-step workloads, four cases per workload were replayed twice per model, with the transcript, tools and repository rules the agent had at that step. A judge scored each step on whether it was a defensible next action, whether its arguments matched the transcript, and whether it advanced the task, together with checks drawn from the repository's rules. These workloads measure a single step taken from a real session, not the performance of the software factory as a whole. End-to-end performance may be different. For design, each model received eight design requests with the code as it stood before the change, and its design was scored on checks about consequences the request left unstated and on how many of the human reviewers' findings it anticipated.
Claude Sonnet 5.5 judged the non-Anthropic models and GPT-6.1 Sol judged the Anthropic models. Both judged every design, and the design score is their mean. Cost is per task, relative to Sonnet 5.5 on the agent workloads and to Opus 5.5 on design. With a small number of cases per workload, the results can rule a model out, but they cannot rank the models nearest the top. Gemini 4 Argon was not available to us at the time of testing. The provider refused two of Grok 4.1 Fast Reasoning's eight design requests, so its design score covers six.
References
Model prices come from the providers' own pricing pages: OpenAI at https://platform.openai.com/docs/pricing, Anthropic at https://www.anthropic.com/pricing, Google at https://ai.google.dev/gemini-api/docs/pricing, xAI at https://docs.x.ai/docs/models, Moonshot AI at https://platform.moonshot.ai/docs/pricing/chat and Z.ai at https://docs.z.ai/guides/overview/pricing.
The models were called through the Vercel AI Gateway, whose model list at https://vercel.com/ai-gateway/models shows the routes and prices used.
The workloads were replayed and graded using Metergraph, at https://metergraph.dev.