Back to the article / 8 September 2026 / Research / Metergraph

Cost per task: study protocol and appendix

Status: the main comparison includes 11 models, with 90 scored attempts per model. See measured results. The source studies retain 1,080 scored attempts across 12 models. Muse Spark is excluded from the main comparison and all charts because one timeout charge remains unresolved. Its original 90 attempts and known charges remain in the source data. All task data is original and synthetic.

Question and population

Given an identical task, tool environment, and execution budget, how much inference spend buys one successful business outcome? The audience is AI app builders selecting models for bounded workflows. The population represented is this hand-designed collection of synthetic tasks, not all business work or all agent applications.

There are 30 scored tasks, ten in each workflow, and six separate pilot tasks. Each scored task receives three fresh trials per model. The original ten-model study contains 900 scored runs. At the user's request, GPT-6 Astra and GLM-5.3 Flash subsequently received the identical 30 tasks and three trials, adding 180 scored runs with no changes to the original runs, fixtures, graders, or model-facing settings. Pilot cases are excluded from scored summaries. The pilot uses every model on each of the six cases so it samples all three workflows. Preflight tool-use checks are also excluded.

Cases include straightforward requests and complications involving duplicated records, absent information, conflicting records, partial payments, and policy exceptions. Expected answers and rationales are reviewed independently before freezing the scored study. The review records who or what performed it; agent review must never be represented as review by a human.

Models

The table lists the eleven included models. Muse Spark (meta/muse-spark-1.3) was tested in the original study but excluded from the main comparison for incomplete billing, at the user's request. The two new models passed separate live tool-use/billing checks and the same six-case pilot before their extension manifest was frozen.

Opus 5 replaces Haiku 4.5 at the user's request. Exact IDs are fixed. Catalog availability, live tool compatibility, and bill reconciliation are separate checks. An unavailable model is recorded explicitly and is not silently replaced. A scope change requires a revised protocol before scored inference.

Agent environment

Node 24 and libfx@0.0.8 with the native backend. FX owns the agent loop. Every model receives the same tools for reading/searching task-local records, calculation, and structured submission. Agents have no shell, filesystem browsing, external web access, or answer-key tool. A new task state and agent are created for each trial. A preflight test exposed a libfx 0.0.8 native bridge failure on otherwise valid multiline tool-call JSON. The transport compacts valid JSON whitespace for every model before passing it to FX, preserving parsed argument values and the original stream in the trace. It does not repair malformed JSON. The fix was reproduced with saved Qwen output and an offline native regression before the scored freeze. Grading occurs after completion and cannot be used to retry against the hidden answer.

The seeded schedule interleaves models. It began at concurrency two. At the operator's request, concurrency increased to eight after 134 completed scored attempts; the remaining 766 attempts used the eight-worker entrypoint. The original two-worker process drained its active tasks before the handoff, with no aborts or replays. The core agent, fixtures, tools, graders, request/output limits, and per-task deadlines did not change. After 850 attempts, a timed-out request without a generation ID paused scheduling. A separately reviewed continuation completed the remaining 50 at eight workers while preserving the missing charge as unresolved. No task was replayed and no missing cost was priced at zero. The separate concurrency amendment records the frozen core fingerprint, operational entrypoint hash, boundary, and effective worker limit per subsequent trial. This mid-study change can affect latency, cache behavior, and provider routing, so aggregate latency is not a comparison under one constant concurrency setting. Settings use provider-default reasoning and temperature, standard service tiers, default Gateway provider routing, and no cross-model fallback. These defaults are part of the treatment and need not imply identical reasoning behavior across vendors. Actual request bodies and observable routing/cache metadata are saved. Unobservable provider internals are labeled unknown.

Each trial has a 12-request limit, 4,096 output tokens per request, and a 180-second deadline. Transport retries count toward the request limit. Gateway's internal provider routing attempts may not be observable as separate client requests; the appendix must report that boundary explicitly. A bounded client deadline does not prove an upstream provider stopped work immediately.

The Astra/GLM extension used eight workers throughout and imported the original frozen agent execution and scheduling functions. Its separate manifest pins the original fingerprint and extension entrypoints; its results and bills use a separate journal. This later cohort can experience different routing or cache conditions. No original attempt was modified or replayed.

Costs and execution gates

The initial allocation was $25 for preflight and pilot combined, $200 for scored runs, and $25 contingency. Before scored inference, the operator explicitly removed dollar-budget stops and authorized the full study with account recharge enabled. The pilot cost projection is therefore informational. Request limits and task deadlines remain fixed; reservation overruns are recorded without terminating inference.

Reserve a conservative request charge before sending it. Persist that reservation before network activity. Settle only against validated generation billing records, deduplicated by generation ID. Unknown costs remain unresolved and reserved. Save failed calls and retries. Never interpret an empty or missing billing field as zero. An interrupted run is not silently replayed; journal recovery must retain any possible charge.

The scored matrix starts only after independent fixture/grader review, a frozen manifest, live tool-use and reconciled billing for every model, and a completed pilot. The original $200 projection gate was removed by the operator before scored runs. The reviewed budget-only amendment records the prior qualification fingerprint and the exact new runtime fingerprint, preserving completed smoke/pilot work without changing the model-facing treatment.

Gateway generation records provide billed costs, provider identity, usage, cache fields, and timing. Preserve the original validated records so the report can be reproduced after prices change. Unaccounted external BYOK charges prevent complete economics. Current price-list arithmetic is for planning and comparison, not a substitute for the saved bill.

Grading

Each task is all-or-nothing for the primary success count. Correct syntax alone cannot pass. A support answer with the wrong refund amount fails even if its explanation sounds plausible. A reconciliation with the right total but the wrong payment allocation fails. Extraction must retain the required provenance and must not invent missing fields.

Graders check substantive values and evidence, reject malformed structures, and catch missing, extra, and duplicate results. Equivalent orderings may be accepted where order is irrelevant. Monetary values use exact cents where the task requires cents. Dates use a documented normalized representation. No model-based judge determines the primary outcome.

All revisions to tasks, answer keys, grading, prompts, settings, runtime code, or dependency versions before scored runs are reflected in a new manifest. After freezing, changes invalidate the freeze. A discovered grader defect after scored inference must be disclosed, corrected consistently for every saved output, and versioned rather than selectively changing a losing model's grade.

Sol pricing and the article comparison

The primary metric uses Gateway total_cost, the actual debit, including observable routing and discounts. Gateway separately reports market_cost at market list inference rates (API definition). These fields are not interchangeable.

The captured catalog quoted Sol at $2/M input, $10/M output, $0.20/M cache reads, and $2.50/M cache writes. Those prices already reflect its advertised Gateway promotion. Applying them to the native prompt, completion plus reasoning, cache-read, and cache-write counts reproduces 364 of the 365 scored Sol generation bills exactly. The remaining Bedrock request cost $0.000088 rather than the $0.000044 implied by the OpenAI rates. No additional 50% reduction should be applied to the quoted catalog prices.

Sol's actual scored bill is $0.8361001, or $0.0092900011 per successful run. Its separate market list total is $1.6721122, or $0.0185790244 per success. Qwen Max's billed and market totals both equal $1.09892398, or $0.0124877725 per success. Thus Sol is 25.6072% cheaper at the rates actually charged, while Qwen is cheaper on the Gateway-reported market list measure. The matched 81-position comparison gives Sol a 23.7671% billed-cost advantage, conditional on shared success.

Vercel announced 50% off Sol on the OpenAI provider through September 18. That announcement's older price table is not the source for our captured rates. The source is our September 8 catalog snapshot, consistent with the model page checked for publication. The public charts use the captured offered rates and actual bills, not an undiscounted or permanent price ranking. Machine-readable billed and market sums.

The article's external context comes from the September 7 Artificial Analysis Intelligence Index v4.3 report. Its cost measure averages across all tasks with evaluation weights; our cost-per-success denominator counts only passing attempts. The workloads and settings also differ. We do not compare our dollar values directly with its dollar values or claim to reproduce its ranking.

Metrics

For each model, per workflow and overall:

cost per successful task = sum(inference spend on all attempts) / count(successful attempts)

The denominator counts successful independent trials, not distinct fixture IDs. The three repeats estimate run-to-run variation; they are not an operational retry policy. A zero-success group has infinite cost per success when billing is complete. A group with missing costs has incomplete economics, even if its success count is known. No-run groups are not measured zero-success groups.

Also report success rate, inference cost per attempt, total and failed-run spend, latency, input/output/reasoning/cache token usage where available, and tool calls. The overall result weights each of the three workflows equally because each contains ten tasks with the same repetition count. Do not extrapolate the workflow mix to a customer's production traffic.

Use a seeded nonparametric bootstrap that resamples whole task IDs, retaining their repeated trials together. For overall intervals, resample tasks within workflow to preserve the designed mix. For model comparisons, use the same sampled task IDs across models. Report percentile intervals and the bootstrap seed/repetition count. Preserve infinite endpoints when a resample has zero successes. A group that passes every observed task can have a degenerate 100% to 100% bootstrap interval; that describes these sampled tasks and does not establish perfect production reliability. These intervals describe uncertainty across this small task set; they cannot correct for unrepresentative synthetic data.

Tables, figures, and trace examples

All published values must come from saved, versioned JSONL results and billing, then deterministic grading and reporting. Regeneration must require no inference. Publish the fixture/code manifest, all denominators, the actual number of completed runs, and unresolved billing counts.

1. Cost and success: show pass counts alongside inference dollars per success, with units labeled. Include uncertainty and separate workflow views where the overall plot hides important differences.
2. Advertised pricing versus observed economics: state a fixed illustrative input/output token basket for the advertised-price side. It is a price-list comparison, not a prediction of agent token use. The observed side includes failed-run spend.
3. Spend on successful versus failed runs: show dollars and the success denominator. Money spent on failures is not proof that all such spend can be eliminated.

Select two or three trace examples after scoring. Prefer a contrast on the same task, a failure with a clear substantive error, and a successful case that explains a cost difference. Include task ID, model ID, trial ID, relevant tool steps, submitted result, grader finding, generation IDs, and reconciled charge. Avoid choosing only examples that flatter the sponsor or a preferred model.

Post-hoc matched comparison

The user requested 85 matched runs per model, excluding aligned task/trial positions failed by any included model. We select only positions where every one of the eleven included models passed, up to 85. When more than 85 qualify, selection takes the lowest SHA-256 hashes of 90826:fixtureId:trial, independently of prices. When fewer qualify, we publish the actual count. No supplemental task, extra trial, replay, changed grade, or deleted source run is permitted.

Matched results and the machine-readable selection retain every selected position and every excluded position with its failures. This outcome-dependent subset measures costs conditional on shared success. It cannot estimate reliability, eliminate failed-run spend from the main results, or support an unconditional cost-per-success claim. Its bootstrap intervals describe variation within the selected subset only. The full comparison of 90 attempts per model remains the primary result.

Interpretation and limitations

This measures task-level inference economics under one harness. It does not measure production reliability, adversarial security, user satisfaction, human escalation cost, or end-to-end labor savings. Local fixture tools have no external service fees. Real systems can have larger records, different context lengths, more costly mistakes, latency requirements, or additional retries.

Provider defaults, routing, and caching can affect both cost and success. The small number of independently designed tasks limits precise rankings, even with 1,080 source trials. Closely overlapping estimates should be presented as inconclusive. No headline requires a particular winner, a reversal of token-price rankings, or statistical significance.

The public project is Apache-2.0, original synthetic inputs only, with ungated code and results. The benchmark code and original synthetic source data are available in the accompanying download.