How analysis works
Analysis is the part of Metergraph that has an opinion. Everything else reports what happened; this asks whether it could have happened for less.
The question it answers is narrow on purpose: for this workload, is there a model that costs less and does the job as well? Not “is this prompt good”. Not “is this the best possible architecture”. One question, answered from your own traffic rather than from a benchmark somebody else ran on somebody else’s data.
Where it runs
Section titled “Where it runs”An analysis is not part of the capture path and not part of the always-on worker. It is a separate, isolated, one-off task.
When you start a run, the API snapshots the deployment’s fixed analysis profile,
writes a queued row, and launches one task. That task exports a window of your
stored calls, hands them to a pinned pipeline, and imports the resulting report
back into your workspace. Then it exits.
Every input to the run is derived on the server: the workspace, the credentials, the window, the sample, the candidates, the paths. There is no command an operator or a browser can supply, which is what keeps a run from being a way to ask the analysis infrastructure to do something it was not designed to do.
The run’s phases are queued, exporting, analyzing, importing, and then
either completed or failed. A run whose task dies is reconciled to failed
and its temporary read key revoked, rather than sitting active forever.
The profile is fixed
Section titled “The profile is fixed”You do not tune an analysis, and neither does support. The deployment owns a versioned profile, and every run snapshots it verbatim so a report can always be read against the settings that produced it.
| Setting | Value |
|---|---|
| Capture window | The last four completed UTC days |
| Samples per workload | 100 traces |
| Workloads per run | The three eligible workloads with the most samples |
| Objective | Lower cost at the same or better quality |
| Quality floor | Candidate quality at least 0.9 of the baseline |
| Safe switch floor | At least 0.9 of graded samples must be safe |
| Cost ceiling | Strictly below the baseline. Equal cost is not an opportunity |
| Candidates | A fixed list, named in full before you start the run |
This is a deliberate refusal of a knob. A tunable objective is a tunable conclusion, and a report you could have configured into existence is not evidence.
The consequences are worth internalizing. The four-day window means traffic from
this morning is not in it. The 100-sample cap means a workload with 40,000 calls
and a workload with 400 are both judged on 100 replayed cases, so the sample is
the unit of confidence, not your total volume. And the top-three rule means a
run has a scope of three named workloads, which is another reason
naming routes matters: unnamed traffic scattered
across template: groups may not produce three eligible workloads at all.
What replay actually does
Section titled “What replay actually does”Replay takes a captured request and sends it to a candidate model. The candidate’s answer is then compared against the answer your own system actually produced, not against a reference answer written by someone else and not against another candidate.
That is the whole reason the hosted service stores content. Every graded case carries six things: the sample identifier, the request, the model that produced your recorded answer, that recorded answer, the candidate model, and the candidate’s answer. Without stored content, four of those six do not exist and there is nothing to replay.
Your baseline is therefore your own production behaviour on your own inputs. A candidate does not have to be good in the abstract. It has to be as good as what you are already shipping, on the traffic you are actually serving.
Equivalence grading
Section titled “Equivalence grading”Each replayed case gets one of four letter grades for how the candidate’s answer compares to the recorded one:
| Grade | Meaning | Counts as |
|---|---|---|
| A | Equivalent | Safe |
| B | Acceptably different | Safe |
| C | Worse in a way that matters | Unsafe |
| F | Wrong, or failed | Unsafe |
The safe rate is the share of graded samples that came back A or B. The objective’s safe switch floor is applied to this. Grades are recorded per case with the judge’s reason attached, so a suspicious aggregate can always be taken apart into the individual cases that produced it.
What “confidence” is made of
Section titled “What “confidence” is made of”There is no single confidence percentage on a recommendation, and you should be suspicious of any product that offers you one. What a report gives you instead is four separable things, each of which you can disagree with independently.
Grade counts. The full A/B/C/F distribution, not just the summary. Twenty A grades and two F grades is a different situation from twenty-two B grades, even though both produce a safe rate above the floor.
Measured effects as ratios. Cost, latency and quality are reported relative to your current model, from the replay, rather than as absolute numbers taken from a vendor’s pricing page.
Evidence coverage. How many cases were evaluated, and how many were complete. A 100% safe rate over six cases is a much weaker claim than a 94% safe rate over ninety, and the report makes you able to see the difference.
A smoothed workload grade. Metergraph derives a letter grade for a workload from its unsafe counts, using a Laplace-smoothed estimate rather than the raw rate:
unsafe_rate = (unsafe + 1) / (evaluated + 2)
<= 0.05 -> A<= 0.10 -> B<= 0.25 -> C > 0.25 -> FThe smoothing is the interesting part. A raw rate lets three clean samples out of three report a perfect 0% unsafe rate, which is not what three samples can tell you. Smoothing means a small sample cannot earn a top grade, and the grade improves as evidence accumulates rather than being maximal from the first observation. A workload with nothing evaluated gets no letter at all, which is an honest statement of ignorance rather than a grade it did not earn.
What a recommendation contains
Section titled “What a recommendation contains”Recommendations are typed. The report registry knows five kinds: model swap, search-enabled stack swap, prompt context reduction, result caching and batch-first. Which of them a deployment actually looks for is part of the fixed profile, and the confirmation screen names them before you start the run, so you always know what a run was and was not looking at.
Each recommendation carries the workload it applies to, the baseline and the candidate, the measured effects, the grade distribution and safe switch rate, a projection of the saving over the population the sample came from, the evidence coverage, and the replayed samples themselves.
It arrives with one of two statuses:
- Recommended. The evidence clears the objective.
- Recommended pending validation. The evidence points the same way but is not strong enough to stand on its own. Try it on your own traffic before committing.
Options that were evaluated and rejected are recorded as not_recommended,
inconclusive or not_applicable in the report’s decision record. They are not
surfaced as recommendations, because a rejected option is not a suggestion. They
are the reason you can tell a candidate was considered rather than missed.
Why an analysis may return nothing
Section titled “Why an analysis may return nothing”A completed run lands on exactly one of four outcomes, and three of them contain no recommendation.
| Outcome | What it means | Is something wrong? |
|---|---|---|
| Opportunities | At least one swap beat your current setup | No |
| No opportunities | Workloads were evaluated, candidates were replayed and graded, and none won | No. This is a result |
| Partial coverage | Some workloads or candidates did not complete | Yes, retry |
| No eligible workloads | Nothing had enough samples to analyze | No, but you have work to do |
The outcome is derived, not asserted: findings win outright; otherwise an incomplete evaluation is reported as partial coverage; otherwise a run that evaluated workloads and found nothing is reported as such; and a run that evaluated nothing at all says that instead.
“No opportunities” is the outcome most worth understanding. It means the engine replayed your traffic against every candidate, graded the answers, and concluded that your current models are already the cheaper choice at the quality you need. That is a positive finding. It is the evidence you would otherwise have to generate by hand to justify not switching, and most teams never get it.
A run can also fail before it produces any outcome at all. The common cause is pricing: if the window contains a model the catalog cannot price, the run fails rather than reporting a saving it cannot compute. See the model catalog.
What the analysis costs you
Section titled “What the analysis costs you”The analysis makes model calls of its own, for case selection, evaluation
construction and grading. Where those go is stated by configuration, never
inferred from which credentials happen to be present. The control channel is one
of vercel-ai-gateway (the default), anthropic-api or openai-api.
On the hosted cloud that is Metergraph’s problem. On a deployment you run yourself, those calls are made from inside your own infrastructure with your own key, and they reach that provider and no one else.