Evaluate models with your coding agent

The evaluation loop answers one question: which model or configuration should this workload use, and why. It has six steps: choose a workload, define and test checks, run an analysis, read the report, inspect the evidence, and refine and rerun.

The intended split is that your coding agent does the choosing, defining and refining, because it can read your codebase, while the Metergraph app holds the detailed report and case evidence. Your agent links to the report rather than writing its own.

Before you start

Connect your agent and confirm the workspace. See Connect a coding agent.
Have real application traces in the workspace. Start with Send your first trace. Test, mock and demo traffic does not show how your application behaves.
Install the Metergraph skills, so your agent follows the same loop every time. They are open source at metergraph/skills. In Claude Desktop, open Customize, then Plugins, then Add marketplace, and enter metergraph/skills. In Claude Code, run claude plugin marketplace add metergraph/skills and claude plugin install metergraph@metergraph. For Codex and Cursor, run npx --yes metergraph-cli@latest skills install --client codex --runtime local in your project. Then ask your agent to use metergraph-model-swap.

What each step can do today

Step	In your coding agent	In the Metergraph app
Choose a workload	Read routes, usage and traces to pick one. Reading classified workloads and model readiness is not available yet.	Choose the workload in the analysis flow
Define and test checks	Draft a check and its passing and failing cases from evidence you allow it to read. Saving and testing are not available yet.	Create, test and save checks
Run an analysis	Not available yet. The agent explains the plan, inputs and cost limits for you to approve in the app.	Start the run
Follow the run	Not available yet	Run status
Read the report	List and read reports, with links back to the app. Reading case evidence needs the Debug role.	Full report and case evidence
Refine and rerun	Propose a new revision of a check and what changed. Saving and rerunning are not available yet.	Save the revision and start a new run

The agent decides from what your connection advertises, so a step becomes available in your client once your deployment offers it. Ask the agent to call metergraph_get_capabilities to see the current set.

Authority and cost

Reading evidence and reports needs only the access your connection already has. Reading retained prompts and outputs needs the Debug role.

Running checks with a model judge, or running an analysis, calls a model provider and can cost money. Before that happens, the agent must tell you the selected traces, where data goes, the candidate models and the cost limits, and you must approve it. A read connection, a configured provider key, or a replay is not approval to run an evaluation, and replaying traces is not a substitute for one.

Read a report with your agent

Ask your agent to summarize the latest report for a workload. A useful summary:

separates quality failures from evaluator or execution failures and from checks or cases that were not covered;
names the workload, the candidate model or configuration, the checks that applied, and the cost and latency trade-offs where they were measured;
says whether the evidence supports a change, no change, or more testing;
links to the report in the Metergraph app.

When your codebase shows where that workload’s model is configured, the agent can name the call site and suggest an experiment to validate the change. It does not edit code, open a pull request or roll anything out unless you ask it to separately.

Use the passing and failing cases to check the checks themselves. A grade difference can come from a weak check as easily as from a better or worse model.

Refine without losing history

A refined check is a new revision, and a rerun is a new run. The earlier revision, run and report stay as they were. Compare two runs only when they used comparable traces and checks, and say so when they did not.