Build a dataset

A dataset is a sample of your own captured traffic for one route, optionally expanded with synthetic examples, graded by a judge model, and exported as JSONL. It is how you get from “we have production traffic” to “we have an eval set” without hand-picking examples.

Datasets live on the Datasets page in the dashboard.

Before you start

A dataset can only be built from calls whose content was captured. The route needs at least two calls with both a request and a response stored, or the API answers 409 route has fewer than two consented examples. If the route runs with capture_text=False, no amount of traffic will make it eligible. See Keep content private.

The route picker in the dashboard only offers routes that already clear this bar, and shows how many eligible examples each one has.

Build one

1. Choose the route and describe the task

   The task description is the instruction the judge and the generator both work from, so write it as a specification, not a label. Say what a correct answer looks like and what a wrong one looks like. Up to 10,000 characters.

2. Set the size and the synthetic share

   target_size is how many examples you want, from 2 to 1,000. synthetic_target is how many of those may be generated rather than sampled from production, from 0 up to the smaller of 500 and target_size. Production failures seed the synthetic expansion, so it fills gaps around the errors you actually had rather than inventing unrelated cases.

3. Pick a sampling method

   coverage_gap scores each candidate on relevance to the task, novelty against what is already selected, and variability, weighting failures more heavily, then collapses near-duplicates. metadata_stratified spreads the sample across metadata strata instead.

4. Choose a cadence

   0 builds once. Otherwise 24 to 2,160 hours creates a schedule, and each run only covers production since the last successful export. A sparse route accumulates rather than losing the window.

5. Build

   The job is queued immediately and moves through sampling, expanding, judging and exporting to completed. stopped, failed and canceled are the terminal alternatives. A job that finds nothing that meets the confidence policy stops with that as its reason rather than exporting an empty file.

Or drive it from the API

Create a dataset job

curl -X POST https://app.metergraph.dev/v1/datasets \
  -H "Content-Type: application/json" \
  -H "Cookie: metergraph_session=..." \
  -d '{
    "route": "ticket-classifier",
    "name": "Q3 escalation eval",
    "task_description": "Classify an inbound ticket into one of five queues...",
    "target_size": 100,
    "synthetic_target": 25,
    "sampling_method": "coverage_gap",
    "cadence_hours": 168,
    "budget_usd": 5
  }'

Every control, with its bounds:

Field	Default	Range
route, name, task_description	—	Required. Name up to 200 characters, description up to 10,000
target_size	50	2 to 1,000
synthetic_target	10	0 to the smaller of 500 and target_size
cadence_hours	168	0, or 24 to 2,160
sampling_method	coverage_gap	coverage_gap or metadata_stratified
dimensions	intent, error_type, complexity	Up to 20 strings
variability	outcome, cost_band, template_hash	Strings
failure_weight	3	1 to 20
near_duplicate_distance	0.08	0 to 1
judge_sample_rate	0.25	0.01 to 1
confidence_floor	0.70	0 to 1
judge_route	judge	A configured judge route
budget_usd	5	Above 0, up to 1,000

An unrecognised field is a 422 rather than being ignored, so a typo fails loudly. The dashboard form sends the defaults above and exposes the route, name, task, size, synthetic share, sampling method, judge route and cadence.

Semantic coverage needs a vector index that is off by default

coverage_gap uses trace embeddings to find semantic gaps. Trace vector indexing is disabled by default, and when there is no index the engine falls back to metadata stratification and records that it did: the job’s stats report metadata_stratified_no_vector_index rather than pretending it did semantic sampling. If your completed job shows that, the two methods gave you the same thing.

Running jobs

• Cancel a job that has not reached a terminal state. Reserved budget is released back.
• Pause and resume a schedule without touching the jobs it has already produced.
• Each job reports accepted items against total, how many came from production against synthetic, the propagated confidence, and spend against budget.

Owners and members can create, cancel, pause and resume. A viewer can read.

The two export formats

Exports appear once the job’s items clear the confidence policy. Both are newline-delimited JSON, one item per line, downloaded from GET /v1/datasets/{id}/exports/{format}. Any other format name is a 404.

evals_code_jsonl

For an eval harness. Carries the input, the expected output, and the metadata you need to slice results.

evals_code_jsonl (one line, formatted here)

{
  "schema_version": 1,
  "id": "8f3c...",
  "input": { "messages": [{ "role": "user", "content": "..." }] },
  "expected": { "output": "..." },
  "metadata": {
    "origin": "production",
    "dimensions": { "intent": "...", "error_type": "...", "complexity": "..." },
    "variability": { "outcome": "...", "cost_band": "...", "template_hash": "..." },
    "confidence": 0.82,
    "confidence_source": "sampled_judge"
  }
}

fine_tuning_jsonl

The shape a provider fine-tuning endpoint expects: a user turn and the assistant turn it should have produced, plus enough metadata to trace a line back to the item it came from.

fine_tuning_jsonl (one line, formatted here)

{
  "messages": [
    { "role": "user", "content": "..." },
    { "role": "assistant", "content": "..." }
  ],
  "metadata": {
    "metergraph_item_id": "8f3c...",
    "origin": "production",
    "confidence": 0.82
  }
}

origin is production or synthetic. confidence_source is one of production, human_reference, sampled_judge or propagated_judge, which tells you whether that line was judged directly or inherited its confidence. dimensions and variability are objects keyed by the dimension names the job was created with.

Both files contain the same items in the same order. Pick by what you are feeding, not by what the dataset is.

A dataset carries prompt and completion text

Exports contain the real prompts and responses that were captured, and the synthetic examples generated from them. That is the point of a dataset, but it means the export is as sensitive as the traffic it came from. Treat the downloaded file the way you would treat a database export.

Delivery to a repository

A completed dataset can also be delivered as a pull request, and the Datasets page shows the link when there is one. Like alert delivery, that repository is a server-level setting rather than a per-workspace one, so on the hosted service the file you want is the one you download. See Set up alerts.

How analysis works Replay, equivalence grading and confidence.

See also

• Keep content private, which decides whether a route is eligible at all
• Name a route
• The dashboard