Set up alerts

There is nothing to set up. Six detectors run continuously against every route in every environment, on the calls you are already capturing. This page is about what they watch, what makes one fire, and what you do about it.

Alerts live on the Alerts page in the dashboard, and on GET /v1/alerts for a key with the read scope.

The six detectors

Each detector compares a recent window against that route’s own trailing baseline in the same environment, so a route that is expensive by nature does not alert simply for being expensive.

Key	Watches	Fires when	Severity
runaway-spend	Individual call cost against the environment’s cost distribution	At least 2 calls in the window are improbably expensive and together cost at least $5	critical at $25 or more, otherwise warning
cost-drift	Median cost per call, or per unit where a route declares one	The recent median is above the baseline median and the bootstrapped difference clears its floor	warning
failure-spike	Failure rate, using Wilson intervals on both windows	The recent lower bound is above both 15% and the baseline upper bound plus 10 points. Needs at least 20 recent and 100 baseline calls	critical once the lower bound reaches 50%, otherwise warning
retry-loop	Failed calls that share a template hash, session and model	At least 5 such failures with at least 2 distinct request or trace ids	critical at 10 or more, otherwise warning
latency-drift	Latency against the baseline p95	At least 80% of recent calls are above the baseline p95, and the median has both doubled and risen by at least 200 ms. Needs at least 20 recent calls with latency	warning
template-change	The structural template hash of your prompts	A template hash that is not in the baseline appears at least 5 times	info

runaway-spend is additionally corrected for multiple comparisons across all of a workspace’s route and environment streams, so watching many routes does not manufacture a steady drip of false alarms.

template-change is an observation rather than an incident. It tells you a prompt changed, which is usually the explanation for one of the other five. It is never delivered anywhere outside the dashboard.

What makes an alert open, and close

1. A scan matches

   Detectors run on a short interval. The first matching scan does not open an alert: it records the condition as pending.

2. A second scan matches

   A condition has to hold across two consecutive scans before an alert opens. This is what keeps a single unlucky minute out of your inbox.

3. It stays open while the condition holds

   Each further matching scan updates the alert in place: the diagnosis, the window, the severity and the occurrences count. It does not create a second alert. Identity comes from a fingerprint over the detector, the route, the environment and the condition version.

4. Two clean scans close it

   When the condition stops matching for two consecutive scans, the alert is resolved. If it comes back later, the same alert reopens with a new generation number, so the history stays in one place.

Acknowledging

Acknowledging is how you say “seen, and being handled” to the rest of your team. Open the alert on the Alerts page and acknowledge it, or:

curl -X POST https://app.metergraph.dev/v1/alerts/$ALERT_ID/ack \
  -H "Cookie: metergraph_session=..."

Owners and members can acknowledge. A viewer cannot. Only an alert that is currently open can be acknowledged: a resolved one returns 404.

Acknowledging is not snoozing

An acknowledgement is cleared the next time the detector matches the same condition, and the alert returns to open. That is deliberate: an acknowledged alert whose cause is still burning money should not stay quiet. If you want it to stop, fix the route or change what it is doing.

Reading the diagnosis

Every alert carries a diagnosis object with the numbers the detector used, not just a message. Which fields are present depends on the detector, but the useful ones are consistent:

• explanation, one sentence about what was observed
• the recent and baseline values that were compared, and the window lengths
• confidence and the interval or tail probability behind the claim
• persistence, showing how many consecutive scans matched

Start with the comparison, not the title. An alert that says a route’s median cost doubled is only interesting once you know the baseline had 4,000 calls behind it and the recent window had 300.

Delivery outside the dashboard

Slack and GitHub delivery are server-level, not per workspace

Metergraph can post an alert to Slack and open a GitHub issue, but the destination is configured once for the whole server: one webhook URL and one repository, shared by every workspace on it. There is no per-workspace setting, no OAuth connect flow, and no way to point alerts at your own channel or your own repository from the product today.

On the hosted service that means alert delivery goes to Metergraph’s own destination, not yours. Read your alerts on the Alerts page, or poll GET /v1/alerts with a read-scoped key and route them yourself.

If you run Metergraph in your own AWS account or locally, the same two settings are yours to configure, because the server is yours.

Each open alert with a severity above info records a delivery row per sink, and the API returns them under deliveries with a status, an attempt count and an error if one occurred. Delivery is retried up to five times. On the BYOC core profile delivery is skipped by design, and the rows say so.

Polling alerts yourself

Fetch open alerts

curl -s https://app.metergraph.dev/v1/alerts?limit=50 \
  -H "Authorization: Bearer $METERGRAPH_READ_KEY"

The response leads with a count of open and critical alerts, then lists alerts newest first with open ones first. Ingest keys minted in the dashboard carry the ingest scope only and will be refused here. See Manage your workspace.

The dashboard Every page in the dashboard and what each one is for.

See also

• Record real outcomes, so a failure spike can be read against real task success
• Errors