Operations

Everything after the first boot: creating workspaces and keys, adding people, proving tenant isolation still holds, taking a backup, upgrading, and deleting a tenant for good.

This page covers the two stacks that ship the operator CLI: the customer-local bundle and your own AWS account. The open source server has no CLI and no tenants. Its whole administrative surface is the MG_TOKENS list you start it with.

Running a command

The CLI has no console script. It is a module, always invoked as python -m metergraph_app.cli, and it needs the database owner role, so it runs in a container that has one.

On the customer-local stack, use a one-shot bootstrap container. That service already carries the owner credentials and the LocalStack endpoints, and --no-deps keeps it from restarting anything that is already up. This is the same path bin/demo-ingest uses.

docker compose --env-file .env --env-file .runtime.env --env-file .images.env \
  -f docker-compose.yml run --rm --no-deps -T bootstrap \
  python -m metergraph_app.cli list-tenants

All three --env-file arguments are required, in that order. Drop .images.env and Compose has no image reference at all; drop .runtime.env and the least-privilege database password is missing.

On AWS, the same commands run as a container override on the admin task definition. Terraform emits its ARN as migration_admin_task_definition_arn, and the subnets and security groups as admin_network_configuration. The container is named admin. There is a worked example on Your own AWS account.

The examples below are written as the bare command. Wrap each one in whichever of those two forms applies to you.

The commands

Command	What it does	Where it runs
list-tenants	Every tenant, with its active key count	Anywhere
create-tenant SLUG	Migrates, creates the tenant, mints an initial ingest key	Anywhere
issue-key SLUG	Mints one more key for an existing tenant	Anywhere
invite-user SLUG EMAIL	Adds an email-identity member	Local and hosted
bootstrap-workspace SLUG	Creates a workspace and its exact OIDC owner	BYOC
grant-member SLUG	Grants an exact OIDC identity a role	BYOC
set-platform-admin EMAIL	Grants or revokes global catalog access	Hosted
bootstrap-local	Creates or updates the local password administrator	Local profile only
demo-ingest-key issue or revoke	Mints or revokes the disposable demo credential	Local profile only
setup-local	Migrates and aligns the Metergraph development workspace	Development checkouts only
verify-rls	Proves row-level tenant isolation on the live database	Anywhere
verify-ingest --url URL	Proves the ingest path end to end	Anywhere
purge-tenant SLUG --confirm SLUG	Permanently removes a tenant everywhere	Anywhere
verify-tenant-purge --url URL	Proves a purge really removed live resources	Hosted and BYOC

bootstrap-local and demo-ingest-key check METERGRAPH_DEPLOYMENT_PROFILE before they read a password or touch anything, and exit 2 on any other profile. They cannot be pointed at a hosted deployment by accident.

Workspaces and members

Create a workspace

python -m metergraph_app.cli create-tenant acme --name "Acme Corp"

It runs migrations first, then creates the tenant and mints one initial key. The token is printed once, and only its hash is kept. Save it before you close the terminal. If the slug already exists the command exits 1 and changes nothing.

--zero-raw-retention makes the tenant delete raw objects as soon as the normalized rows commit.

Add people

Which command you use depends on how the deployment authenticates.

On the customer-local stack, sign-in is an email and password, and the admin was created by bootstrap-local from .env. To change that account, edit .env and run ./bin/start again: it re-runs bootstrap, which updates the administrator in place.

python -m metergraph_app.cli invite-user acme person@example.com \
  --name "Person Name" --role member

invite-user upserts the user by email and sets their role, so running it again is how you change someone’s role. Roles are owner, member and viewer.

On AWS, identity comes from Cognito and email grants nothing. Use bootstrap-workspace for the first owner and grant-member afterwards, both keyed on the immutable (identity_issuer, identity_subject) pair. See Your own AWS account.

Keys

python -m metergraph_app.cli issue-key acme --label "checkout service"

The token is printed and never stored in the clear. Scopes default to ingest and read, which is what the SDK needs. --scope is repeatable and accepts ingest, read, agent:read and agent:replay.

Note

agent:read is the MCP server scope. Workspace owners and members can create coding-agent connections in Keys; operator commands are a separate administrative path. BYOC core includes the Agent Access API, but withholds retained trace content and replay. Confirm client runtime reachability in Connect a coding agent.

Ingest keys created in the dashboard are ingest only. Coding-agent connections use separate scopes. Everything about key format and scope is on Identity model.

Proving the deployment still works

Tenant isolation

python -m metergraph_app.cli verify-rls

This is a live test, not a schema inspection. It creates two throwaway tenants with a batch, a key, a user and a membership each, connects as the least-privilege runtime role, and asserts two things: that the role has none of rolsuper, rolcreaterole, rolbypassrls or rolinherit, and that with no tenant context set it can see zero tenants, zero batches and zero keys. It needs METERGRAPH_RUNTIME_DB_PASSWORD, which the bootstrap container already has. Run it after any upgrade that touched migrations.

The ingest path

python -m metergraph_app.cli verify-ingest --url https://your-host --timeout 120

Add --zero-raw-retention when the tenant under test has it set, so the check expects the raw object to be gone.

The stack as a whole, on customer-local

./bin/status prints a JSON readiness report from inside the api container: API health, the deployment profile, the LocalStack bucket and queue, whether the analytics store is serving, analysis-run counts, and the workspace ID. It exits 0 when every check passes, 3 when the only failed check is the analytics store still catching up (see Upgrading), and 1 otherwise. ./bin/verify runs that and then sends one demo trace through the real ingestion API with a disposable key that is revoked either way. Neither reads trace content, and both work before any provider key is set.

On AWS, the status_command Terraform output runs scripts/byoc_status.py. It prints the same report shape over the private hostname with no token. The smoke_test_command output is the AWS equivalent of ./bin/verify’s demo trace.

Backup

Your data is in Postgres. On the customer-local stack that is the pgdata volume, holding database metergraph owned by role metergraph, and nothing else in the stack is stateful in a way you need to keep: LocalStack holds only raw ingested payloads, which are already normalized into the database.

docker compose --env-file .env --env-file .runtime.env --env-file .images.env \
  -f docker-compose.yml exec -T postgres \
  pg_dump -U metergraph metergraph | gzip > metergraph-$(date +%F).sql.gz

Caution

The volume also holds the database password it was created with, and that password is not in the dump. Keep the .env that goes with a backup, or you will restore into a stack that cannot connect to it.

On AWS, RDS handles this: point-in-time recovery is on, and backup_retention_days controls the window. Deletion protection defaults on, so an ordinary terraform destroy leaves the database, the buckets and the key in place unless break_glass_allow_destroy is set.

Upgrading

Customer-local. Download the new bundle and manifest from the releases page, verify them the way Verifying a release describes, extract the bundle over the old directory, and run ./bin/start. It resolves the new digest-pinned images and bootstrap re-runs migrations without touching your data. Your .env is not part of the bundle, so it survives.

Caution

Leave METERGRAPH_DB_PASSWORD exactly as it was. The existing volume holds the password it was created with, and a new value in .env only stops bootstrap from connecting.

Log in to the registry again first if your pull credential has rotated.

The trace summary, performance, usage, environment and route views read only the analytics store, a copy of your call metadata in the stack’s clickhouse service. Upgrading from a release without it, or with a copy that is behind, leaves those views showing a busy message until the worker has copied your history and a nightly pass of its reconciler has found the copy complete. Ingestion and the rest of the dashboard work throughout, and ./bin/start finishes with a note rather than waiting. The worker copies history in passes of up to 500 tenant-hours (an hour of one workspace’s history that holds traces), each starting 5 seconds after the last, so allow about 2 minutes per 500 tenant-hours. ./bin/status shows the progress in its analytics_store check, for example catching up, 777 tenant-hour(s) deferred, and passes once the views are back.

With a long history, copy it in one go instead, then run the nightly pass at once. Both commands run in the worker container, the only one with the store’s write credentials:

docker compose --env-file .env --env-file .runtime.env --env-file .images.env \
  -f docker-compose.yml exec worker python -m metergraph_app.cli \
  rebuild-analytics-store --all-tenants --since 400d --chunk-hours 24 --sleep-seconds 0
docker compose --env-file .env --env-file .runtime.env --env-file .images.env \
  -f docker-compose.yml exec worker python -m metergraph_app.cli \
  reconcile-analytics-store --nightly

--since must reach back past your oldest trace; the rebuild’s last line then says "populated": true. The reconcile exits 0 once the pass is clean, and the views come back within a minute. If it reports that another reconcile pass holds the lock, run it again a minute later. The bundle’s README.md, under Upgrading, has the timings, the query that lists each pass, and what happens to traces that arrive while the clickhouse service is stopped.

AWS. Upgrades change image and analysis_runner_image to new digest-pinned references and re-apply. Both variables refuse anything that is not @sha256: followed by 64 hex characters, so an upgrade cannot silently follow a moving tag. The migration runs as the admin task, not as part of the service deployment. Your release package carries the version-specific order.

After any upgrade, verify-rls and verify-ingest are the two checks worth repeating.

Deleting a tenant

python -m metergraph_app.cli purge-tenant acme --confirm acme

Danger

This is permanent and there is no undo. --confirm must repeat the slug exactly, or the command exits 2 before touching anything.

The purge is ordered so that nothing can slip through behind it. The tenant is marked purging_at first, which fails new authorization closed. Then a shared advisory lock waits out any API or worker operation that was already authorized when the mark committed. Only then does deletion run: every object version under the tenant’s prefix, then the database rows and any dashboard user left with no other membership, in one transaction.

It is resumable, so running it again after an interruption is safe. It prints a JSON summary naming exactly what it removed, including the object-version count and the number of orphaned users deleted.

Vector indexes, inference profiles and Cognito users are hosted-only resources. Under byoc-core those steps are skipped, because the deployment never created them.

To prove a purge on a deployment that had live cloud resources, verify-tenant-purge creates real resources, purges them, and asserts they are gone:

python -m metergraph_app.cli verify-tenant-purge --url https://your-host \
  --timeout 240

The development-only commands

Two commands exist for working on Metergraph itself, not for running it.

setup-local migrates and aligns the dev-local workspace in a development checkout. It refuses before migrating or connecting unless every local-only safety condition holds, and it never seeds reports or touches Docker. seed-dev-reports and seed-dev-traces load fixture data into that same workspace.

Neither belongs on a machine holding real traffic.