kindgi eval-runs

Eval runs: run an eval suite, or compare an agent version on a test set (start / show / list / cancel / rescore).

`kindgi eval-runs start`

Start an eval run. On a test set (a judged suite), the run compares a version (--agent with --agent-version, or --flow with --flow-version) with the recorded runs: each case is replayed without doing anything the past run didn't, and the result's summary has the metrics.

kindgi eval-runs start <suite-id> --project=<id> (--agent=<id> [--agent-version=<v>] | --flow=<id> [--flow-version=<v>] [--with=<id>@<version> ...]) [--baseline=recorded|<agentId>@<version>|live] [--baseline-project=<id>] [--baseline-segment=<key>:<value> ...] [--reads=recorded|live] [--repetitions=<n>] [--k=<n>] [--class-weights=as-recorded|restricted-only] [--dry-run] [--wait]

Flags:

--agent <value>: Run this agent.
--agent-version <value>: The agent version to run. Needs --agent. Default: the latest.
--baseline <value>: What a test set's run compares the agent version with: recorded (the default: each case's output as it was judged), <agentId>@<version> (another version, replayed the same way) or live (the version live in a project or segment). Only recorded runs today.
--baseline-project <value>: With --baseline=live: the project whose live version to compare with.
--baseline-segment <value>: With --baseline=live and --baseline-project: the version live for this segment path, one <key>:<value> step per flag, coarse to fine (e.g. --baseline-segment=company:acme --baseline-segment=role:buyer). (repeatable)
--class-weights <value>: Which judgments count: as-recorded (the default: each at its judge class's weight) or restricted-only (only judgments of classes restricted to some judges; a gate policy with onlyRestrictedClasses needs it).
--dry-run: Check the request without running anything.
--flow <value>: Run this flow.
--flow-version <value>: The flow version to run. Needs --flow. Default: the latest.
--k <value>: How many ranked items weighted precision@k looks at (1 to 100, default 10).
--project <value>: The project the run belongs to. Required.
--reads <value>: Whether replayed reads use the past run's results when it has them (recorded, the default) or run live.
--repetitions <value>: Run each case this many times (1 to 10, default 1); the summary shows the spread.
--wait: Wait until the run is done (at most 30 minutes), then print it.
--with <value>: With --flow and --flow-version: run one of the flow's agents or tools at another version, as <id>@<version>, without publishing a new flow version. Repeat the flag for several. (repeatable)

`kindgi eval-runs show`

Show an eval run: its status and, once done, its summary.

kindgi eval-runs show <run-id>

`kindgi eval-runs list`

List eval runs, newest first.

kindgi eval-runs list [--suite=<id>] [--status=pending|running|completed|failed|cancelled] [--agent=<id>] [--limit=<n>] [--cursor=<c>]

Flags:

--agent <value>: Only runs of this agent.
--cursor <value>: Resume after this cursor, from the previous page's nextCursor.
--limit <value>: The most to return (default 25, at most 100).
--status <value>: Only this status: pending, running, completed, failed, cancelled.
--suite <value>: Only runs of this suite.

`kindgi eval-runs cancel`

Cancel an eval run that is pending or running.

kindgi eval-runs cancel <run-id>

`kindgi eval-runs rescore`

Rescore a completed comparison of a test set: a new eval run that replays nothing and scores the run's replays again, with what people judged on them since (judge a changed answer on its replay, then rescore). The run rescored stays as it was.

kindgi eval-runs rescore <run-id> [--project=<project-id>] [--wait]

Flags:

--project <value>: The run's project, only for a runtime that doesn't record it on the run.
--wait: Wait until the rescore finishes, and show it.

Every command also takes the global flags.