kindgi eval-runs

`kindgi eval-runs`

Synopsis

kindgi eval-runs <command>

Signature

Eval runs: run an eval suite, or compare an agent version on a test set (start / show / list / cancel / rescore).

Commands

Command	Description
`kindgi eval-runs start`	Start an eval run.
`kindgi eval-runs show`	Show an eval run: its status and, once done, its summary.
`kindgi eval-runs list`	List eval runs, newest first.
`kindgi eval-runs cancel`	Cancel an eval run that is pending or running.
`kindgi eval-runs rescore`	Rescore a completed comparison of a test set: a new eval run that replays nothing and scores the run's replays again, with what people judged on them since (judge a changed answer on its replay, then rescore).

See also

`kindgi eval-runs start`

Synopsis

kindgi eval-runs start <suite-id> --project=<id> (--agent=<id> [--agent-version=<v>] | --flow=<id> [--flow-version=<v>] [--with=<id>@<version> ...]) [--baseline=recorded|<agentId>@<version>|live] [--baseline-project=<id>] [--baseline-segment=<key>:<value> ...] [--reads=recorded|live] [--repetitions=<n>] [--k=<n>] [--class-weights=as-recorded|restricted-only] [--dry-run] [--wait]

Signature

Start an eval run. On a test set (a judged suite), the run compares a version (--agent with --agent-version, or --flow with --flow-version) with the recorded runs: each case is replayed without doing anything the past run didn't, and the result's summary has the metrics.

Options

Option	Default	Description
`--agent <value>`	required: one of `--agent`, `--flow`	Run this agent.
`--agent-version <value>`	the latest version	The agent version to run. Needs `--agent`. Default: the latest.
`--baseline <value>`	`recorded`	What a test set's run compares the agent version with: `recorded` (the default: each case's output as it was judged), `<agentId>@<version>` (another version, replayed the same way) or `live` (the version live in a project or segment). Only `recorded` runs today.
`--baseline-project <value>`	none	With `--baseline=live`: the project whose live version to compare with.
`--baseline-segment <value>`	none	With `--baseline=live` and `--baseline-project`: the version live for this segment path, one `<key>:<value>` step per flag, coarse to fine (e.g. `--baseline-segment=company:acme --baseline-segment=role:buyer`). Repeatable.
`--class-weights <value>`	`as-recorded`	Which judgments count: `as-recorded` (the default: each at its judge class's weight) or `restricted-only` (only judgments of classes restricted to some judges; a gate policy with `onlyRestrictedClasses` needs it).
`--dry-run`	off	Check the request without running anything.
`--flow <value>`	required: one of `--agent`, `--flow`	Run this flow.
`--flow-version <value>`	the latest version	The flow version to run. Needs `--flow`. Default: the latest.
`--k <value>`	`10`	How many ranked items weighted precision@k looks at (1 to 100, default 10).
`--project <value>`	required	The project the run belongs to. Required.
`--reads <value>`	`recorded`	Whether replayed reads use the past run's results when it has them (`recorded`, the default) or run live.
`--repetitions <value>`	`1`	Run each case this many times (1 to 10, default 1); the summary shows the spread.
`--wait`	off	Wait until the run is done (at most 30 minutes), then print it.
`--with <value>`	none	With `--flow` and `--flow-version`: run one of the flow's agents or tools at another version, as `<id>@<version>`, without publishing a new flow version. Repeat the flag for several. Repeatable.

Every command also takes the global flags.

Examples

From Compare a version on a test set:

pnpm exec kindgi eval-runs start acme.article-picks --project=<project-id> --agent=acme-ops.article-picker --agent-version=0.2.0 --wait
pnpm exec kindgi eval-runs start acme.flow-picks --project=<project-id> --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait

See also

`kindgi eval-runs show`

Synopsis

kindgi eval-runs show <run-id>

Signature

Show an eval run: its status and, once done, its summary.

Options

Every command also takes the global flags.

Examples

From Propose a change, and test it:

pnpm exec kindgi eval-runs show <eval-run-id>

See also

`kindgi eval-runs list`

Synopsis

kindgi eval-runs list [--suite=<id>] [--status=pending|running|completed|failed|cancelled] [--agent=<id>] [--limit=<n>] [--cursor=<c>]

Signature

List eval runs, newest first.

Options

Option	Default	Description
`--agent <value>`	none	Only runs of this agent.
`--cursor <value>`	none	Resume after this cursor, from the previous page's `nextCursor`.
`--limit <value>`	`25` (at most `100`)	The most to return (default 25, at most 100).
`--status <value>`	none	Only this status: pending, running, completed, failed, cancelled.
`--suite <value>`	none	Only runs of this suite.

Every command also takes the global flags.

Examples

List the newest eval runs of one suite.

kindgi eval-runs list --suite=acme.article-picks

List failed runs of an agent, ten at a time.

kindgi eval-runs list --agent=acme-ops.article-picker --status=failed --limit=10

See also

`kindgi eval-runs cancel`

Synopsis

kindgi eval-runs cancel <run-id>

Signature

Cancel an eval run that is pending or running.

Options

Every command also takes the global flags.

Examples

Cancel a pending or running eval run.

kindgi eval-runs cancel <run-id>

See also

`kindgi eval-runs rescore`

Synopsis

kindgi eval-runs rescore <run-id> [--project=<project-id>] [--wait]

Signature

Rescore a completed comparison of a test set: a new eval run that replays nothing and scores the run's replays again, with what people judged on them since (judge a changed answer on its replay, then rescore). The run rescored stays as it was.

Options

Option	Default	Description
`--project <value>`	none	The run's project, only for a runtime that doesn't record it on the run.
`--wait`	off	Wait until the rescore finishes, and show it.

Every command also takes the global flags.

Examples

Rescore a completed comparison and wait for the new run.

kindgi eval-runs rescore <run-id> --wait

Rescore against a runtime that does not record the project on the run.

kindgi eval-runs rescore <run-id> --project=<project-id>

See also