# kindgi eval-runs > Eval runs: run an eval suite, or compare an agent version on a test set (start / show / list / cancel / rescore) ## `kindgi eval-runs` **Synopsis** ```sh signature kindgi eval-runs ``` Eval runs: run an eval suite, or compare an agent version on a test set (start / show / list / cancel / rescore). **Commands** | Command | Description | | ---------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`kindgi eval-runs start`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs-start) | Start an eval run. | | [`kindgi eval-runs show`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs-show) | Show an eval run: its status and, once done, its summary. | | [`kindgi eval-runs list`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs-list) | List eval runs, newest first. | | [`kindgi eval-runs cancel`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs-cancel) | Cancel an eval run that is pending or running. | | [`kindgi eval-runs rescore`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs-rescore) | Rescore a completed comparison of a test set: a new eval run that replays nothing and scores the run's replays again, with what people judged on them since (judge a changed answer on its replay, then rescore). | **See also** - [Compare a version on a test set](https://docs.kindgi.com/v0.1.9/guides/evals/compare-an-agent-version.md) - [Propose a change, and test it](https://docs.kindgi.com/v0.1.9/guides/evals/propose-a-change.md) ### `kindgi eval-runs start` **Synopsis** ```sh signature kindgi eval-runs start --project= (--agent= [--agent-version=] | --flow= [--flow-version=] [--with=@ ...]) [--baseline=recorded|@|live] [--baseline-project=] [--baseline-segment=: ...] [--reads=recorded|live] [--repetitions=] [--k=] [--class-weights=as-recorded|restricted-only] [--dry-run] [--wait] ``` Start an eval run. On a test set (a `judged` suite), the run compares a version (`--agent` with `--agent-version`, or `--flow` with `--flow-version`) with the recorded runs: each case is replayed without doing anything the past run didn't, and the result's summary has the metrics. **Options** | Option | Default | Description | | ---------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--agent ` | required: one of `--agent`, `--flow` | Run this agent. | | `--agent-version ` | the latest version | The agent version to run. Needs `--agent`. Default: the latest. | | `--baseline ` | `recorded` | What a test set's run compares the agent version with: `recorded` (the default: each case's output as it was judged), `@` (another version, replayed the same way) or `live` (the version live in a project or segment). Only `recorded` runs today. | | `--baseline-project ` | none | With `--baseline=live`: the project whose live version to compare with. | | `--baseline-segment ` | none | With `--baseline=live` and `--baseline-project`: the version live for this segment path, one `:` step per flag, coarse to fine (e.g. `--baseline-segment=company:acme --baseline-segment=role:buyer`). Repeatable. | | `--class-weights ` | `as-recorded` | Which judgments count: `as-recorded` (the default: each at its judge class's weight) or `restricted-only` (only judgments of classes restricted to some judges; a gate policy with `onlyRestrictedClasses` needs it). | | `--dry-run` | off | Check the request without running anything. | | `--flow ` | required: one of `--agent`, `--flow` | Run this flow. | | `--flow-version ` | the latest version | The flow version to run. Needs `--flow`. Default: the latest. | | `--k ` | `10` | How many ranked items weighted precision\@k looks at (1 to 100, default 10). | | `--project ` | required | The project the run belongs to. Required. | | `--reads ` | `recorded` | Whether replayed reads use the past run's results when it has them (`recorded`, the default) or run live. | | `--repetitions ` | `1` | Run each case this many times (1 to 10, default 1); the summary shows the spread. | | `--wait` | off | Wait until the run is done (at most 30 minutes), then print it. | | `--with ` | none | With `--flow` and `--flow-version`: run one of the flow's agents or tools at another version, as `@`, without publishing a new flow version. Repeat the flag for several. Repeatable. | Every command also takes the [global flags](https://docs.kindgi.com/v0.1.9/reference/cli.md#global-flags). **Examples** From [Compare a version on a test set](https://docs.kindgi.com/v0.1.9/guides/evals/compare-an-agent-version.md): ```sh pnpm exec kindgi eval-runs start acme.article-picks --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait pnpm exec kindgi eval-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` **See also** - [Compare a version on a test set](https://docs.kindgi.com/v0.1.9/guides/evals/compare-an-agent-version.md) - [`kindgi eval-runs`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs) ### `kindgi eval-runs show` **Synopsis** ```sh signature kindgi eval-runs show ``` Show an eval run: its status and, once done, its summary. **Options** Every command also takes the [global flags](https://docs.kindgi.com/v0.1.9/reference/cli.md#global-flags). **Examples** From [Propose a change, and test it](https://docs.kindgi.com/v0.1.9/guides/evals/propose-a-change.md): ```sh pnpm exec kindgi eval-runs show ``` **See also** - [Propose a change, and test it](https://docs.kindgi.com/v0.1.9/guides/evals/propose-a-change.md) - [`kindgi eval-runs`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs) ### `kindgi eval-runs list` **Synopsis** ```sh signature kindgi eval-runs list [--suite=] [--status=pending|running|completed|failed|cancelled] [--agent=] [--limit=] [--cursor=] ``` List eval runs, newest first. **Options** | Option | Default | Description | | ------------------ | -------------------- | ----------------------------------------------------------------- | | `--agent ` | none | Only runs of this agent. | | `--cursor ` | none | Resume after this cursor, from the previous page's `nextCursor`. | | `--limit ` | `25` (at most `100`) | The most to return (default 25, at most 100). | | `--status ` | none | Only this status: pending, running, completed, failed, cancelled. | | `--suite ` | none | Only runs of this suite. | Every command also takes the [global flags](https://docs.kindgi.com/v0.1.9/reference/cli.md#global-flags). **Examples** List the newest eval runs of one suite. ```sh kindgi eval-runs list --suite=acme.article-picks ``` List failed runs of an agent, ten at a time. ```sh kindgi eval-runs list --agent=acme-ops.article-picker --status=failed --limit=10 ``` **See also** - [Compare a version on a test set](https://docs.kindgi.com/v0.1.9/guides/evals/compare-an-agent-version.md) - [Propose a change, and test it](https://docs.kindgi.com/v0.1.9/guides/evals/propose-a-change.md) - [`kindgi eval-runs`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs) ### `kindgi eval-runs cancel` **Synopsis** ```sh signature kindgi eval-runs cancel ``` Cancel an eval run that is pending or running. **Options** Every command also takes the [global flags](https://docs.kindgi.com/v0.1.9/reference/cli.md#global-flags). **Examples** Cancel a pending or running eval run. ```sh kindgi eval-runs cancel ``` **See also** - [Compare a version on a test set](https://docs.kindgi.com/v0.1.9/guides/evals/compare-an-agent-version.md) - [Propose a change, and test it](https://docs.kindgi.com/v0.1.9/guides/evals/propose-a-change.md) - [`kindgi eval-runs`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs) ### `kindgi eval-runs rescore` **Synopsis** ```sh signature kindgi eval-runs rescore [--project=] [--wait] ``` Rescore a completed comparison of a test set: a new eval run that replays nothing and scores the run's replays again, with what people judged on them since (judge a changed answer on its replay, then rescore). The run rescored stays as it was. **Options** | Option | Default | Description | | ------------------- | ------- | ------------------------------------------------------------------------ | | `--project ` | none | The run's project, only for a runtime that doesn't record it on the run. | | `--wait` | off | Wait until the rescore finishes, and show it. | Every command also takes the [global flags](https://docs.kindgi.com/v0.1.9/reference/cli.md#global-flags). **Examples** Rescore a completed comparison and wait for the new run. ```sh kindgi eval-runs rescore --wait ``` Rescore against a runtime that does not record the project on the run. ```sh kindgi eval-runs rescore --project= ``` **See also** - [Compare a version on a test set](https://docs.kindgi.com/v0.1.9/guides/evals/compare-an-agent-version.md) - [Propose a change, and test it](https://docs.kindgi.com/v0.1.9/guides/evals/propose-a-change.md) - [`kindgi eval-runs`](https://docs.kindgi.com/v0.1.9/reference/cli/eval-runs.md#kindgi-eval-runs)