kindgi test-runs
kindgi test-runs
Section titled “kindgi test-runs”Synopsis
kindgi test-runs <command>Test runs: run a test set, or try an agent version on the test cases (start / show / list / cancel / rescore).
Commands
| Command | Description |
|---|---|
kindgi test-runs start |
Start a test run. |
kindgi test-runs show |
Show a test run: its status and, once done, its summary. |
kindgi test-runs list |
List test runs, newest first. |
kindgi test-runs cancel |
Cancel a test run that is pending or running. |
kindgi test-runs rescore |
Rescore a completed run on the test cases: a new test run that replays nothing and scores the run's replays again, with what people corrected on them since (correct a changed answer on its replay, then rescore). |
See also
kindgi test-runs start
Section titled “kindgi test-runs start”Synopsis
kindgi test-runs start <test-set-id> --project=<id> (--agent=<id> [--agent-version=<v>] | --flow=<id> [--flow-version=<v>] [--with=<id>@<version> ...]) [--baseline=recorded|<agentId>@<version>|live] [--baseline-project=<id>] [--baseline-segment=<key>:<value> ...] [--reads=recorded|live] [--repetitions=<n>] [--k=<n>] [--group-weights=as-recorded|experts-only] [--dry-run] [--wait]Start a test run. On test cases (a corrections test set), the run tries a version (--agent with --agent-version, or --flow with --flow-version) against the recorded runs: each case is replayed without doing anything the past run didn't, and the result's summary has the metrics.
Options
| Option | Default | Description |
|---|---|---|
--agent <value> |
required: one of --agent, --flow |
Run this agent. |
--agent-version <value> |
the latest version | The agent version to run. Needs --agent. Default: the latest. |
--baseline <value> |
recorded |
What a run on the test cases tries the agent version against: recorded (the default: each case's output as it was marked right or wrong), <agentId>@<version> (another version, replayed the same way) or pinned (the version pinned in a project or segment). Only recorded runs today. |
--baseline-project <value> |
none | With --baseline=pinned: the project whose pinned version to try it against. |
--baseline-segment <value> |
none | With --baseline=pinned and --baseline-project: the version pinned for this segment path, one <key>:<value> step per flag, coarse to fine (e.g. --baseline-segment=company:acme --baseline-segment=role:buyer). Repeatable. |
--dry-run |
off | Check the request without running anything. |
--flow <value> |
required: one of --agent, --flow |
Run this flow. |
--flow-version <value> |
the latest version | The flow version to run. Needs --flow. Default: the latest. |
--group-weights <value> |
as-recorded |
Which corrections count: as-recorded (the default: each at its corrector group's weight) or experts-only (only corrections of groups restricted to experts; a release gate with onlyExpertGroups needs it). |
--k <value> |
10 |
How many ranked items weighted precision@k looks at (1 to 100, default 10). |
--project <value> |
required | The project the run belongs to. Required. |
--reads <value> |
recorded |
Whether replayed reads use the past run's results when it has them (recorded, the default) or run live. |
--repetitions <value> |
1 |
Run each case this many times (1 to 10, default 1); the summary shows the spread. |
--wait |
off | Wait until the run is done (at most 30 minutes), then print it. Exits 3 when it failed or was cancelled. |
--with <value> |
none | With --flow and --flow-version: run one of the flow's agents or tools at another version, as <id>@<version>, without publishing a new flow version. Repeat the flag for several. Repeatable. |
Every command also takes the global flags.
Examples
From Try a version on the test cases:
pnpm exec kindgi test-runs start agent.acme-ops.article-picker.<project-id> --project=<project-id> --agent=acme-ops.article-picker --agent-version=0.2.0 --waitpnpm exec kindgi test-runs start acme.flow-picks --project=<project-id> --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --waitSee also
kindgi test-runs show
Section titled “kindgi test-runs show”Synopsis
kindgi test-runs show <test-run-id>Show a test run: its status and, once done, its summary.
Options
Every command also takes the global flags.
Examples
Show a test run's status and, once it's done, its summary.
kindgi test-runs show <test-run-id>See also
kindgi test-runs list
Section titled “kindgi test-runs list”Synopsis
kindgi test-runs list [--test-set=<id>] [--status=pending|running|completed|failed|cancelled] [--agent=<id>] [--limit=<n>] [--cursor=<c>]List test runs, newest first.
Options
| Option | Default | Description |
|---|---|---|
--agent <value> |
none | Only runs of this agent. |
--cursor <value> |
none | Resume after this cursor, from the previous page's nextCursor. |
--limit <value> |
25 (at most 100) |
The most to return (default 25, at most 100). |
--status <value> |
none | Only this status: pending, running, completed, failed, cancelled. |
--test-set <value> |
none | Only runs of this test set. |
Every command also takes the global flags.
Examples
List the newest test runs of one test set.
kindgi test-runs list --test-set=acme.article-picksList failed runs of an agent, ten at a time.
kindgi test-runs list --agent=acme-ops.article-picker --status=failed --limit=10See also
kindgi test-runs cancel
Section titled “kindgi test-runs cancel”Synopsis
kindgi test-runs cancel <test-run-id>Cancel a test run that is pending or running.
Options
Every command also takes the global flags.
Examples
Cancel a pending or running test run.
kindgi test-runs cancel <test-run-id>See also
kindgi test-runs rescore
Section titled “kindgi test-runs rescore”Synopsis
kindgi test-runs rescore <test-run-id> [--project=<project-id>] [--wait]Rescore a completed run on the test cases: a new test run that replays nothing and scores the run's replays again, with what people corrected on them since (correct a changed answer on its replay, then rescore). The run rescored stays as it was.
Options
| Option | Default | Description |
|---|---|---|
--project <value> |
none | The run's project, only for a runtime that doesn't record it on the run. |
--wait |
off | Wait until the rescore finishes, and show it. Exits 3 when it failed or was cancelled. |
Every command also takes the global flags.
Examples
Rescore a completed run on the test cases and wait for the new run.
kindgi test-runs rescore <test-run-id> --waitRescore against a runtime that does not record the project on the run.
kindgi test-runs rescore <test-run-id> --project=<project-id>See also