kindgi eval-suites
kindgi eval-suites
Section titled “kindgi eval-suites”Synopsis
kindgi eval-suites <command>Signature
Eval suites and test sets built from judgments (list / show / from-judgments / cases).
Commands
| Command | Description |
|---|---|
kindgi eval-suites list |
List eval suites (the latest version of each). |
kindgi eval-suites show |
Show an eval suite: its latest version, or the one named by --suite-version. |
kindgi eval-suites from-judgments |
Build a test set: publish a judged suite version whose cases are copies of an agent's or flow's judged runs, with each item's judgments summed up. |
kindgi eval-suites cases |
List the cases of a judged suite version. |
See also
kindgi eval-suites list
Section titled “kindgi eval-suites list”Synopsis
kindgi eval-suites list [--project=<id>] [--kind=accuracy|pairwise|regression|human-review|benchmark|custom|judged] [--include-retired] [--limit=<n>] [--cursor=<c>]Signature
List eval suites (the latest version of each).
Options
| Option | Default | Description |
|---|---|---|
--cursor <value> |
none | Resume after this cursor, from the previous page's nextCursor. |
--include-retired |
off | Include retired suites (every version unregistered), each as its highest version with unregisteredAt. |
--kind <value> |
none | Only this kind: accuracy, pairwise, regression, human-review, benchmark, custom, judged. |
--limit <value> |
25 (at most 100) |
The most to return (default 25, at most 100). |
--project <value> |
none | Only suites in this project. |
Every command also takes the global flags.
Examples
List the judged suites of a project.
kindgi eval-suites list --project=<project-id> --kind=judgedList suites including retired ones.
kindgi eval-suites list --include-retiredSee also
kindgi eval-suites show
Section titled “kindgi eval-suites show”Synopsis
kindgi eval-suites show <suite-id> [--suite-version=<semver>]Signature
Show an eval suite: its latest version, or the one named by --suite-version.
Options
| Option | Default | Description |
|---|---|---|
--suite-version <value> |
the latest version | The version to show. Default: the latest. |
Every command also takes the global flags.
Examples
Show a suite's latest version.
kindgi eval-suites show acme.article-picksShow one version of a suite.
kindgi eval-suites show acme.article-picks --suite-version=1.0.0See also
kindgi eval-suites from-judgments
Section titled “kindgi eval-suites from-judgments”Synopsis
kindgi eval-suites from-judgments <suite-id> --suite-version=<semver> --project=<id> (--agent=<id> [--agent-version=<v>] | --flow=<id>) [--since=<iso>] [--until=<iso>] [--class=<judge-class-id> ...] [--min-judgments=<n>] [--segment=<key:value> ...] [--description=<text>]Signature
Build a test set: publish a judged suite version whose cases are copies of an agent's or flow's judged runs, with each item's judgments summed up.
Options
| Option | Default | Description |
|---|---|---|
--agent <value> |
required: one of --agent, --flow |
Use judged runs of this agent. |
--agent-version <value> |
none | Only runs of this agent version. Needs --agent. |
--class <value> |
none | Count only judgments under this judge class id. Repeat the flag for several classes. Repeatable. |
--description <value> |
none | A description for the suite version. |
--flow <value> |
required: one of --agent, --flow |
Use judged runs of this flow. |
--min-judgments <value> |
1 |
Leave out runs with fewer counted judgments (default 1). |
--project <value> |
required | The project whose judged runs to use. Required. |
--segment <value> |
none | Only runs started in this segment or below it, as key:value; repeat it for a path, coarse to fine (--segment=company:acme --segment=role:cfo). Repeatable. |
--since <value> |
none | Only runs first judged at or after this time: an ISO 8601 time with a zone, or a date (its start, UTC). |
--suite-version <value> |
required | The version to publish. Required. |
--until <value> |
none | Only runs first judged before this time: an ISO 8601 time with a zone, or a date (its start, UTC). |
Every command also takes the global flags.
Examples
From Build a test set from judgments:
pnpm exec kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-pickerpnpm exec kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \See also
kindgi eval-suites cases
Section titled “kindgi eval-suites cases”Synopsis
kindgi eval-suites cases <suite-id> --suite-version=<semver> [--limit=<n>] [--cursor=<c>]Signature
List the cases of a judged suite version.
Options
| Option | Default | Description |
|---|---|---|
--cursor <value> |
none | Resume after this cursor, from the previous page's nextCursor. |
--limit <value> |
25 (at most 100) |
The most to return (default 25, at most 100). |
--suite-version <value> |
required | The suite version. Required. |
Every command also takes the global flags.
Examples
List the first page of a test set's cases.
kindgi eval-suites cases acme.article-picks --suite-version=1.0.0Fetch the next page of cases.
kindgi eval-suites cases acme.article-picks --suite-version=1.0.0 --limit=50 --cursor=<cursor>From Build a test set from judgments:
pnpm exec kindgi eval-suites cases acme.article-picks --suite-version=1.0.0See also