# kindgi eval-suites

> Eval suites and test sets built from judgments (list / show / from-judgments / cases)

## `kindgi eval-suites`

**Synopsis**

```sh signature
kindgi eval-suites <command>
```

Eval suites and test sets built from judgments (list / show / from-judgments / cases).

**Commands**

| Command                                                                                                                              | Description                                                                                                                                          |
| ------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`kindgi eval-suites list`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites-list)                     | List eval suites (the latest version of each).                                                                                                       |
| [`kindgi eval-suites show`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites-show)                     | Show an eval suite: its latest version, or the one named by `--suite-version`.                                                                       |
| [`kindgi eval-suites from-judgments`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites-from-judgments) | Build a test set: publish a `judged` suite version whose cases are copies of an agent's or flow's judged runs, with each item's judgments summed up. |
| [`kindgi eval-suites cases`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites-cases)                   | List the cases of a judged suite version.                                                                                                            |

**See also**

- [Build a test set from judgments](https://docs.kindgi.com/v0.1.8/guides/evals/build-a-test-set.md)

### `kindgi eval-suites list`

**Synopsis**

```sh signature
kindgi eval-suites list [--project=<id>] [--kind=accuracy|pairwise|regression|human-review|benchmark|custom|judged] [--include-retired] [--limit=<n>] [--cursor=<c>]
```

List eval suites (the latest version of each).

**Options**

| Option              | Default              | Description                                                                                             |
| ------------------- | -------------------- | ------------------------------------------------------------------------------------------------------- |
| `--cursor <value>`  | none                 | Resume after this cursor, from the previous page's `nextCursor`.                                        |
| `--include-retired` | off                  | Include retired suites (every version unregistered), each as its highest version with `unregisteredAt`. |
| `--kind <value>`    | none                 | Only this kind: accuracy, pairwise, regression, human-review, benchmark, custom, judged.                |
| `--limit <value>`   | `25` (at most `100`) | The most to return (default 25, at most 100).                                                           |
| `--project <value>` | none                 | Only suites in this project.                                                                            |

Every command also takes the [global flags](https://docs.kindgi.com/v0.1.8/reference/cli.md#global-flags).

**Examples**

List the judged suites of a project.

```sh
kindgi eval-suites list --project=<project-id> --kind=judged
```

List suites including retired ones.

```sh
kindgi eval-suites list --include-retired
```

**See also**

- [Build a test set from judgments](https://docs.kindgi.com/v0.1.8/guides/evals/build-a-test-set.md)
- [`kindgi eval-suites`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites)

### `kindgi eval-suites show`

**Synopsis**

```sh signature
kindgi eval-suites show <suite-id> [--suite-version=<semver>]
```

Show an eval suite: its latest version, or the one named by `--suite-version`.

**Options**

| Option                    | Default            | Description                               |
| ------------------------- | ------------------ | ----------------------------------------- |
| `--suite-version <value>` | the latest version | The version to show. Default: the latest. |

Every command also takes the [global flags](https://docs.kindgi.com/v0.1.8/reference/cli.md#global-flags).

**Examples**

Show a suite's latest version.

```sh
kindgi eval-suites show acme.article-picks
```

Show one version of a suite.

```sh
kindgi eval-suites show acme.article-picks --suite-version=1.0.0
```

**See also**

- [Build a test set from judgments](https://docs.kindgi.com/v0.1.8/guides/evals/build-a-test-set.md)
- [`kindgi eval-suites`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites)

### `kindgi eval-suites from-judgments`

**Synopsis**

```sh signature
kindgi eval-suites from-judgments <suite-id> --suite-version=<semver> --project=<id> (--agent=<id> [--agent-version=<v>] | --flow=<id>) [--since=<iso>] [--until=<iso>] [--class=<judge-class-id> ...] [--min-judgments=<n>] [--segment=<key:value> ...] [--description=<text>]
```

Build a test set: publish a `judged` suite version whose cases are copies of an agent's or flow's judged runs, with each item's judgments summed up.

**Options**

| Option                    | Default                              | Description                                                                                                                                                  |
| ------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--agent <value>`         | required: one of `--agent`, `--flow` | Use judged runs of this agent.                                                                                                                               |
| `--agent-version <value>` | none                                 | Only runs of this agent version. Needs `--agent`.                                                                                                            |
| `--class <value>`         | none                                 | Count only judgments under this judge class id. Repeat the flag for several classes. Repeatable.                                                             |
| `--description <value>`   | none                                 | A description for the suite version.                                                                                                                         |
| `--flow <value>`          | required: one of `--agent`, `--flow` | Use judged runs of this flow.                                                                                                                                |
| `--min-judgments <value>` | `1`                                  | Leave out runs with fewer counted judgments (default 1).                                                                                                     |
| `--project <value>`       | required                             | The project whose judged runs to use. Required.                                                                                                              |
| `--segment <value>`       | none                                 | Only runs started in this segment or below it, as key:value; repeat it for a path, coarse to fine (`--segment=company:acme --segment=role:cfo`). Repeatable. |
| `--since <value>`         | none                                 | Only runs first judged at or after this time: an ISO 8601 time with a zone, or a date (its start, UTC).                                                      |
| `--suite-version <value>` | required                             | The version to publish. Required.                                                                                                                            |
| `--until <value>`         | none                                 | Only runs first judged before this time: an ISO 8601 time with a zone, or a date (its start, UTC).                                                           |

Every command also takes the [global flags](https://docs.kindgi.com/v0.1.8/reference/cli.md#global-flags).

**Examples**

From [Build a test set from judgments](https://docs.kindgi.com/v0.1.8/guides/evals/build-a-test-set.md):

```sh
pnpm exec kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-picker
pnpm exec kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \
```

**See also**

- [Build a test set from judgments](https://docs.kindgi.com/v0.1.8/guides/evals/build-a-test-set.md)
- [`kindgi eval-suites`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites)

### `kindgi eval-suites cases`

**Synopsis**

```sh signature
kindgi eval-suites cases <suite-id> --suite-version=<semver> [--limit=<n>] [--cursor=<c>]
```

List the cases of a judged suite version.

**Options**

| Option                    | Default              | Description                                                      |
| ------------------------- | -------------------- | ---------------------------------------------------------------- |
| `--cursor <value>`        | none                 | Resume after this cursor, from the previous page's `nextCursor`. |
| `--limit <value>`         | `25` (at most `100`) | The most to return (default 25, at most 100).                    |
| `--suite-version <value>` | required             | The suite version. Required.                                     |

Every command also takes the [global flags](https://docs.kindgi.com/v0.1.8/reference/cli.md#global-flags).

**Examples**

List the first page of a test set's cases.

```sh
kindgi eval-suites cases acme.article-picks --suite-version=1.0.0
```

Fetch the next page of cases.

```sh
kindgi eval-suites cases acme.article-picks --suite-version=1.0.0 --limit=50 --cursor=<cursor>
```

From [Build a test set from judgments](https://docs.kindgi.com/v0.1.8/guides/evals/build-a-test-set.md):

```sh
pnpm exec kindgi eval-suites cases acme.article-picks --suite-version=1.0.0
```

**See also**

- [Build a test set from judgments](https://docs.kindgi.com/v0.1.8/guides/evals/build-a-test-set.md)
- [`kindgi eval-suites`](https://docs.kindgi.com/v0.1.8/reference/cli/eval-suites.md#kindgi-eval-suites)