# Build test cases from corrections

> Gather an agent's corrected runs into versioned test cases, with what people said about each item, to test new versions on.

This guide gathers an agent's corrected runs into versioned test cases, to test new versions on.

**Test cases** are a set of past runs that people corrected. Each case is a copy of one run, with the corrections of each item of its output added up. You [try a new agent version](https://docs.kindgi.com/v0.3.1/guides/evals/compare-an-agent-version.md) on them to see whether people would rate its answers higher, before anyone uses it.

## Before you start

- **An agent with finished runs** in a project, on a runtime such as `kindgi dev`, with Kindgi 0.2's CLI.
- **`admin` on the project,** when authorization is on.
- **For the app's half:** Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.2's client, and `KINDGI_API_URL` and `KINDGI_API_TOKEN` set.

## Correct some runs

Test cases are made of corrected runs, so correct some first. The examples on this page use an agent, `acme-ops.article-picker`, that picks the help articles answering a customer's question. Its [typed answer](https://docs.kindgi.com/v0.3.1/guides/agents/typed-answer.md) is a list of articles, each with its own id:

```json
{
  "articles": [
    { "id": "kb-101", "title": "Reset your password" },
    { "id": "kb-102", "title": "Change the email on your account" },
    { "id": "kb-103", "title": "Turn on two-step sign-in" }
  ]
}
```

Correct each article by its id. Then a correction of `kb-101` still applies when a new version returns `kb-101`, at any place in its list.

- **TypeScript**

  ```sh
  pnpm exec kindgi judgments add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password"
  pnpm exec kindgi judgments add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in"
  ```

- **Python**

  ```sh
  uv run kindgi judgments add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password"
  uv run kindgi judgments add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in"
  ```

- **Java**

  ```sh
  ./kindgiw judgments add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password"
  ./kindgiw judgments add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in"
  ```

- **Scala**

  ```sh
  ./kindgiw judgments add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password"
  ./kindgiw judgments add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in"
  ```

* `<run-id>` is the run's id.
* `--item` is the article's id, and `--pointer` where it is in the output.
* `--rank` is its place in the list, 0 first.

Each prints its correction, as [Correct an item](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md#correct-an-item) shows.

In the console, a finished run's **Correct this run** opens its Corrections tab. It lists the answer, and each element of a list in the output, each with **Yes** and **No**. [Correct a run's output](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md) has the rest: reasons, your users' corrections, and judge classes.

## Build the test cases

Build version 1.0.0 of the `acme.article-picks` test cases from the agent's corrected runs in a project:

- **TypeScript**

  ```sh
  pnpm exec kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-picker
  ```

- **Python**

  ```sh
  uv run kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-picker
  ```

- **Java**

  ```sh
  ./kindgiw eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-picker
  ```

- **Scala**

  ```sh
  ./kindgiw eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-picker
  ```

* `acme.article-picks` is the test cases' id, and `--suite-version` their version.
* `--project=<project-id>` is the project the runs are in, where `<project-id>` is its id.
* `--agent` is the agent whose corrected runs it takes.

It prints the test cases:

```json
{
  "suiteId": "acme.article-picks",
  "version": "1.0.0",
  "kind": "judged",
  "caseCount": 3,
  "truncated": false
}
```

With [authorization](https://docs.kindgi.com/v0.3.1/deploy/authorization.md) on, building test cases under a new id takes `admin` on the project it names.

From your code, ask your coding agent: "I want a script that builds test cases from my agent's corrected runs, and shows me the cases." For reference, the script looks like this, with the project's id as its argument:

- **TypeScript**

  `test-set.ts`

  ```ts
  import { createClient } from '@kindgi/sdk/client';

  const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev
  const projectId = process.argv[2];

  // Version 1.0.0 of the test set: copies of the agent's judged runs.
  const built = await kindgi.evalSuites.buildFromJudgments('acme.article-picks', {
    version: '1.0.0',
    projectId,
    agentId: 'acme-ops.article-picker',
  });
  console.log(`${built.caseCount} cases`);

  const cases = await kindgi.evalSuites.listCases('acme.article-picks', '1.0.0');
  for (const c of cases.data) {
    console.log(c.input, c.items.map((item) => `${item.key} ${item.yes} yes, ${item.no} no`));
  }
  ```

  ```text
  3 cases
  I was charged twice for one order. How do I get my money back? [ 'kb-201 1 yes, 0 no', 'kb-203 0 yes, 1 no', 'kb-202 0 yes, 1 no' ]
  I forgot my password and can't sign in. [ 'kb-101 1 yes, 0 no', 'kb-102 0 yes, 1 no', 'kb-103 0 yes, 1 no' ]
  My package hasn't arrived yet. Where is it? [ 'kb-301 1 yes, 0 no', 'kb-303 0 yes, 1 no', 'kb-201 0 yes, 1 no' ]
  ```

- **Python**

  `test_set.py`

  ```python
  import sys

  from kindgi.client import Kindgi

  kindgi = Kindgi()  # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev
  project_id = sys.argv[1]

  # Version 1.0.0 of the test set: copies of the agent's judged runs.
  built = kindgi.eval_suites.build_from_judgments(
      "acme.article-picks",
      version="1.0.0",
      project_id=project_id,
      agent_id="acme-ops.article-picker",
  )
  print(f"{built.case_count} cases")

  cases = kindgi.eval_suites.list_cases("acme.article-picks", "1.0.0")
  for case in cases.data:
      print(case.input, [f"{item.key} {item.yes} yes, {item.no} no" for item in case.items])
  ```

  ```text
  3 cases
  I was charged twice for one order. How do I get my money back? ['kb-201 1 yes, 0 no', 'kb-203 0 yes, 1 no', 'kb-202 0 yes, 1 no']
  I forgot my password and can't sign in. ['kb-101 1 yes, 0 no', 'kb-102 0 yes, 1 no', 'kb-103 0 yes, 1 no']
  My package hasn't arrived yet. Where is it? ['kb-301 1 yes, 0 no', 'kb-303 0 yes, 1 no', 'kb-201 0 yes, 1 no']
  ```

It prints how many cases there are, then each case's input and its items' counts.

- **The project's id:** `kindgi projects get-default` prints the Default project, with its `id`, and `kindgi projects list` lists them all. In code, it's `kindgi.projects.getDefault()`, or `kindgi.projects.get_default()` in Python.

- **Which runs:** every run of the agent in the project with at least one correction. These narrow it:

  - `--agent-version`: one version's runs;
  - `--since` and `--until`: when the run was first corrected;
  - `--class`: count only corrections of these [judge classes](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md#weight-corrections-by-judge-class);
  - `--min-judgments`: leave out runs with fewer corrections.

- **Up to 1000 cases.** `truncated: true` says there were more: narrow the runs, or build several sets of test cases.

- **Versions never change.** When more runs are corrected, build the next version: the same id and a new `--suite-version`. A taken version is refused: `Eval suite "acme.article-picks" version "1.0.0" is already registered.`

- **Building test cases takes `admin` on the project.**

### Narrow it to a segment

`--segment=<key:value>` keeps the runs started in that segment or below it. Repeat it for a path, coarse to fine. In the clients, it's `segments`:

- **TypeScript**

  ```sh
  pnpm exec kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \
    --agent=acme.rank.scorer --segment=company:acme --description="acme company only"
  ```

- **Python**

  ```sh
  uv run kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \
    --agent=acme.rank.scorer --segment=company:acme --description="acme company only"
  ```

- **Java**

  ```sh
  ./kindgiw eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \
    --agent=acme.rank.scorer --segment=company:acme --description="acme company only"
  ```

- **Scala**

  ```sh
  ./kindgiw eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \
    --agent=acme.rank.scorer --segment=company:acme --description="acme company only"
  ```

`--description` says what the test cases hold. It prints the test cases:

```json
{
  "suiteId": "acme.rank-acme-segment",
  "version": "1.0.0",
  "kind": "judged",
  "caseCount": 20,
  "truncated": false
}
```

A corrected run keeps the segment path it was started with. A run corrected without one, such as one an older runtime recorded, is in no segment, and narrowed test cases leave it out.

## What a case keeps

To see what each case holds, list the cases:

- **TypeScript**

  ```sh
  pnpm exec kindgi eval-suites cases acme.article-picks --suite-version=1.0.0
  ```

- **Python**

  ```sh
  uv run kindgi eval-suites cases acme.article-picks --suite-version=1.0.0
  ```

- **Java**

  ```sh
  ./kindgiw eval-suites cases acme.article-picks --suite-version=1.0.0
  ```

- **Scala**

  ```sh
  ./kindgiw eval-suites cases acme.article-picks --suite-version=1.0.0
  ```

One of them:

```json excerpt
{
  "caseId": "240c0a0b-542b-4cb1-9156-fe7153a4377a",
  "subject": { "kind": "agent", "id": "acme-ops.article-picker", "version": "0.1.0" },
  "input": "I forgot my password and can't sign in.",
  "items": [
    {
      "key": "kb-101",
      "pointer": "/output/articles/0",
      "rank": 0,
      "yes": 1,
      "no": 0,
      "yesWeight": 1,
      "totalWeight": 1,
      "restricted": { "yesWeight": 0, "totalWeight": 0, "judgmentIds": [] },
      "judgmentIds": ["8f2c4e1a-6b0d-4c3e-9a57-2d1f0b7e4c93"],
      "reasons": [{ "verdict": "yes", "reason": "Resets the password" }]
    },
    …
  ],
  "output": { "output": { "articles": [ … ] }, "appended": [ … ], … },
  "context": { … }
}
```

- **`caseId`** is the id of the corrected run.
- **`items`** has each corrected item: its yes and no counts, the same weighted by judge class, and the reasons people gave. The weighted counts are `yesWeight` of `totalWeight`, and a correction with no class weighs 1. `restricted` has the weights of the [restricted](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md#restrict-who-may-correct-as-a-class) corrections alone, such as `{ "yesWeight": 3, "totalWeight": 3 }`. `judgmentIds` (runtime 0.2.0 or later) names every correction counted in the item, and `restricted.judgmentIds` the restricted ones. Test cases built before these ids have neither.
- **`corrections`** (runtime 0.2.0 or later) has the run's corrections that say [what it should have done](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md#say-what-it-should-have-done), whatever their verdict: each one's `calls` and `notCalled`, its `judgmentId`, its `judgeClassId` when it had a class, and `restricted: true` when that class was restricted. A case without them has no `corrections`.
- **`output`** is the run's output, as it was when people corrected it.
- **`context`**, for an agent turn, is what the turn had before it answered: the conversation so far, up to 200 messages, and what it retrieved. A run on the test cases replays the turn with the same context. Its `toolEnv` has the [env values](https://docs.kindgi.com/v0.3.1/guides/tools/give-a-tool-env-values.md) each tool was sent, by tool id. A read-only tool that run uses live gets those, not the values set now.

## In the console

A project's **Test cases** section lists its test cases, and its form **Build test cases from corrections** takes the id, the agent, and optionally one version and the corrections a run needs. Each set of test cases has a page that shows its versions and its cases: each item with its yes and no counts, its weighted yes share, up to two reasons, and a link to the original run.

## Next

- [Try a version on the test cases](https://docs.kindgi.com/v0.3.1/guides/evals/compare-an-agent-version.md)
- [Correct a run's output](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md)
- [Corrections reference](https://docs.kindgi.com/v0.3.1/guides/evals/judging-reference.md)