Skip to content
Kindgi

Build test cases from corrections

View as Markdown

This guide gathers an agent's corrected runs into versioned test cases, to test new versions on.

Test cases are a set of past runs that people corrected. Each case is a copy of one run, with the corrections of each item of its output added up. You try a new agent version on them to see whether people would rate its answers higher, before anyone uses it.

Before you start

Section titled “Before you start”
  • An agent with finished runs in a project, on a runtime such as kindgi dev, with Kindgi 0.2's CLI.
  • admin on the project, when authorization is on.
  • For the app's half: Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.2's client, and KINDGI_API_URL and KINDGI_API_TOKEN set.

Correct some runs

Section titled “Correct some runs”

Test cases are made of corrected runs, so correct some first. The examples on this page use an agent, acme-ops.article-picker, that picks the help articles answering a customer's question. Its typed answer is a list of articles, each with its own id:

{
  "articles": [
    { "id": "kb-101", "title": "Reset your password" },
    { "id": "kb-102", "title": "Change the email on your account" },
    { "id": "kb-103", "title": "Turn on two-step sign-in" }
  ]
}

Correct each article by its id. Then a correction of kb-101 still applies when a new version returns kb-101, at any place in its list.

Terminal window
pnpm exec kindgi judgments add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password"
pnpm exec kindgi judgments add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in"
  • <run-id> is the run's id.
  • --item is the article's id, and --pointer where it is in the output.
  • --rank is its place in the list, 0 first.

Each prints its correction, as Correct an item shows.

In the console, a finished run's Correct this run opens its Corrections tab. It lists the answer, and each element of a list in the output, each with Yes and No. Correct a run's output has the rest: reasons, your users' corrections, and judge classes.

Build the test cases

Section titled “Build the test cases”

Build version 1.0.0 of the acme.article-picks test cases from the agent's corrected runs in a project:

Terminal window
pnpm exec kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-picker
  • acme.article-picks is the test cases' id, and --suite-version their version.
  • --project=<project-id> is the project the runs are in, where <project-id> is its id.
  • --agent is the agent whose corrected runs it takes.

It prints the test cases:

{
  "suiteId": "acme.article-picks",
  "version": "1.0.0",
  "kind": "judged",
  "caseCount": 3,
  "truncated": false
}

With authorization on, building test cases under a new id takes admin on the project it names.

From your code, ask your coding agent: "I want a script that builds test cases from my agent's corrected runs, and shows me the cases." For reference, the script looks like this, with the project's id as its argument:

test-set.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev
const projectId = process.argv[2];


// Version 1.0.0 of the test set: copies of the agent's judged runs.
const built = await kindgi.evalSuites.buildFromJudgments('acme.article-picks', {
  version: '1.0.0',
  projectId,
  agentId: 'acme-ops.article-picker',
});
console.log(`${built.caseCount} cases`);


const cases = await kindgi.evalSuites.listCases('acme.article-picks', '1.0.0');
for (const c of cases.data) {
  console.log(c.input, c.items.map((item) => `${item.key} ${item.yes} yes, ${item.no} no`));
}
3 cases
I was charged twice for one order. How do I get my money back? [ 'kb-201 1 yes, 0 no', 'kb-203 0 yes, 1 no', 'kb-202 0 yes, 1 no' ]
I forgot my password and can't sign in. [ 'kb-101 1 yes, 0 no', 'kb-102 0 yes, 1 no', 'kb-103 0 yes, 1 no' ]
My package hasn't arrived yet. Where is it? [ 'kb-301 1 yes, 0 no', 'kb-303 0 yes, 1 no', 'kb-201 0 yes, 1 no' ]

It prints how many cases there are, then each case's input and its items' counts.

  • The project's id: kindgi projects get-default prints the Default project, with its id, and kindgi projects list lists them all. In code, it's kindgi.projects.getDefault(), or kindgi.projects.get_default() in Python.
  • Which runs: every run of the agent in the project with at least one correction. These narrow it:
    • --agent-version: one version's runs;
    • --since and --until: when the run was first corrected;
    • --class: count only corrections of these judge classes;
    • --min-judgments: leave out runs with fewer corrections.
  • Up to 1000 cases. truncated: true says there were more: narrow the runs, or build several sets of test cases.
  • Versions never change. When more runs are corrected, build the next version: the same id and a new --suite-version. A taken version is refused: Eval suite "acme.article-picks" version "1.0.0" is already registered.
  • Building test cases takes admin on the project.

Narrow it to a segment

Section titled “Narrow it to a segment”

--segment=<key:value> keeps the runs started in that segment or below it. Repeat it for a path, coarse to fine. In the clients, it's segments:

Terminal window
pnpm exec kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \
  --agent=acme.rank.scorer --segment=company:acme --description="acme company only"

--description says what the test cases hold. It prints the test cases:

{
  "suiteId": "acme.rank-acme-segment",
  "version": "1.0.0",
  "kind": "judged",
  "caseCount": 20,
  "truncated": false
}

A corrected run keeps the segment path it was started with. A run corrected without one, such as one an older runtime recorded, is in no segment, and narrowed test cases leave it out.

What a case keeps

Section titled “What a case keeps”

To see what each case holds, list the cases:

Terminal window
pnpm exec kindgi eval-suites cases acme.article-picks --suite-version=1.0.0

One of them:

{
  "caseId": "240c0a0b-542b-4cb1-9156-fe7153a4377a",
  "subject": { "kind": "agent", "id": "acme-ops.article-picker", "version": "0.1.0" },
  "input": "I forgot my password and can't sign in.",
  "items": [
    {
      "key": "kb-101",
      "pointer": "/output/articles/0",
      "rank": 0,
      "yes": 1,
      "no": 0,
      "yesWeight": 1,
      "totalWeight": 1,
      "restricted": { "yesWeight": 0, "totalWeight": 0, "judgmentIds": [] },
      "judgmentIds": ["8f2c4e1a-6b0d-4c3e-9a57-2d1f0b7e4c93"],
      "reasons": [{ "verdict": "yes", "reason": "Resets the password" }]
    },
    …
  ],
  "output": { "output": { "articles": [ … ] }, "appended": [ … ], … },
  "context": { … }
}
Excerpt
  • caseId is the id of the corrected run.
  • items has each corrected item: its yes and no counts, the same weighted by judge class, and the reasons people gave. The weighted counts are yesWeight of totalWeight, and a correction with no class weighs 1. restricted has the weights of the restricted corrections alone, such as { "yesWeight": 3, "totalWeight": 3 }. judgmentIds (runtime 0.2.0 or later) names every correction counted in the item, and restricted.judgmentIds the restricted ones. Test cases built before these ids have neither.
  • corrections (runtime 0.2.0 or later) has the run's corrections that say what it should have done, whatever their verdict: each one's calls and notCalled, its judgmentId, its judgeClassId when it had a class, and restricted: true when that class was restricted. A case without them has no corrections.
  • output is the run's output, as it was when people corrected it.
  • context, for an agent turn, is what the turn had before it answered: the conversation so far, up to 200 messages, and what it retrieved. A run on the test cases replays the turn with the same context. Its toolEnv has the env values each tool was sent, by tool id. A read-only tool that run uses live gets those, not the values set now.

In the console

Section titled “In the console”

A project's Test cases section lists its test cases, and its form Build test cases from corrections takes the id, the agent, and optionally one version and the corrections a run needs. Each set of test cases has a page that shows its versions and its cases: each item with its yes and no counts, its weighted yes share, up to two reasons, and a link to the original run.

Next

Section titled “Next”