Skip to content
Kindgi

Build a test set from judgments

View as Markdown

This guide gathers an agent's judged runs into a versioned test set, to test new versions on.

A test set is a set of past runs that people judged. Each case is a copy of one run, with the judgments of each item of its output added up. You compare a new agent version on it to see whether people would rate its answers higher, before anyone uses it.

Before you start

Section titled “Before you start”
  • An agent with finished runs in a project, on a runtime such as kindgi dev, with Kindgi 0.1's CLI.
  • admin on the project, when authorization is on.
  • For the app's half: Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.1's client, and KINDGI_API_URL and KINDGI_API_TOKEN set.

Judge some runs

Section titled “Judge some runs”

A test set is made of judged runs, so judge some first. The examples on this page use an agent, acme-ops.article-picker, that picks the help articles answering a customer's question. Its typed answer is a list of articles, each with its own id:

{
  "articles": [
    { "id": "kb-101", "title": "Reset your password" },
    { "id": "kb-102", "title": "Change the email on your account" },
    { "id": "kb-103", "title": "Turn on two-step sign-in" }
  ]
}

Judge each article by its id. Then a judgment of kb-101 still applies when a new version returns kb-101, at any place in its list.

Terminal window
pnpm exec kindgi judgments add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password"
pnpm exec kindgi judgments add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in"
  • <run-id> is the run's id.
  • --item is the article's id, and --pointer where it is in the output.
  • --rank is its place in the list, 0 first.

Each prints its judgment, as Judge an item shows.

In the console, a finished run's Judge this run opens its Judgments tab. It lists the answer, and each element of a list in the output, each with Yes and No. Judge a run's output has the rest: reasons, your users' judgments, and judge classes.

Build the test set

Section titled “Build the test set”

Build version 1.0.0 of a test set, acme.article-picks, from the agent's judged runs in a project:

Terminal window
pnpm exec kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project=<project-id> --agent=acme-ops.article-picker
  • acme.article-picks is the test set's id, and --suite-version its version.
  • --project=<project-id> is the project the runs are in, where <project-id> is its id.
  • --agent is the agent whose judged runs it takes.

It prints the test set:

{
  "suiteId": "acme.article-picks",
  "version": "1.0.0",
  "kind": "judged",
  "caseCount": 3,
  "truncated": false
}

With authorization on, building a test set under a new id takes admin on the project it names.

From your code, ask your coding agent: "I want a script that builds a test set from my agent's judged runs, and shows me its cases." For reference, the script looks like this, with the project's id as its argument:

test-set.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev
const projectId = process.argv[2];


// Version 1.0.0 of the test set: copies of the agent's judged runs.
const built = await kindgi.evalSuites.buildFromJudgments('acme.article-picks', {
  version: '1.0.0',
  projectId,
  agentId: 'acme-ops.article-picker',
});
console.log(`${built.caseCount} cases`);


const cases = await kindgi.evalSuites.listCases('acme.article-picks', '1.0.0');
for (const c of cases.data) {
  console.log(c.input, c.items.map((item) => `${item.key} ${item.yes} yes, ${item.no} no`));
}
3 cases
I was charged twice for one order. How do I get my money back? [ 'kb-201 1 yes, 0 no', 'kb-203 0 yes, 1 no', 'kb-202 0 yes, 1 no' ]
I forgot my password and can't sign in. [ 'kb-101 1 yes, 0 no', 'kb-102 0 yes, 1 no', 'kb-103 0 yes, 1 no' ]
My package hasn't arrived yet. Where is it? [ 'kb-301 1 yes, 0 no', 'kb-303 0 yes, 1 no', 'kb-201 0 yes, 1 no' ]

It prints how many cases the test set has, then each case's input and its items' counts.

  • The project's id: kindgi projects get-default prints the Default project, with its id, and kindgi projects list lists them all. In code, it's kindgi.projects.getDefault(), or kindgi.projects.get_default() in Python.
  • Which runs: every run of the agent in the project with at least one judgment. These narrow it:
    • --agent-version: one version's runs;
    • --since and --until: when the run was first judged;
    • --class: count only judgments of these judge classes;
    • --min-judgments: leave out runs with fewer.
  • Up to 1000 cases. truncated: true says there were more: narrow the runs, or build several sets.
  • Versions never change. When more runs are judged, build the next version: the same id and a new --suite-version. A taken version is refused: Eval suite "acme.article-picks" version "1.0.0" is already registered.
  • Building a test set takes admin on the project.

Narrow it to a segment

Section titled “Narrow it to a segment”

--segment=<key:value> keeps the runs started in that segment or below it. Repeat it for a path, coarse to fine. In the clients, it's segments:

Terminal window
pnpm exec kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project=<project-id> \
  --agent=acme.rank.scorer --segment=company:acme --description="acme company only"

--description says what the test set holds. It prints the test set:

{
  "suiteId": "acme.rank-acme-segment",
  "version": "1.0.0",
  "kind": "judged",
  "caseCount": 20,
  "truncated": false
}

A judged run keeps the segment path it was started with. A run judged without one, such as one an older runtime recorded, is in no segment, and a narrowed test set leaves it out.

What a case keeps

Section titled “What a case keeps”

To see what each case holds, list the cases:

Terminal window
pnpm exec kindgi eval-suites cases acme.article-picks --suite-version=1.0.0

One of them:

{
  "caseId": "240c0a0b-542b-4cb1-9156-fe7153a4377a",
  "subject": { "kind": "agent", "id": "acme-ops.article-picker", "version": "0.1.0" },
  "input": "I forgot my password and can't sign in.",
  "items": [
    {
      "key": "kb-101",
      "pointer": "/output/articles/0",
      "rank": 0,
      "yes": 1,
      "no": 0,
      "yesWeight": 1,
      "totalWeight": 1,
      "restricted": { "yesWeight": 0, "totalWeight": 0 },
      "reasons": [{ "verdict": "yes", "reason": "Resets the password" }]
    },
    …
  ],
  "output": { "output": { "articles": [ … ] }, "appended": [ … ], … },
  "context": { … }
}
Excerpt
  • caseId is the id of the judged run.
  • items has each judged item: its yes and no counts, the same weighted by judge class, and the reasons people gave. The weighted counts are yesWeight of totalWeight, and a judgment with no class weighs 1. restricted has the weights of the restricted judgments alone, such as { "yesWeight": 3, "totalWeight": 3 }.
  • output is the run's output, as it was judged.
  • context, for an agent turn, is what the turn had before it answered: the conversation so far, up to 200 messages, and what it retrieved. A comparison replays the turn with the same context. Its toolEnv has the env values each tool was sent, by tool id. A read-only tool the comparison runs live gets those, not the values set now.

In the console

Section titled “In the console”

A project's Test sets section lists its test sets, and its form Build a test set from judgments takes the id, the agent, and optionally one version and the judgments a run needs. A test set's page shows its versions and its cases: each item with its yes and no counts, its weighted yes share, up to two reasons, and a link to the original run.

Next

Section titled “Next”