Skip to content
Kindgi

An agent's test set

View as Markdown

An agent's test set is the cases a new version of it is checked on. Every correction of the agent's runs is a case, so correcting runs is how the set grows. You can add a case by hand too, and retire one that no longer applies. You try a new agent version on the set to see whether it does what the corrections asked, before anyone uses it.

There's one test set per agent, read one project at a time: a case belongs to the project its run was in. It's computed when it's read, so there's nothing to build or copy.

Before you start

Section titled “Before you start”
  • An agent with finished runs in a project, on a runtime such as kindgi dev, with Kindgi 0.4's CLI.
  • read on the project to read the test set, when authorization is on. To add or retire cases, publish on the agent as well, or, for an agent more than one project serves, write on the case's project.
  • For the app's half: Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.4's client, and KINDGI_API_URL and KINDGI_API_TOKEN set.

Correct some runs

Section titled “Correct some runs”

An agent's test set is made of its corrections, so correct some runs first. The examples on this page use an agent, acme-ops.article-picker, that picks the help articles answering a customer's question. Its typed answer is a list of articles, each with its own id:

{
  "articles": [
    { "id": "kb-101", "title": "Reset your password" },
    { "id": "kb-102", "title": "Change the email on your account" },
    { "id": "kb-103", "title": "Turn on two-step sign-in" }
  ]
}

Correct each article by its id. Then a correction of kb-101 still applies when a new version returns kb-101, at any place in its list.

Terminal window
pnpm exec kindgi corrections add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes
pnpm exec kindgi corrections add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no
  • <run-id> is the run's id.
  • --item is the article's id, and --pointer where it is in the output.
  • --rank is its place in the list, 0 first.
  • --yes says the article was right there, and --no that it wasn't. A wrong item needs nothing more.

Each call adds the article to your one correction of the run, as one of its items, and prints your correction so far, as Correct a run's answer shows. The clients' corrections.create takes the whole list at once: items: [{ key: 'kb-101', pointer: '/output/articles/0', rank: 0, verdict: 'right' }, …].

In the console, a finished run's Correct this run opens its Corrections page. It lists the answer, and each element of a list in the output, each with Yes and No. Correct a run's output has the rest: reasons, your users' corrections, and corrector groups.

Read the test set

Section titled “Read the test set”

From your code, ask your coding agent: "I want a script that shows my agent's test set in a project." For reference, the script looks like this, with the project's id as its argument:

test-set.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev
const projectId = process.argv[2];
const agentId = 'acme-ops.article-picker';


const set = await kindgi.agents.testSet.get(agentId, projectId);
console.log(`${set.cases.active} cases, ${set.cases.retired} retired`);


const cases = await kindgi.agents.testSet.listCases(agentId, { projectId });
for (const c of cases.data) {
  console.log(c.kind, (c.expect.items ?? []).map((item) => `${item.key} ${item.verdict}`));
}
3 cases, 0 retired
correction [ 'kb-201 right', 'kb-203 wrong', 'kb-202 wrong' ]
correction [ 'kb-101 right', 'kb-102 wrong', 'kb-103 wrong' ]
correction [ 'kb-301 right', 'kb-303 wrong', 'kb-201 wrong' ]

It prints how many cases the set has, then each case's kind and the articles it expects, newest first.

  • The project's id: kindgi projects get-default prints the default project, with its id, and kindgi projects list lists them all. In code, it's kindgi.projects.getDefault(), or kindgi.projects.get_default() in Python.
  • Which corrections are cases: every correction of the agent's runs in the project, except Won't fix and Removed ones. Fixed ones and ones With the owner stay in: that's how a new version that fails a case it passed before is seen. A correction that a later one replaced isn't a case, and neither is one on a comparison's replay.
  • A correction's case has the correction's id; a case added by hand has an id that starts case-.
  • Retired cases come back only with includeRetired: true (include="retired" in Python), and kinds: ['hand'] (kind="hand") lists only the cases added by hand.
  • A test run checks a frozen version of the set: its cases as they were when the run started. frozenVersion (frozen_version in Python) names the newest one, as { testSetId, version }. Its cases come from kindgi.testSets.listCases(testSetId, version) (kindgi.test_sets.list_cases(test_set_id, version) in Python), or GET /v1/test-sets/{testSetId}/versions/{version}/cases. It's absent until a test run has frozen the set.
  • Over HTTP, it's GET /v1/agents/{agentId}/test-set?projectId=… and GET /v1/agents/{agentId}/test-set/cases?projectId=…, with include=retired and kind=correction,hand.

What a case expects

Section titled “What a case expects”

Each case has an expect, with a part for each part its correction gave a verdict on:

{
  "id": "8f2c4e1a-6b0d-4c3e-9a57-2d1f0b7e4c93",
  "kind": "correction",
  "projectId": "…",
  "runId": "…",
  "madeOn": "0.1.0",
  "expect": {
    "items": [
      { "key": "kb-101", "pointer": "/output/articles/0", "rank": 0, "verdict": "right" },
      { "key": "kb-102", "pointer": "/output/articles/1", "rank": 1, "verdict": "wrong" }
    ]
  },
  "addedBy": { "kind": "user", "id": "…" },
  "addedAt": "2026-11-02T09:14:03.000Z",
  "updatedAt": "2026-11-02T09:14:03.000Z"
}
Excerpt
The correction says The case expects
decisions: "wrong" Every shouldCall tool called (with its args where given) and none of mustNotCall, checked exactly
decisions: "right" The same tools the run called, by name, in any order
answer: "wrong" An answer that does what shouldSay or why asks, checked by a judging model
answer: "right" An answer that says what the recorded one said, checked by a judging model
items The ranked items, scored as corrected items: in caseCoverage and weightedPrecisionAtK, and in passRate beside the other parts' checks
  • madeOn is the agent version the corrected run ran.
  • A correction whose words an erasure cleared leaves nothing to check, so it isn't a case.
  • A test run checks each part on every repetition. A judging model checks answers, and its calls cost money. Each part the case expects, its decisions and its answer, is one check in passRate, beside its items. A case that expects only items has no pass: its items count toward the metrics.
  • expert: true marks an expert's case: a correction's case when the correction was made under a restricted expert group, and every case added by hand. A test run that counts only experts' corrections grades only these.

Add a case by hand

Section titled “Add a case by hand”

A case added by hand takes a correction's body on one of the agent's finished runs: what it should have done, said, or ranked. It has no status, isn't sorted, and nothing that counts corrections counts it, so it doesn't start Improve or count toward the expert queue.

await kindgi.agents.testSet.addCase('acme-ops.article-picker', {
  runId,
  items: [{ key: 'kb-301', pointer: '/output/articles/0', rank: 0, verdict: 'right' }],
});
Excerpt

In Python, it's kindgi.agents.test_set.add_case(agent_id, run_id=run_id, items=[...]), and over HTTP, POST /v1/agents/{agentId}/test-set/cases with the same body.

  • The run must be one of the agent's, and finished. A comparison's replay can't be a case.
  • Adding a case takes publish on the agent and read on the run's project. For an agent more than one project serves, it takes write on the run's project instead.
  • A case added by hand is an expert's (expert: true), since adding one takes these rights.

Retire a case

Section titled “Retire a case”

A case that no longer applies, such as one about a tool the agent doesn't have any more, can be retired, with a reason:

await kindgi.agents.testSet.retireCase('acme-ops.article-picker', caseId, 'kb-301 was merged into kb-302');
Excerpt

In Python, it's kindgi.agents.test_set.retire_case(agent_id, case_id, reason="…"), and over HTTP, POST /v1/agents/{agentId}/test-set/cases/{caseId}/retire with { "reason": "…" }.

  • A retired case is kept, and with includeRetired: true it comes back with retired: its reason, who retired it and when. No test run checks it again.
  • There's no un-retire. Retiring a correction's case doesn't change the correction's status.
  • Retiring a case takes what adding one takes, on the case's project. A case retired already is 409 test-case-retired.

Test sets built before

Section titled “Test sets built before”

Kindgi 0.3 built judged eval suites from judged runs, with kindgi eval-suites from-judgments. Such a suite is a test set of the corrections kind now, and nothing builds one. One built before is still read by its id, and its cases with kindgi test-sets cases, and kindgi test-sets list lists it only with --kind=corrections, with the agents' own test sets. The corrections it was built from are cases of their agent's test set.

Next

Section titled “Next”