# An agent's test set > The cases a new version of an agent is checked on, made from its corrections and the cases you add by hand. An agent's **test set** is the cases a new version of it is checked on. Every correction of the agent's runs is a case, so correcting runs is how the set grows. You can add a case by hand too, and retire one that no longer applies. You [try a new agent version](https://docs.kindgi.com/v0.4.0/guides/improve/compare-an-agent-version.md) on the set to see whether it does what the corrections asked, before anyone uses it. There's one test set per agent, read one project at a time: a case belongs to the project its run was in. It's computed when it's read, so there's nothing to build or copy. ## Before you start - **An agent with finished runs** in a project, on a runtime such as `kindgi dev`, with Kindgi 0.4's CLI. - **`read` on the project** to read the test set, when authorization is on. To add or retire cases, `publish` on the agent as well, or, for an agent more than one project serves, `write` on the case's project. - **For the app's half:** Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.4's client, and `KINDGI_API_URL` and `KINDGI_API_TOKEN` set. ## Correct some runs An agent's test set is made of its corrections, so correct some runs first. The examples on this page use an agent, `acme-ops.article-picker`, that picks the help articles answering a customer's question. Its [typed answer](https://docs.kindgi.com/v0.4.0/guides/agents/typed-answer.md) is a list of articles, each with its own id: ```json { "articles": [ { "id": "kb-101", "title": "Reset your password" }, { "id": "kb-102", "title": "Change the email on your account" }, { "id": "kb-103", "title": "Turn on two-step sign-in" } ] } ``` Correct each article by its id. Then a correction of `kb-101` still applies when a new version returns `kb-101`, at any place in its list. - **TypeScript** ```sh pnpm exec kindgi corrections add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes pnpm exec kindgi corrections add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no ``` - **Python** ```sh uv run kindgi corrections add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes uv run kindgi corrections add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no ``` - **Java** ```sh ./kindgiw corrections add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes ./kindgiw corrections add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no ``` - **Scala** ```sh ./kindgiw corrections add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes ./kindgiw corrections add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no ``` * `` is the run's id. * `--item` is the article's id, and `--pointer` where it is in the output. * `--rank` is its place in the list, 0 first. * `--yes` says the article was right there, and `--no` that it wasn't. A wrong item needs nothing more. Each call adds the article to your one correction of the run, as one of its `items`, and prints your correction so far, as [Correct a run's answer](https://docs.kindgi.com/v0.4.0/guides/improve/correct-a-runs-output.md#correct-a-runs-answer) shows. The clients' `corrections.create` takes the whole list at once: `items: [{ key: 'kb-101', pointer: '/output/articles/0', rank: 0, verdict: 'right' }, …]`. In the console, a finished run's **Correct this run** opens its Corrections page. It lists the answer, and each element of a list in the output, each with **Yes** and **No**. [Correct a run's output](https://docs.kindgi.com/v0.4.0/guides/improve/correct-a-runs-output.md) has the rest: reasons, your users' corrections, and corrector groups. ## Read the test set From your code, ask your coding agent: "I want a script that shows my agent's test set in a project." For reference, the script looks like this, with the project's id as its argument: - **TypeScript** `test-set.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const projectId = process.argv[2]; const agentId = 'acme-ops.article-picker'; const set = await kindgi.agents.testSet.get(agentId, projectId); console.log(`${set.cases.active} cases, ${set.cases.retired} retired`); const cases = await kindgi.agents.testSet.listCases(agentId, { projectId }); for (const c of cases.data) { console.log(c.kind, (c.expect.items ?? []).map((item) => `${item.key} ${item.verdict}`)); } ``` ```text 3 cases, 0 retired correction [ 'kb-201 right', 'kb-203 wrong', 'kb-202 wrong' ] correction [ 'kb-101 right', 'kb-102 wrong', 'kb-103 wrong' ] correction [ 'kb-301 right', 'kb-303 wrong', 'kb-201 wrong' ] ``` - **Python** `test_set.py` ```python import sys from kindgi.client import Kindgi kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev project_id = sys.argv[1] agent_id = "acme-ops.article-picker" test_set = kindgi.agents.test_set.get(agent_id, project_id=project_id) print(f"{test_set.cases.active} cases, {test_set.cases.retired} retired") cases = kindgi.agents.test_set.list_cases(agent_id, project_id=project_id) for case in cases.data: print(case.kind, [f"{item.key} {item.verdict}" for item in case.expect.items or []]) ``` ```text 3 cases, 0 retired correction ['kb-201 right', 'kb-203 wrong', 'kb-202 wrong'] correction ['kb-101 right', 'kb-102 wrong', 'kb-103 wrong'] correction ['kb-301 right', 'kb-303 wrong', 'kb-201 wrong'] ``` It prints how many cases the set has, then each case's kind and the articles it expects, newest first. - **The project's id:** `kindgi projects get-default` prints the default project, with its `id`, and `kindgi projects list` lists them all. In code, it's `kindgi.projects.getDefault()`, or `kindgi.projects.get_default()` in Python. - **Which corrections are cases:** every correction of the agent's runs in the project, except Won't fix and Removed ones. Fixed ones and ones With the owner stay in: that's how a new version that fails a case it passed before is seen. A correction that a later one replaced isn't a case, and neither is one on a [comparison's](https://docs.kindgi.com/v0.4.0/guides/improve/compare-an-agent-version.md) replay. - **A correction's case has the correction's id;** a case added by hand has an id that starts `case-`. - **Retired cases** come back only with `includeRetired: true` (`include="retired"` in Python), and `kinds: ['hand']` (`kind="hand"`) lists only the cases added by hand. - **A test run checks a frozen version** of the set: its cases as they were when the run started. `frozenVersion` (`frozen_version` in Python) names the newest one, as `{ testSetId, version }`. Its cases come from `kindgi.testSets.listCases(testSetId, version)` (`kindgi.test_sets.list_cases(test_set_id, version)` in Python), or `GET /v1/test-sets/{testSetId}/versions/{version}/cases`. It's absent until a test run has frozen the set. - Over HTTP, it's `GET /v1/agents/{agentId}/test-set?projectId=…` and `GET /v1/agents/{agentId}/test-set/cases?projectId=…`, with `include=retired` and `kind=correction,hand`. ## What a case expects Each case has an `expect`, with a part for each part its correction gave a verdict on: ```json excerpt { "id": "8f2c4e1a-6b0d-4c3e-9a57-2d1f0b7e4c93", "kind": "correction", "projectId": "…", "runId": "…", "madeOn": "0.1.0", "expect": { "items": [ { "key": "kb-101", "pointer": "/output/articles/0", "rank": 0, "verdict": "right" }, { "key": "kb-102", "pointer": "/output/articles/1", "rank": 1, "verdict": "wrong" } ] }, "addedBy": { "kind": "user", "id": "…" }, "addedAt": "2026-11-02T09:14:03.000Z", "updatedAt": "2026-11-02T09:14:03.000Z" } ``` | The correction says | The case expects | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `decisions: "wrong"` | Every `shouldCall` tool called (with its `args` where given) and none of `mustNotCall`, checked exactly | | `decisions: "right"` | The same tools the run called, by name, in any order | | `answer: "wrong"` | An answer that does what `shouldSay` or `why` asks, checked by a judging model | | `answer: "right"` | An answer that says what the recorded one said, checked by a judging model | | `items` | The ranked items, scored as corrected items: in `caseCoverage` and `weightedPrecisionAtK`, and in `passRate` beside the other parts' checks | - **`madeOn`** is the agent version the corrected run ran. - A correction whose words an [erasure](https://docs.kindgi.com/v0.4.0/guides/agents/erase-a-persons-data.md) cleared leaves nothing to check, so it isn't a case. - A [test run](https://docs.kindgi.com/v0.4.0/guides/improve/compare-an-agent-version.md#what-a-case-expects) checks each part on every repetition. A judging model checks answers, and its calls cost money. Each part the case expects, its decisions and its answer, is one check in `passRate`, beside its items. A case that expects only `items` has no pass: its items count toward the metrics. - **`expert: true`** marks an expert's case: a correction's case when the correction was made under a restricted expert group, and every case added by hand. A test run that counts only experts' corrections grades only these. ## Add a case by hand A case added by hand takes a correction's body on one of the agent's finished runs: what it should have done, said, or ranked. It has no status, isn't sorted, and nothing that counts corrections counts it, so it doesn't start Improve or count toward the expert queue. ```ts excerpt await kindgi.agents.testSet.addCase('acme-ops.article-picker', { runId, items: [{ key: 'kb-301', pointer: '/output/articles/0', rank: 0, verdict: 'right' }], }); ``` In Python, it's `kindgi.agents.test_set.add_case(agent_id, run_id=run_id, items=[...])`, and over HTTP, `POST /v1/agents/{agentId}/test-set/cases` with the same body. - The run must be one of the agent's, and finished. A comparison's replay can't be a case. - Adding a case takes `publish` on the agent and `read` on the run's project. For an agent more than one project serves, it takes `write` on the run's project instead. - A case added by hand is an expert's (`expert: true`), since adding one takes these rights. ## Retire a case A case that no longer applies, such as one about a tool the agent doesn't have any more, can be retired, with a reason: ```ts excerpt await kindgi.agents.testSet.retireCase('acme-ops.article-picker', caseId, 'kb-301 was merged into kb-302'); ``` In Python, it's `kindgi.agents.test_set.retire_case(agent_id, case_id, reason="…")`, and over HTTP, `POST /v1/agents/{agentId}/test-set/cases/{caseId}/retire` with `{ "reason": "…" }`. - A retired case is kept, and with `includeRetired: true` it comes back with `retired`: its reason, who retired it and when. No test run checks it again. - There's no un-retire. Retiring a correction's case doesn't change the correction's status. - Retiring a case takes what adding one takes, on the case's project. A case retired already is `409 test-case-retired`. ## Test sets built before Kindgi 0.3 built `judged` eval suites from judged runs, with `kindgi eval-suites from-judgments`. Such a suite is a test set of the `corrections` kind now, and nothing builds one. One built before is still read by its id, and its cases with `kindgi test-sets cases`, and `kindgi test-sets list` lists it only with `--kind=corrections`, with the agents' own test sets. The corrections it was built from are cases of their agent's test set. ## Next - [Try a version on the test set](https://docs.kindgi.com/v0.4.0/guides/improve/compare-an-agent-version.md) - [Correct a run's output](https://docs.kindgi.com/v0.4.0/guides/improve/correct-a-runs-output.md) - [Corrections reference](https://docs.kindgi.com/v0.4.0/guides/improve/expert-queue-reference.md)