# Build a test set from judgments > Gather an agent's judged runs into a versioned test set, with what people said about each item, to test new versions on. This guide gathers an agent's judged runs into a versioned test set, to test new versions on. A **test set** is a set of past runs that people judged. Each case is a copy of one run, with the judgments of each item of its output added up. You [compare a new agent version](https://docs.kindgi.com/v0.1.8/guides/evals/compare-an-agent-version.md) on it to see whether people would rate its answers higher, before anyone uses it. ## Before you start - **An agent with finished runs** in a project, on a runtime such as `kindgi dev`, with Kindgi 0.1's CLI. - **`admin` on the project,** when authorization is on. - **For the app's half:** Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.1's client, and `KINDGI_API_URL` and `KINDGI_API_TOKEN` set. ## Judge some runs A test set is made of judged runs, so judge some first. The examples on this page use an agent, `acme-ops.article-picker`, that picks the help articles answering a customer's question. Its [typed answer](https://docs.kindgi.com/v0.1.8/guides/agents/typed-answer.md) is a list of articles, each with its own id: ```json { "articles": [ { "id": "kb-101", "title": "Reset your password" }, { "id": "kb-102", "title": "Change the email on your account" }, { "id": "kb-103", "title": "Turn on two-step sign-in" } ] } ``` Judge each article by its id. Then a judgment of `kb-101` still applies when a new version returns `kb-101`, at any place in its list. - **TypeScript** ```sh pnpm exec kindgi judgments add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password" pnpm exec kindgi judgments add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in" ``` - **Python** ```sh uv run kindgi judgments add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password" uv run kindgi judgments add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in" ``` - **Java** ```sh ./kindgiw judgments add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password" ./kindgiw judgments add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in" ``` - **Scala** ```sh ./kindgiw judgments add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password" ./kindgiw judgments add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in" ``` * `` is the run's id. * `--item` is the article's id, and `--pointer` where it is in the output. * `--rank` is its place in the list, 0 first. Each prints its judgment, as [Judge an item](https://docs.kindgi.com/v0.1.8/guides/evals/judge-a-runs-output.md#judge-an-item) shows. In the console, a finished run's **Judge this run** opens its Judgments tab. It lists the answer, and each element of a list in the output, each with **Yes** and **No**. [Judge a run's output](https://docs.kindgi.com/v0.1.8/guides/evals/judge-a-runs-output.md) has the rest: reasons, your users' judgments, and judge classes. ## Build the test set Build version 1.0.0 of a test set, `acme.article-picks`, from the agent's judged runs in a project: - **TypeScript** ```sh pnpm exec kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project= --agent=acme-ops.article-picker ``` - **Python** ```sh uv run kindgi eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project= --agent=acme-ops.article-picker ``` - **Java** ```sh ./kindgiw eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project= --agent=acme-ops.article-picker ``` - **Scala** ```sh ./kindgiw eval-suites from-judgments acme.article-picks --suite-version=1.0.0 --project= --agent=acme-ops.article-picker ``` * `acme.article-picks` is the test set's id, and `--suite-version` its version. * `--project=` is the project the runs are in, where `` is its id. * `--agent` is the agent whose judged runs it takes. It prints the test set: ```json { "suiteId": "acme.article-picks", "version": "1.0.0", "kind": "judged", "caseCount": 3, "truncated": false } ``` With [authorization](https://docs.kindgi.com/v0.1.8/deploy/authorization.md) on, building a test set under a new id takes `admin` on the project it names. From your code, ask your coding agent: "I want a script that builds a test set from my agent's judged runs, and shows me its cases." For reference, the script looks like this, with the project's id as its argument: - **TypeScript** `test-set.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const projectId = process.argv[2]; // Version 1.0.0 of the test set: copies of the agent's judged runs. const built = await kindgi.evalSuites.buildFromJudgments('acme.article-picks', { version: '1.0.0', projectId, agentId: 'acme-ops.article-picker', }); console.log(`${built.caseCount} cases`); const cases = await kindgi.evalSuites.listCases('acme.article-picks', '1.0.0'); for (const c of cases.data) { console.log(c.input, c.items.map((item) => `${item.key} ${item.yes} yes, ${item.no} no`)); } ``` ```text 3 cases I was charged twice for one order. How do I get my money back? [ 'kb-201 1 yes, 0 no', 'kb-203 0 yes, 1 no', 'kb-202 0 yes, 1 no' ] I forgot my password and can't sign in. [ 'kb-101 1 yes, 0 no', 'kb-102 0 yes, 1 no', 'kb-103 0 yes, 1 no' ] My package hasn't arrived yet. Where is it? [ 'kb-301 1 yes, 0 no', 'kb-303 0 yes, 1 no', 'kb-201 0 yes, 1 no' ] ``` - **Python** `test_set.py` ```python import sys from kindgi.client import Kindgi kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev project_id = sys.argv[1] # Version 1.0.0 of the test set: copies of the agent's judged runs. built = kindgi.eval_suites.build_from_judgments( "acme.article-picks", version="1.0.0", project_id=project_id, agent_id="acme-ops.article-picker", ) print(f"{built.case_count} cases") cases = kindgi.eval_suites.list_cases("acme.article-picks", "1.0.0") for case in cases.data: print(case.input, [f"{item.key} {item.yes} yes, {item.no} no" for item in case.items]) ``` ```text 3 cases I was charged twice for one order. How do I get my money back? ['kb-201 1 yes, 0 no', 'kb-203 0 yes, 1 no', 'kb-202 0 yes, 1 no'] I forgot my password and can't sign in. ['kb-101 1 yes, 0 no', 'kb-102 0 yes, 1 no', 'kb-103 0 yes, 1 no'] My package hasn't arrived yet. Where is it? ['kb-301 1 yes, 0 no', 'kb-303 0 yes, 1 no', 'kb-201 0 yes, 1 no'] ``` It prints how many cases the test set has, then each case's input and its items' counts. - **The project's id:** `kindgi projects get-default` prints the Default project, with its `id`, and `kindgi projects list` lists them all. In code, it's `kindgi.projects.getDefault()`, or `kindgi.projects.get_default()` in Python. - **Which runs:** every run of the agent in the project with at least one judgment. These narrow it: - `--agent-version`: one version's runs; - `--since` and `--until`: when the run was first judged; - `--class`: count only judgments of these [judge classes](https://docs.kindgi.com/v0.1.8/guides/evals/judge-a-runs-output.md#weight-judges-by-class); - `--min-judgments`: leave out runs with fewer. - **Up to 1000 cases.** `truncated: true` says there were more: narrow the runs, or build several sets. - **Versions never change.** When more runs are judged, build the next version: the same id and a new `--suite-version`. A taken version is refused: `Eval suite "acme.article-picks" version "1.0.0" is already registered.` - **Building a test set takes `admin` on the project.** ### Narrow it to a segment `--segment=` keeps the runs started in that segment or below it. Repeat it for a path, coarse to fine. In the clients, it's `segments`: - **TypeScript** ```sh pnpm exec kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project= \ --agent=acme.rank.scorer --segment=company:acme --description="acme company only" ``` - **Python** ```sh uv run kindgi eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project= \ --agent=acme.rank.scorer --segment=company:acme --description="acme company only" ``` - **Java** ```sh ./kindgiw eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project= \ --agent=acme.rank.scorer --segment=company:acme --description="acme company only" ``` - **Scala** ```sh ./kindgiw eval-suites from-judgments acme.rank-acme-segment --suite-version=1.0.0 --project= \ --agent=acme.rank.scorer --segment=company:acme --description="acme company only" ``` `--description` says what the test set holds. It prints the test set: ```json { "suiteId": "acme.rank-acme-segment", "version": "1.0.0", "kind": "judged", "caseCount": 20, "truncated": false } ``` A judged run keeps the segment path it was started with. A run judged without one, such as one an older runtime recorded, is in no segment, and a narrowed test set leaves it out. ## What a case keeps To see what each case holds, list the cases: - **TypeScript** ```sh pnpm exec kindgi eval-suites cases acme.article-picks --suite-version=1.0.0 ``` - **Python** ```sh uv run kindgi eval-suites cases acme.article-picks --suite-version=1.0.0 ``` - **Java** ```sh ./kindgiw eval-suites cases acme.article-picks --suite-version=1.0.0 ``` - **Scala** ```sh ./kindgiw eval-suites cases acme.article-picks --suite-version=1.0.0 ``` One of them: ```json excerpt { "caseId": "240c0a0b-542b-4cb1-9156-fe7153a4377a", "subject": { "kind": "agent", "id": "acme-ops.article-picker", "version": "0.1.0" }, "input": "I forgot my password and can't sign in.", "items": [ { "key": "kb-101", "pointer": "/output/articles/0", "rank": 0, "yes": 1, "no": 0, "yesWeight": 1, "totalWeight": 1, "restricted": { "yesWeight": 0, "totalWeight": 0 }, "reasons": [{ "verdict": "yes", "reason": "Resets the password" }] }, … ], "output": { "output": { "articles": [ … ] }, "appended": [ … ], … }, "context": { … } } ``` - **`caseId`** is the id of the judged run. - **`items`** has each judged item: its yes and no counts, the same weighted by judge class, and the reasons people gave. The weighted counts are `yesWeight` of `totalWeight`, and a judgment with no class weighs 1. `restricted` has the weights of the [restricted](https://docs.kindgi.com/v0.1.8/guides/evals/judge-a-runs-output.md#restrict-who-may-judge-as-a-class) judgments alone, such as `{ "yesWeight": 3, "totalWeight": 3 }`. - **`output`** is the run's output, as it was judged. - **`context`**, for an agent turn, is what the turn had before it answered: the conversation so far, up to 200 messages, and what it retrieved. A comparison replays the turn with the same context. Its `toolEnv` has the [env values](https://docs.kindgi.com/v0.1.8/guides/tools/give-a-tool-env-values.md) each tool was sent, by tool id. A read-only tool the comparison runs live gets those, not the values set now. ## In the console A project's **Test sets** section lists its test sets, and its form **Build a test set from judgments** takes the id, the agent, and optionally one version and the judgments a run needs. A test set's page shows its versions and its cases: each item with its yes and no counts, its weighted yes share, up to two reasons, and a link to the original run. ## Next - [Compare a version on a test set](https://docs.kindgi.com/v0.1.8/guides/evals/compare-an-agent-version.md) - [Judge a run's output](https://docs.kindgi.com/v0.1.8/guides/evals/judge-a-runs-output.md) - [Judging reference](https://docs.kindgi.com/v0.1.8/guides/evals/judging-reference.md)