# Compare a version on a test set > Replay a new agent or flow version on a test set's cases, without doing anything the past runs didn't, and see whether people's judgments rate it higher. This guide shows whether people would rate a new agent or flow version higher than the runs they judged, before anyone uses it. A **comparison** runs an agent version on each case of a [test set](https://docs.kindgi.com/v0.1.9/guides/evals/build-a-test-set.md), as a replay of the past run. It rates the version's answers with the judgments people already made, beside the recorded runs. ## Before you start - **A test set,** as [Build a test set](https://docs.kindgi.com/v0.1.9/guides/evals/build-a-test-set.md) shows, and the new version of its agent or flow. Under `kindgi dev`, saving the agent's file with its new `version` is enough. On a runtime you deploy to, deploy the new version first. - **Kindgi 0.1's CLI**, and the project's id. - **The right to start one:** an editor of the test set's project, or anyone who may execute the test set, who may write to the project the run lands in and run its agent or flow. - **For the app's half:** Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.1's client, and `KINDGI_API_URL` and `KINDGI_API_TOKEN` set. ## Compare a version A comparison replays the version on the test set's cases. Version 0.1.0 of `acme-ops.article-picker` answered with three articles each time, and people said no to most of them. Version 0.2.0 changes its instructions. Ask your coding agent: "Make a new version of my agent that answers with only the articles that answer the question." For reference, 0.2.0's instructions look like this: ```ts excerpt instructions: "Call acme-ops.search-articles with the customer's question. Answer with only the articles that answer it, most relevant first: one is often enough.", ``` The test set keeps its copies of the 0.1.0 runs. Compare 0.2.0 on it: - **TypeScript** ```sh pnpm exec kindgi eval-runs start acme.article-picks --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` - **Python** ```sh uv run kindgi eval-runs start acme.article-picks --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` - **Java** ```sh ./kindgiw eval-runs start acme.article-picks --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` - **Scala** ```sh ./kindgiw eval-runs start acme.article-picks --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` * `acme.article-picks` is the test set. * `--project=` is the project the replay runs land in. * `--agent` and `--agent-version` are the version to compare. * `--wait` waits for the comparison to finish. It prints the comparison, with its metrics: ```json excerpt { "runId": "b0b9f27a-14d9-4a4d-bc46-31aea6948d5f", "status": "completed", "result": { "summary": { "cases": 3, "candidate": { "kind": "agent", "agentId": "acme-ops.article-picker", "version": "0.2.0" }, "baseline": { "kind": "recorded", "versions": [{ "agentId": "acme-ops.article-picker", "version": "0.1.0", "cases": 3 }] }, "metrics": { "weightedYesShare": { "baseline": 0.3333333333333333, "candidate": 1, "delta": 0.6666666666666667, "n": 3, … }, "judgedCoverage": { "baseline": 0.75, "candidate": 0.5, "delta": -0.25, "n": 3, … }, "weightedPrecisionAtK": { "k": 10, "baseline": 0.3333333333333333, "candidate": 1, "delta": 0.6666666666666667, "n": 3, … } }, "diverged": 1, "refusedWrites": 0, "errors": 0, "sampling": { "models": [{ "providerId": "anthropic", "model": "claude-sonnet-5-5", "runs": 3 }] }, … }, "perCase": [ … ] }, … } ``` Without `--wait`, the command returns the run's id at once, and `kindgi eval-runs show ` reads it later. From your code, ask your coding agent: "I want a script that compares my new agent version on my test set, and shows me the scores." For reference, the script looks like this, with the project's id as its argument: - **TypeScript** `compare.ts` ```ts import { comparisonOf, createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const projectId = process.argv[2]; // Replay version 0.2.0 on each case of the test set. const { runId } = await kindgi.evalRuns.start( 'acme.article-picks', { agentRef: { agentId: 'acme-ops.article-picker', version: '0.2.0' } }, { projectId }, ); let run = await kindgi.evalRuns.get(runId); while (run.status === 'pending' || run.status === 'running') { await new Promise((resolve) => setTimeout(resolve, 2000)); run = await kindgi.evalRuns.get(runId); } const comparison = comparisonOf(run); // undefined until it's finished for (const [name, metric] of Object.entries(comparison?.summary.metrics ?? {})) { console.log(`${name}: ${metric.baseline} → ${metric.candidate}`); } ``` ```text judgedCoverage: 0.75 → 0.5 weightedYesShare: 0.3333333333333333 → 1 weightedPrecisionAtK: 0.3333333333333333 → 1 ``` - **Python** `compare.py` ```python import sys import time from kindgi.client import Kindgi, comparison_of kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev project_id = sys.argv[1] # Replay version 0.2.0 on each case of the test set. started = kindgi.eval_runs.start( "acme.article-picks", project_id=project_id, agent_ref={"agent_id": "acme-ops.article-picker", "version": "0.2.0"}, ) run = kindgi.eval_runs.get(str(started.run_id)) while run.status in ("pending", "running"): time.sleep(2) run = kindgi.eval_runs.get(str(started.run_id)) comparison = comparison_of(run) # None until it's finished if comparison is not None: metrics = comparison.summary.metrics for name, metric in [ ("judgedCoverage", metrics.judged_coverage), ("weightedYesShare", metrics.weighted_yes_share), ("weightedPrecisionAtK", metrics.weighted_precision_at_k), ]: print(f"{name}: {metric.baseline} → {metric.candidate}") ``` ```text judgedCoverage: 0.75 → 0.5 weightedYesShare: 0.3333333333333333 → 1.0 weightedPrecisionAtK: 0.3333333333333333 → 1.0 ``` `comparisonOf`, or `comparison_of` in Python, reads a finished comparison's result, typed: its `summary` and each case. It returns `undefined`, or `None`, for a comparison that hasn't finished, or a dry run. A model doesn't answer the same way every time: run the same comparison again, and the scores may differ. `--repetitions=` runs each case several times, up to 10, and each metric gets a `spread`: its highest value minus its lowest. In the clients, it's `repetitions`. ## Read the result Each metric has its value for the recorded runs (`baseline`) and for the version (`candidate`), the `delta`, and `n`, the cases it's based on. - **`weightedYesShare`,** "Judged yes" in the console: of the items in the answer that people judged, the share they said yes to, weighted by judge class. 0.1.0's answers had 3 yes of 9 judged articles. 0.2.0's had 3 of 3: it kept only the right article each time. - **`judgedCoverage`,** "Judged items": the share of the answer's items that people judged. Items no one judged bring it down. Here, each answer's text is an item too, and no one judged it. - **`weightedPrecisionAtK`,** "Top 10 judged yes": judged yes among the first `k` items of a ranked list. `k` is 10 unless you set `--k`. **How a judgment carries over.** An item with its own id, an `id` or `key` field, is the same item when the id matches, at any place in the list. So the `yes` on `kb-301` counts for 0.2.0 too. The answer's text, and a list element without an id, carry a judgment only when they're unchanged. A reworded answer is a new item, with no judgments yet. ### Count only restricted judges `--class-weights=restricted-only` counts only the judgments made under a [restricted class](https://docs.kindgi.com/v0.1.9/guides/evals/judge-a-runs-output.md#restrict-who-may-judge-as-a-class). An item none of them judged counts as unjudged. In TypeScript, it's `classWeights: 'restricted-only'`, and in Python `class_weights="restricted-only"`. The summary says which it used, in `classWeights`, `as-recorded` by default. On a test set of three cases, one of them judged only under a class restricted afterwards: | | `as-recorded` | `restricted-only` | | ------------------ | -------------- | ----------------- | | `weightedYesShare` | 0.556 (5 of 9) | 0.5 (3 of 6) | | `judgedCoverage` | 1 | 0.667 | **Case by case,** `perCase` has, for each case: ```json excerpt { "caseId": "240c0a0b-542b-4cb1-9156-fe7153a4377a", "runIds": ["732235ae-2321-46fe-876e-9e9818671cbf"], "changes": { "kept": [{ "key": "kb-101", "rankBefore": 0, "rank": 0 }], "dropped": [{ "key": "kb-102", "rankBefore": 1 }, { "key": "kb-103", "rankBefore": 2 }], "new": [{ "key": "answer", "pointer": "/appended/3/content" }] }, "tools": [ { "toolId": "acme-ops.search-articles", "toolVersion": "0.1.0", "arguments": { "query": "I forgot my password and can't sign in" }, "source": "live", … } ], "diverged": true, "refusedWrites": 0, … } ``` - **`changes`:** the judged items the version kept, with their place before and now, the ones it dropped, and the new ones, for people to judge. - **`tools`:** each tool call, and what happened to it: `live` when it ran, `recorded` when it got the past run's result, or `refused`. - **`diverged`:** the version read something the past run didn't, so the read ran live. Here, 0.2.0 left the final period off the customer's question when it searched, so the call wasn't the same as 0.1.0's. ## What a replay does A replay does nothing the past run didn't do. "The same call" means the same tool with the same arguments. - **A read-only tool** gets the past run's result for the same call. A read-only tool is `mutating: false`, has no effect that changes things, and no approval to ask for. It runs live when the past run didn't make the call, and the case is then `diverged`. - **A read-only tool also runs live** when the version you compare pins other settings blocks than the past run's version. So a tool that reads its [settings](https://docs.kindgi.com/v0.1.9/guides/evals/change-a-prompt-or-setting.md) shows the change. That counts as `diverged` too. With `--reads=live`, or `reads: 'live'` in the clients, it always runs live. - **A read that reaches outside,** with a `network` or `sensitive-data-egress` effect, only ever gets the past run's result for the same call. - **Any other tool** gets the past run's result for the same call, or doesn't run. The model is told `replay: this call changes things and has no recorded result`, or that the call needs an approval, or reaches outside. The case shows what it would have done, and `refusedWrites` counts these calls. - **A replay never waits for an approval.** Where the past run's conversation was approved or rejected, the replay follows that decision. Where there's no decision, the case says `approvalSkipped`. - **A replay runs for the past turn's end user.** An agent whose memory is per end user, `same-user`, is offered the same tools as the past turn, `kindgi_remember` included, which stores nothing in a replay. A replay's conversation is never recalled later as an earlier conversation. - **A replay of a run a webhook or an event started reads its message as data** (it needs runtime 0.1.9 or later). Its model reads the turn's message, or a flow's agent step's, as an [`` block](https://docs.kindgi.com/v0.1.9/guides/webhooks/start-runs-from-webhooks.md#what-its-agents-read), data and not instructions, even when the past run was on an earlier runtime and read it as it came: its answer can differ for that alone. The replay's own run records no `startedBy`, as no replay does. A delivery earlier in the conversation reads that way too, unless an earlier runtime kept it: that one reads as it came. **The replay runs** are ordinary runs, marked with `replayOf`, the past run, and `evalRunId`. Lists of runs leave them out. `kindgi runs list --eval-run=` lists one comparison's, and `--replays=include` or `--replays=only` adds them to any list. Conversation lists leave out their conversations the same way: `kindgi conversations list --replays=only` lists them. You can [judge](https://docs.kindgi.com/v0.1.9/guides/evals/judge-a-runs-output.md) them like any run: that's how a version's new items get their judgments. **A comparison stopped** before it finished, because the runtime stopped, ends `failed` with the cases it got through: `Interrupted: the eval run's executor stopped before it finished. 2 of 3 cases finished.` Start it again. ## Compare a flow version A test set built from a flow's judged runs, with `kindgi eval-suites from-judgments … --flow=`, compares a flow version. Each case runs the past run's input through that version, as a replay. The flow's whole output is rated: the output itself, the `output` item, and each element of a list in it. To try some of its tools or agents at another version, without publishing a new flow version, add `--with=@` once for each. Here the flow `acme-ops.suggest-articles` 0.1.0 runs its `acme-ops.find-articles` tool at 0.2.0, which matches only the articles a question asks about: - **TypeScript** ```sh pnpm exec kindgi eval-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` - **Python** ```sh uv run kindgi eval-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` - **Java** ```sh ./kindgiw eval-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` - **Scala** ```sh ./kindgiw eval-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` `--flow` and `--flow-version` are the flow version to compare, and `--reads=live` runs read-only tools live. Its summary names the swapped tool: ```json excerpt "summary": { "candidate": { "kind": "flow", "flowId": "acme-ops.suggest-articles", "version": "0.1.0", "versions": { "tools": { "acme-ops.find-articles": "0.2.0" } } }, "metrics": { "weightedYesShare": { "baseline": 0.5, "candidate": 1, … }, … }, … } ``` - **In the clients:** `flowRef: { flowId, version }` in place of `agentRef`, and `versions: { tools: { 'acme-ops.find-articles': '0.2.0' } }`. Agents go under `agents`. - **A read-only tool's new version shows only with live reads.** With the default, recorded reads, its call gets the past run's result. So the same comparison without `--reads=live` rates 50% before and after. - **Under `kindgi dev`,** a pack serves two versions of a tool side by side, so a tool swap works there. An agent has one version there, its file's, so swapping an agent needs a runtime you deploy to. - **A `--with` the flow doesn't use is refused:** `flow acme-ops.suggest-articles 0.1.0 doesn't use acme-ops.article-picker`. **A flow stops at a write the replay refuses.** `acme-ops.suggest-and-log` also records the articles it suggested. With 0.2.0 of the tool, it would have recorded other articles than the past run did for one question, so that case stops there: ```json excerpt "stopped": { "toolId": "acme-ops.log-suggestion", "arguments": { "articles": [{ "id": "kb-201", "title": "Request a refund" }], "question": "I was charged twice for one order. How do I get my money back?" }, "reason": "replay: this call changes things and has no recorded result" } ``` A stopped case isn't an error: nothing was written, and it has no output to rate, so it's left out of the metrics. The summary counts it in `stopped`, and the refused call in `refusedWrites`. The other cases are rated as usual. A step's agent turn doesn't stop the flow: its refused calls go back to the model, as in an agent comparison. When a version does better, make it live where you choose, as [Release an agent version](https://docs.kindgi.com/v0.1.9/guides/evals/release-an-agent-version.md) shows. ## In the console A test set's page has **Compare an agent version**: pick the version, how many runs per case, whether to read live data instead of the past runs' results, and **Count only restricted judge classes**. Its **Comparisons** list links each one to its page, which has: - a headline in words: "People's judgments rate acme-ops.article-picker 0.2.0 better than the recorded runs: 33% yes before, 100% now, on the 3 of 3 cases with answers people judged. In 1 case it read something the recording didn't have."; - the metrics, for the recorded runs and the version, with the change. A comparison that counted only restricted classes says so: "Only judgments in judge classes restricted to some judges counted; the others weighed 0.", and lists mark it "restricted classes only"; - **Case by case:** the recorded answer beside the version's, the judged items it kept and the ones no longer there, the new ones, and each tool call; - **The raw result**, as JSON. A flow's test set has **Compare a flow version** instead, with a picker for each agent its steps run, to run it at another version. Tools are swapped from the CLI and the clients. A flow comparison's page shows the recorded output beside the version's, and a stopped case says what it would have done: "It stopped before a write the recorded run didn't make: it would have called acme-ops.log-suggestion with `{"articles":[…],"question":"…"}`. Nothing was written, and there's no output to rate." ## Next - [Release an agent version](https://docs.kindgi.com/v0.1.9/guides/evals/release-an-agent-version.md) - [Change a prompt or a setting without a deploy](https://docs.kindgi.com/v0.1.9/guides/evals/change-a-prompt-or-setting.md) - [Judge a run's output](https://docs.kindgi.com/v0.1.9/guides/evals/judge-a-runs-output.md)