# Try a version on the test cases > Replay a new agent or flow version on the test cases, without doing anything the past runs didn't, and see whether people's corrections rate it higher. This guide shows whether people would rate a new agent or flow version higher than the runs they corrected, before anyone uses it. A **run on the test cases** tries an agent version on each case of the [test cases](https://docs.kindgi.com/v0.5.0/guides/improve/build-a-test-set.md), as a replay of the past run. It rates the version's answers with the corrections people already made, beside the recorded runs. ## Before you start - **Test cases:** the agent's [test set](https://docs.kindgi.com/v0.5.0/guides/improve/build-a-test-set.md), and the new version of their agent or flow. Under `kindgi dev`, saving the agent's file with its new `version` is enough. On a runtime you deploy to, deploy the new version first. - **Kindgi 0.4's CLI**, and the project's id. - **The right to start one:** write on the project the run lands in, and the right to run its agent or flow. A test set other than the agent's own also takes the right to execute it, which its project's editors have. - **For the app's half:** Node 22.12 or later, or Python 3.11 or later, with Kindgi 0.4's client, and `KINDGI_API_URL` and `KINDGI_API_TOKEN` set. ## Try a version on the test cases A run on the test cases replays the version on each case. Version 0.1.0 of `acme-ops.article-picker` answered with three articles each time, and people marked most of them wrong. Version 0.2.0 changes its instructions. Ask your coding agent: "Make a new version of my agent that answers with only the articles that answer the question." For reference, 0.2.0's instructions look like this: ```ts excerpt instructions: "Call acme-ops.search-articles with the customer's question. Answer with only the articles that answer it, most relevant first: one is often enough.", ``` The test cases keep their copies of the 0.1.0 runs. Try 0.2.0 on them: - **TypeScript** ```sh pnpm exec kindgi test-runs start agent.acme-ops.article-picker. --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` - **Python** ```sh uv run kindgi test-runs start agent.acme-ops.article-picker. --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` - **Java** ```sh ./kindgiw test-runs start agent.acme-ops.article-picker. --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` - **Scala** ```sh ./kindgiw test-runs start agent.acme-ops.article-picker. --project= --agent=acme-ops.article-picker --agent-version=0.2.0 --wait ``` * `agent.acme-ops.article-picker.` is the agent's [test set](https://docs.kindgi.com/v0.5.0/guides/improve/build-a-test-set.md) in the project: `agent.`, the agent's id, a dot, and the project's id. * `--project=` is the project the replay runs land in. It's the test set's project: another agent or project is refused with `400 bad-input`, and the message names the test set id this run would take. * `--agent` and `--agent-version` are the version to try. * `--wait` waits for the run to finish. It prints the run, with its metrics: ```json excerpt { "testRunId": "b0b9f27a-14d9-4a4d-bc46-31aea6948d5f", "testSetId": "agent.acme-ops.article-picker.", "testSetVersion": "1.0.3", "status": "completed", "result": { "summary": { "cases": 3, "candidate": { "kind": "agent", "agentId": "acme-ops.article-picker", "version": "0.2.0" }, "baseline": { "kind": "recorded", "versions": [{ "agentId": "acme-ops.article-picker", "version": "0.1.0", "cases": 3 }] }, "metrics": { "passRate": { "baseline": 0.3333333333333333, "candidate": 1, "delta": 0.6666666666666667, "n": 3, … }, "caseCoverage": { "baseline": 0.75, "candidate": 0.5, "delta": -0.25, "n": 3, … }, "weightedPrecisionAtK": { "k": 10, "baseline": 0.3333333333333333, "candidate": 1, "delta": 0.6666666666666667, "n": 3, … } }, "diverged": 1, "refusedWrites": 0, "errors": 0, "sampling": { "models": [{ "providerId": "anthropic", "model": "claude-sonnet-5-5", "runs": 3 }] }, … }, "perCase": [ … ] }, … } ``` Without `--wait`, the command returns the run's id at once, and `kindgi test-runs show ` reads it later. **The run freezes the test set first:** its cases as they are when it starts, as a version of the set (`1.0.`), which the run records as `testSetVersion`. A new version is made only when the cases changed since the newest; otherwise that one runs again. So two runs of different versions compare on the same cases, unless a case was added or retired between them. With no cases yet, the run is refused with `409 suggested-change-nothing-to-check`, and a runtime that doesn't freeze a project's test set answers `501 suggested-change-check-unsupported`. A [suggested change's check](https://docs.kindgi.com/v0.5.0/guides/improve/suggest-a-change.md#its-check) runs the same two test runs for you: its version and the version it's built on, on one frozen version of the set. From your code, ask your coding agent: "I want a script that tries my new agent version on my test cases, and shows me the scores." For reference, the script looks like this, with the project's id as its argument: - **TypeScript** `compare.ts` ```ts import { comparisonOf, createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const projectId = process.argv[2]; // Replay version 0.2.0 on each case of the agent's test set in the project. const { testRunId } = await kindgi.testRuns.start( `agent.acme-ops.article-picker.${projectId}`, { agentRef: { agentId: 'acme-ops.article-picker', version: '0.2.0' } }, { projectId }, ); let run = await kindgi.testRuns.get(testRunId); while (run.status === 'pending' || run.status === 'running') { await new Promise((resolve) => setTimeout(resolve, 2000)); run = await kindgi.testRuns.get(testRunId); } const comparison = comparisonOf(run); // undefined until it's finished for (const [name, metric] of Object.entries(comparison?.summary.metrics ?? {})) { console.log(`${name}: ${metric.baseline} → ${metric.candidate}`); } ``` ```text caseCoverage: 0.75 → 0.5 passRate: 0.3333333333333333 → 1 weightedPrecisionAtK: 0.3333333333333333 → 1 ``` - **Python** `compare.py` ```python import sys import time from kindgi.client import Kindgi, comparison_of kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev project_id = sys.argv[1] # Replay version 0.2.0 on each case of the agent's test set in the project. started = kindgi.test_runs.start( f"agent.acme-ops.article-picker.{project_id}", project_id=project_id, agent_ref={"agent_id": "acme-ops.article-picker", "version": "0.2.0"}, ) run = kindgi.test_runs.get(str(started.test_run_id)) while run.status in ("pending", "running"): time.sleep(2) run = kindgi.test_runs.get(str(started.test_run_id)) comparison = comparison_of(run) # None until it's finished if comparison is not None: metrics = comparison.summary.metrics for name, metric in [ ("caseCoverage", metrics.case_coverage), ("passRate", metrics.pass_rate), ("weightedPrecisionAtK", metrics.weighted_precision_at_k), ]: print(f"{name}: {metric.baseline} → {metric.candidate}") ``` ```text caseCoverage: 0.75 → 0.5 passRate: 0.3333333333333333 → 1.0 weightedPrecisionAtK: 0.3333333333333333 → 1.0 ``` - **Java** `src/main/java/com/acme/app/Compare.java` ```java package com.acme.app; import com.kindgi.client.Comparisons; import com.kindgi.client.Kindgi; import com.kindgi.client.models.ComparisonMetric; import com.kindgi.client.models.TestRun; import com.kindgi.client.models.TestRunAgentRef; import com.kindgi.client.models.CorrectedComparisonSummary; import com.kindgi.client.models.StartTestRunBody; import com.kindgi.client.models.StartTestRunResult; import java.util.UUID; public final class Compare { public static void main(String[] args) throws InterruptedException { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev UUID projectId = UUID.fromString(args[0]); // Replay version 0.2.0 on each case of the agent's test set in the project. StartTestRunResult started = kindgi.testRuns().start( "agent.acme-ops.article-picker." + projectId, StartTestRunBody.builder() .projectId(projectId) .agentRef( TestRunAgentRef.builder().agentId("acme-ops.article-picker").version("0.2.0").build()) .build()); TestRun run = kindgi.testRuns().get(started.testRunId()); while (run.status().equals(TestRun.Status.PENDING) || run.status().equals(TestRun.Status.RUNNING)) { Thread.sleep(2000); run = kindgi.testRuns().get(started.testRunId()); } // comparisonOf is empty until the run finishes, or for a dry run. Comparisons.comparisonOf(run) .ifPresent( comparison -> { CorrectedComparisonSummary.Metrics metrics = comparison.summary().metrics(); print("caseCoverage", metrics.caseCoverage()); print("passRate", metrics.passRate()); print("weightedPrecisionAtK", metrics.weightedPrecisionAtK()); }); } private static void print(String name, ComparisonMetric metric) { System.out.println(name + ": " + metric.baseline() + " → " + metric.candidate()); } private Compare() {} } ``` ```text caseCoverage: 0.75 → 0.5 passRate: 0.3333333333333333 → 1.0 weightedPrecisionAtK: 0.3333333333333333 → 1.0 ``` - **Scala** Scala apps use the Java helper: call `Comparisons.comparisonOf` from `com.kindgi.client` as the Java tab shows. It takes the `TestRun` and returns a `java.util.Optional` of the `CorrectedComparisonResult`: its `summary()`, and each case in `perCase()`. `comparisonOf` reads a finished run's result, typed: its `summary` and each case. It's `comparison_of` in Python, and `Comparisons.comparisonOf` in Java and Scala. For a run that hasn't finished, or a dry run, it returns `undefined`: `None` in Python, an empty `Optional` in Java and Scala. A model doesn't answer the same way every time: try the same version on the test cases again, and the scores may differ. `--repetitions=` runs each case several times, up to 10, and each metric gets a `spread`: its highest value minus its lowest. In the clients, it's `repetitions`. ## Read the result Each metric has its value for the recorded runs (`baseline`) and for the version (`candidate`), the `delta`, and `n`, the cases it's based on. - **`passRate`,** "Share marked right" in the console: of what people marked, the share marked right. Each item of the answer they corrected counts, weighted by corrector group, and so does each part of a case's [graded `expect`](https://docs.kindgi.com/v0.5.0/guides/improve/compare-an-agent-version.md#what-a-case-expects), its decisions and its answer, as one check of weight 1. The example's cases expect items alone: 0.1.0's answers had 3 marked right of 9 corrected articles. 0.2.0's had 3 of 3: it kept only the right article each time. - **`caseCoverage`,** "Corrected items": the share of the answer's items that people corrected. Items no one corrected bring it down. Here, each answer's text is an item too, and no one corrected it. - **`weightedPrecisionAtK`,** "Top 10 marked right": marked right among the first `k` items of a ranked list. `k` is 10 unless you set `--k`. **How a correction carries over.** An item with its own id, an `id` or `key` field, is the same item when the id matches, at any place in the list. So the `yes` on `kb-301` counts for 0.2.0 too. The answer's text, and a list element without an id, carry a correction only when they're unchanged. A reworded answer is a new item, with no corrections yet. ### Count only experts' corrections `--group-weights=experts-only` counts only the corrections made under an [expert group](https://docs.kindgi.com/v0.5.0/guides/improve/correct-a-runs-output.md#restrict-who-may-correct-as-a-group). An item none of them corrected counts as not corrected. In TypeScript, it's `groupWeights: 'experts-only'`, and in Python `group_weights="experts-only"`. The summary says which it used, in `groupWeights`, `as-recorded` by default. On an agent's test set, it grades only experts' cases, which have `expert: true`: a correction's case when the correction was made under a restricted expert group, and every case added by hand, since adding one takes `publish` on the agent (or `write` on its project). No other case's [`expect`](https://docs.kindgi.com/v0.5.0/guides/improve/compare-an-agent-version.md#what-a-case-expects) is graded. On three test cases, one of them corrected only under a group restricted afterwards: | | `as-recorded` | `experts-only` | | -------------- | -------------- | -------------- | | `passRate` | 0.556 (5 of 9) | 0.5 (3 of 6) | | `caseCoverage` | 1 | 0.667 | **Case by case,** `perCase` has, for each case: ```json excerpt { "caseId": "240c0a0b-542b-4cb1-9156-fe7153a4377a", "runIds": ["732235ae-2321-46fe-876e-9e9818671cbf"], "changes": { "kept": [{ "key": "kb-101", "rankBefore": 0, "rank": 0 }], "dropped": [{ "key": "kb-102", "rankBefore": 1 }, { "key": "kb-103", "rankBefore": 2 }], "new": [{ "key": "answer", "pointer": "/appended/3/content" }] }, "tools": [ { "toolId": "acme-ops.search-articles", "toolVersion": "0.1.0", "arguments": { "query": "I forgot my password and can't sign in" }, "source": "live", … } ], "diverged": true, "refusedWrites": 0, … } ``` - **`changes`:** the corrected items the version kept, with their place before and now, the ones it dropped, and the new ones, for people to correct. - **`tools`:** each tool call, and what happened to it: `live` when it ran, `recorded` when it got the past run's result, or `refused`. - **`diverged`:** the version read something the past run didn't, so the read ran live. Here, 0.2.0 left the final period off the customer's question when it searched, so the call wasn't the same as 0.1.0's. ### What it should have done When the test cases' runs have corrections that say [what the run should have done](https://docs.kindgi.com/v0.5.0/guides/improve/correct-a-runs-output.md#say-what-it-should-have-done) (runtime 0.2.0 or later), each one is checked against what the version did on that case: ```text excerpt CAPTURE: one perCase entry with "corrections": [{ "correctionId": …, "pass": false, "repetitionsPassed": 0, "repetitions": 1 }] and "correctionDifferences": [{ "correctionId": …, "repetition": 1, "differences": [ … ] }] ``` - **`corrections`:** for each one, whether it passed, and in how many of the case's repetitions (`repetitionsPassed` of `repetitions`). It passes only when every repetition did. - **`correctionDifferences`:** for one that failed, what differed in the first repetition that failed it, and that repetition's number. A difference is a call count (`count`), a call that shouldn't have been made (`must-not-call`), or an argument that's missing (`missing-field`) or has another value (`field-value`). `refused` marks a call the replay refused. - **The summary's `corrections`** counts the ones `checked`, and how many `passed`: ```text excerpt CAPTURE: "corrections": { "checked": …, "passed": … } from the same run's summary ``` With `--group-weights=experts-only`, only the ones made under an expert group are checked. A release gate checks them only with [`cases`](https://docs.kindgi.com/v0.5.0/guides/improve/pinned-versions-reference.md#release-gates), so read them before you [pin the version](https://docs.kindgi.com/v0.5.0/guides/improve/release-an-agent-version.md#check-before-you-pin-it). ### What a case expects A case of an [agent's test set](https://docs.kindgi.com/v0.5.0/guides/improve/build-a-test-set.md#what-a-case-expects) is graded on its `expect`, part by part, on each repetition: | The case expects | A repetition passes it when | | -------------------- | --------------------------------------------------------------------------------------------------------- | | `decisions: "wrong"` | It calls every `shouldCall` tool (with its `args` where given) and none of `mustNotCall`, checked exactly | | `decisions: "right"` | It calls the same tools the recorded run called, by id, in any order. A refused call counts as called | | `answer: "wrong"` | A judging model finds that its answer does what `shouldSay` or `why` asks | | `answer: "right"` | A judging model finds that its answer says what the recorded answer said | | `items` | Nothing: the items count toward the metrics, as above, with no pass | The case has one entry in `corrections`, under the case's id, and it's counted in the summary's `corrections`. A repetition passes when every part the case expects passed on it, and the case passes when every repetition did. Its `parts` says whether each part, its decisions and its answer, passed on every repetition. Each graded part is one check in `passRate`, beside the case's items, so a case that expects both is two checks, and a release gate on `passRate` counts them. The recorded run passes a part when people marked it `right`, and each repetition that ran passes it when the part passed on every repetition. For `decisions: "wrong"`, `correctionDifferences` says what differed. When an answer fails, `why` has the judging model's reason: ```json { "correctionId": "8f2c4e1a-6b0d-4c3e-9a57-2d1f0b7e4c93", "pass": false, "repetitionsPassed": 1, "repetitions": 3, "why": "It never says how long the refund takes.", "parts": { "decisions": true, "answer": false } } ``` - **Which model judges.** The version you try judges its own answers, with the model its model-settings block names. A suggested change's check judges both of its runs with the version the change is built on, and an Improve run judges with the version it starts from. So one model judges both sides. The model an Improve run names only writes its new wording. - **What it costs.** Each answer checked is a call to the judging model, which costs money. It's counted in its repetition's `costUsd`, so it's in the test run's cost, even when the reply couldn't be read. Answers are checked in order, and checking stops at the first answer that fails. An answer is checked on a repetition whose decisions failed too, since it's a check of its own. - **No pass.** A case that expects only `items` has no entry, and adds no check to `passRate`: its items count toward the metrics. Neither does a case whose answer couldn't be checked, say because the version's model settings name no model or the call to the judging model failed, or one that expects `decisions: "right"` when the recorded run's tool calls can't be read. The case's `unchecked` says why, and the summary's `unchecked` counts those cases and lists their reasons. A failed call doesn't stop the test run. - **A repetition that errored or stopped** fails every part, and its answer isn't checked, nor any after it. ## What a replay does A replay does nothing the past run didn't do. "The same call" means the same tool with the same arguments. - **A read-only tool** gets the past run's result for the same call. A read-only tool is `mutating: false`, has no effect that changes things, and no approval to ask for. It runs live when the past run didn't make the call, and the case is then `diverged`. - **A read-only tool also runs live** when the version you try pins other settings blocks than the past run's version. So a tool that reads its [settings](https://docs.kindgi.com/v0.5.0/guides/improve/change-a-prompt-or-setting.md) shows the change. That counts as `diverged` too. With `--reads=live`, or `reads: 'live'` in the clients, it always runs live. - **A read that reaches outside,** with a `network` or `sensitive-data-egress` effect, only ever gets the past run's result for the same call. Without one, it's refused, as below. - **Any other tool** gets the past run's result for the same call, or doesn't run. The model is told `replay: this call changes things and has no recorded result`, or that the call needs an approval, or reaches outside. The case shows what it would have done. `refusedWrites` counts every call a replay refuses, reads included, despite its name. - **A case with a correction that says what the run should have done replays strictly** (runtime 0.2.0 or later): no tool runs live, even with `--reads=live`, or when the version pins other settings blocks. A read gets the past run's result for the same call, or doesn't run: the model is told `replay: this read has no recorded result`, and `refusedWrites` counts it. So the version is checked on the results the expert saw. - **A replay never waits for an approval.** Where the past run's conversation was approved or rejected, the replay follows that decision. Where there's no decision, the case says `approvalSkipped`. - **A replay runs for the past turn's end user.** An agent whose memory is per end user, `same-user`, is offered the same tools as the past turn, `kindgi_remember` included, which stores nothing in a replay. A replay's conversation is never recalled later as an earlier conversation. - **A replay of a run a webhook or an event started reads its message as data** (it needs runtime 0.1.9 or later). Its model reads the turn's message, or a flow's agent step's, as an [`` block](https://docs.kindgi.com/v0.5.0/guides/webhooks/start-runs-from-webhooks.md#what-its-agents-read), data and not instructions, even when the past run was on an earlier runtime and read it as it came: its answer can differ for that alone. The replay's own run records no `startedBy`, as no replay does. A delivery earlier in the conversation reads that way too, unless an earlier runtime kept it: that one reads as it came. - **A recorded result that was data from outside stays data** (runtime 0.3.0 or later). Where the past run's model read a tool's result in an `` block, as another system's answer to a call that [paused](https://docs.kindgi.com/v0.5.0/guides/tools/wait-for-another-system.md#in-an-agent), or as the result of a tool whose [results are from outside](https://docs.kindgi.com/v0.5.0/guides/tools/read-only-tools.md#results-from-outside), the replay's model reads the past run's result the same way. On an earlier runtime it reads as it came. In a flow run first corrected on an earlier runtime, an agent step's answer to a paused call reads as it came. **The replay runs** are ordinary runs, marked with `replayOf`, the past run, and `testRunId`. Lists of runs leave them out. `kindgi runs list --test-run=` lists those of one run on the test cases, and `--replays=include` or `--replays=only` adds them to any list. Conversation lists leave out their conversations the same way: `kindgi conversations list --replays=only` lists them. You can [correct](https://docs.kindgi.com/v0.5.0/guides/improve/correct-a-runs-output.md) them like any run: that's how a version's new items get their corrections. **A run on the test cases stopped** before it finished, because the runtime stopped, ends `failed` with the cases it got through: `Interrupted: the test run's executor stopped before it finished. 2 of 3 cases finished.` Start it again. ## Try a flow version on the test cases A `corrections` test set built from a flow's corrected runs tries a flow version. Each case runs the past run's input through that version, as a replay. The flow's whole output is rated: the output itself, the `output` item, and each element of a list in it. To try some of its tools or agents at another version, without publishing a new flow version, add `--with=@` once for each. Here the flow `acme-ops.suggest-articles` 0.1.0 runs its `acme-ops.find-articles` tool at 0.2.0, which matches only the articles a question asks about: - **TypeScript** ```sh pnpm exec kindgi test-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` - **Python** ```sh uv run kindgi test-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` - **Java** ```sh ./kindgiw test-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` - **Scala** ```sh ./kindgiw test-runs start acme.flow-picks --project= --flow=acme-ops.suggest-articles --flow-version=0.1.0 --with=acme-ops.find-articles@0.2.0 --reads=live --wait ``` `--flow` and `--flow-version` are the flow version to try, and `--reads=live` runs read-only tools live. Its summary names the swapped tool: ```json excerpt "summary": { "candidate": { "kind": "flow", "flowId": "acme-ops.suggest-articles", "version": "0.1.0", "versions": { "tools": { "acme-ops.find-articles": "0.2.0" } } }, "metrics": { "passRate": { "baseline": 0.5, "candidate": 1, … }, … }, … } ``` - **In the clients:** `flowRef: { flowId, version }` in place of `agentRef`, and `versions: { tools: { 'acme-ops.find-articles': '0.2.0' } }`. Agents go under `agents`. - **A read-only tool's new version shows only with live reads.** With the default, recorded reads, its call gets the past run's result. So the same run on the test cases without `--reads=live` rates 50% before and after. - **Under `kindgi dev`,** a pack serves two versions of a tool side by side, so a tool swap works there. An agent has one version there, its file's, so swapping an agent needs a runtime you deploy to. - **A `--with` the flow doesn't use is refused:** `flow acme-ops.suggest-articles 0.1.0 doesn't use acme-ops.article-picker`. **A flow stops at a tool call the replay refuses.** `acme-ops.suggest-and-log` also records the articles it suggested. With 0.2.0 of the tool, it would have recorded other articles than the past run did for one question, so that case stops there: ```json excerpt "stopped": { "toolId": "acme-ops.log-suggestion", "arguments": { "articles": [{ "id": "kb-201", "title": "Request a refund" }], "question": "I was charged twice for one order. How do I get my money back?" }, "reason": "replay: this call changes things and has no recorded result" } ``` A stopped case isn't an error: the refused call wasn't carried out, and the case has no output to rate, so it's left out of the metrics. The summary counts it in `stopped`, and the refused call in `refusedWrites`. The other cases are rated as usual. A step's agent turn doesn't stop the flow: its refused calls go back to the model, as they do when you try an agent version. When a version does better, pin it where you choose, as [Release an agent version](https://docs.kindgi.com/v0.5.0/guides/improve/release-an-agent-version.md) shows. ## In the console The test cases' page has **Try an agent version on the test cases**: pick the version, how many runs per case, whether to read live data instead of the past runs' results, and **Count only experts**. Its **Runs on the test cases** list links each one to its page, which has: - a headline in words: "People's corrections rate acme-ops.article-picker 0.2.0 better than the recorded runs: 33% marked right before, 100% now, on the 3 of 3 cases with answers people corrected. In 1 case it read something the recording didn't have."; - the metrics, for the recorded runs and the version, with the change. A run that counted only expert groups says so: "Only experts' corrections counted; the others weighed 0.", and lists mark it "expert groups only"; - **Case by case:** the recorded answer beside the version's, the corrected items it kept and the ones no longer there, the new ones, and each tool call; - **The whole result**, as JSON. A flow's test cases have **Try a flow version on the test cases** instead, with a picker for each agent its steps run, to run it at another version. Tools are swapped from the CLI and the clients. The page of a flow's run on the test cases shows the recorded output beside the version's, and a stopped case says what it would have done: "It stopped at a tool call the replay refused: it would have called acme-ops.log-suggestion with `{"articles":[…],"question":"…"}`. That call wasn't carried out, and there's no output to rate." ## Next - [Release an agent version](https://docs.kindgi.com/v0.5.0/guides/improve/release-an-agent-version.md) - [Change a prompt or a setting without a deploy](https://docs.kindgi.com/v0.5.0/guides/improve/change-a-prompt-or-setting.md) - [Correct a run's output](https://docs.kindgi.com/v0.5.0/guides/improve/correct-a-runs-output.md)