# Correct a run's output > Record a yes or no, with a reason, on each item of a finished run's output, from your app, the CLI or the console, and weight corrections by judge class. This guide records people's yes or no on what a run returned, counts some people's corrections more than others, and queues the runs that need a correction. A **correction** is a yes or no, with an optional reason, about one item of a finished run's output: the whole answer, or one element of a list it returned. People who know the work, your users or your experts, correct it. The corrections then tell you how an agent or flow version does: you [build test cases](https://docs.kindgi.com/v0.3.0/guides/evals/build-a-test-set.md) from them, and [try a new version](https://docs.kindgi.com/v0.3.0/guides/evals/compare-an-agent-version.md) on them. [Corrections reference](https://docs.kindgi.com/v0.3.0/guides/evals/judging-reference.md) has every field, rule and console page. ## Before you start - **A finished run to correct,** and its id, on a runtime such as `kindgi dev`, with Kindgi 0.2's CLI. [List runs](https://docs.kindgi.com/v0.3.0/guides/runs/list-runs.md) finds one. - **Write access to the run's project,** when authorization is on. - **For the app's half:** Node 22.12 or later, Python 3.11 or later, or Java 17 or later, with Kindgi 0.2's client, and `KINDGI_API_URL` and `KINDGI_API_TOKEN` set. ## Correct an item A correction records whether one item of the run's output was right. From the CLI, correct the run's answer: - **TypeScript** ```sh pnpm exec kindgi judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` - **Python** ```sh uv run kindgi judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` - **Java** ```sh ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` - **Scala** ```sh ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` * `--run=` is the run's id. * `--item=answer` is your id for the item: `answer` for the whole answer. * `--pointer=/response/content` is where the item is in the run's output, as a JSON Pointer. * `--yes` is the verdict, or `--no`. * `--reason` (optional) says why. It prints the correction: ```json excerpt { "id": "512df8f6-6e27-4c4a-934e-1b345dfc510a", "projectId": "d4910d76-7355-4022-8f3c-514361cfa986", "runId": "bc6d1b53-3562-48c9-9f29-719779e3d110", "subject": { "kind": "agent", "id": "acme-ops.echo-agent", "version": "0.1.0" }, "item": { "key": "answer", "pointer": "/response/content" }, "verdict": "yes", "reason": "Echoes the message back", "assertedBy": { "kind": "user", "id": "26fc414a-b49f-4e1f-a022-162865770178" }, "createdAt": "2026-10-06T04:08:27.667Z", … } ``` From your app, when your users correct runs in your own UI, ask your coding agent: "I want to record my user's yes or no on a run's answer." For reference, the script looks like this, with the run's id as its argument: - **TypeScript** `judge.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const judgment = await kindgi.judgments.create({ runId: process.argv[2], item: { key: 'answer', pointer: '/response/content' }, verdict: 'yes', reason: 'Echoes the message back', participantId: 'customer-user-17', // your id for the user who judged, if you judge for your users }); console.log(judgment.id, judgment.verdict, judgment.assertedBy); ``` - **Python** `judge.py` ```python import sys from kindgi.client import Kindgi kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev judgment = kindgi.judgments.create( run_id=sys.argv[1], item={"key": "answer", "pointer": "/response/content"}, verdict="yes", reason="Echoes the message back", participant_id="customer-user-18", # your id for the user who judged, if you judge for your users ) print(judgment.id, judgment.verdict, judgment.asserted_by) ``` - **Java** `src/main/java/com/acme/app/Judge.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.CreateJudgmentBody; import com.kindgi.client.models.JudgedItem; import com.kindgi.client.models.Judgment; public final class Judge { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev Judgment judgment = kindgi.judgments().create(CreateJudgmentBody.builder() .runId(args[0]) .item(JudgedItem.builder().key("answer").pointer("/response/content").build()) .verdict(CreateJudgmentBody.Verdict.YES) .reason("Echoes the message back") .participantId("customer-user-19") // your id for the user who judged, if you judge for your users .build()); System.out.println(judgment.id() + " " + judgment.verdict() + " " + judgment.assertedBy()); } } ``` ```text 42a5cdac-a5b4-4bf1-a777-d46ead4ec743 yes JudgmentAssertedBy[kind=user, id=ebdba929-5422-4cb2-b6bb-2c046109feb8] ``` - **Scala** `src/main/scala/acme/app/Judge.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.models.{CreateJudgmentBody, JudgedItem} object Judge { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val judgment = kindgi.judgments().create(CreateJudgmentBody.builder() .runId(args(0)) .item(JudgedItem.builder().key("answer").pointer("/response/content").build()) .verdict(CreateJudgmentBody.Verdict.YES) .reason("Echoes the message back") .participantId("customer-user-20") // your id for the user who judged, if you judge for your users .build()) println(s"${judgment.id} ${judgment.verdict} ${judgment.assertedBy}") } } ``` ```text 339dbf38-f4b0-4a9a-a1f6-0dc4585dae93 yes JudgmentAssertedBy[kind=user, id=ebdba929-5422-4cb2-b6bb-2c046109feb8] ``` * **`participantId`** is your app's own opaque id for the user who made the correction. Who called comes from the token, as `assertedBy`. * **Correcting the same item again,** as the same caller and participant, replaces the earlier correction, and the earlier one stays in the history. The run now has the correction. Only a finished run can be corrected, and the pointer must find something. [Corrections](https://docs.kindgi.com/v0.3.0/guides/evals/judging-reference.md#corrections) has the rules and the refusals. In the console, a finished run's page has **Correct this run**, or a **Corrections** tab once it has corrections. It lists the answer and each element of a list in the output, each with **Yes** and **No** and the counts so far. It lists each correction with who made it, its class and its reason. ## Say what it should have done A correction of an agent's run can also say what the run should have done (runtime 0.2.0 or later): the tool calls it should have made, with the arguments that matter, and the tools it shouldn't have called. It's about the whole run, not one item of its output. When you [try a version](https://docs.kindgi.com/v0.3.0/guides/evals/compare-an-agent-version.md#what-it-should-have-done) on test cases built from the run, it checks what that version does against it. Write what it should have done in a file, such as `refund-fix.json`: ```json { "calls": [ { "tool": "acme-store.offer-store-credit", "args": { "orderId": "A-1042" } } ], "notCalled": ["acme-store.refund"] } ``` - **`calls`** are the calls it should have made. `args` holds only the top-level arguments that matter, each compared whole. Arguments left out aren't compared, and `{}` asks only that the tool was called. - **`notCalled`** are the tools it shouldn't have called at all. Then record it with your mark: - **TypeScript** ```sh pnpm exec kindgi judgments add --run= --no --reason="Final sale: store credit, never a refund" --correction=@refund-fix.json ``` - **Python** ```sh uv run kindgi judgments add --run= --no --reason="Final sale: store credit, never a refund" --correction=@refund-fix.json ``` - **Java** ```sh ./kindgiw judgments add --run= --no --reason="Final sale: store credit, never a refund" --correction=@refund-fix.json ``` - **Scala** ```sh ./kindgiw judgments add --run= --no --reason="Final sale: store credit, never a refund" --correction=@refund-fix.json ``` * `--correction` is what it should have done, as JSON or `@`. * `--no` marks the run's decisions wrong. `--yes` marks them right, and the calls you record then hold a later version to them. * It's about the whole run, so leave out `--item` and `--pointer`. It prints the correction, with what it should have done as its `correction`: ```text CAPTURE: the printed judgment, with "item": { "key": "kindgi:correction", "pointer": "" } and its "correction" ``` - **One per run per person.** Saving it again for the same run, as the same caller and participant, replaces your last one. - **Only the agent's own tools.** A tool the agent version that ran doesn't have is refused: `The judged version has no tool "".` From your app, ask your coding agent: "I want to record what my expert says the run should have done." The clients' `judgments.create` takes it as `correction`, under the item `{ key: 'kindgi:correction', pointer: '' }`. Its types are `Correction` and `ExpectedCall`, and in Python the field is `not_called`. In the console, a completed agent run's **Correct this run**, or its **Corrections** tab once it has corrections, has **What should it have done?**: 1. It lists the run's calls. Keep the ones that were right, and edit or untick the argument fields that matter. 2. Flag the calls that shouldn't have happened, and add any the run missed with **Add a call…**. 3. Choose **The decisions were wrong** or **The decisions were right**, add **Why (optional)**, and **Save correction**. ## See the corrections A run's corrections say how its output was corrected, and by whom: - **TypeScript** ```sh pnpm exec kindgi judgments list --run= # newest first; replaced and removed ones are left out pnpm exec kindgi judgments show # with copies of the run's input and output, and the item's value pnpm exec kindgi judgments remove ``` - **Python** ```sh uv run kindgi judgments list --run= # newest first; replaced and removed ones are left out uv run kindgi judgments show # with copies of the run's input and output, and the item's value uv run kindgi judgments remove ``` - **Java** ```sh ./kindgiw judgments list --run= # newest first; replaced and removed ones are left out ./kindgiw judgments show # with copies of the run's input and output, and the item's value ./kindgiw judgments remove ``` - **Scala** ```sh ./kindgiw judgments list --run= # newest first; replaced and removed ones are left out ./kindgiw judgments show # with copies of the run's input and output, and the item's value ./kindgiw judgments remove ``` * `` is the run's id, and `` a correction's `id`, from the list. * `list` also filters by agent and version, flow, verdict, judge class, participant and project. `list` prints a page of corrections as JSON: `data`, `hasMore`, and `nextCursor` when there's another page. `show` prints one correction, and `remove` prints its id with `"removed": true`. A correction keeps copies of the run's input and output, so it outlives the run's own retention. The clients have the same: `judgments.list`, `judgments.get` and `judgments.unregister`. ## Weight corrections by judge class Without a class, every correction counts once. A **judge class** counts some people's corrections more, for the tenant, a project, or an agent in a project. Add a class, then correct as it: - **TypeScript** ```sh pnpm exec kindgi judge-classes add --name=expert --weight=3 --project= pnpm exec kindgi judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` - **Python** ```sh uv run kindgi judge-classes add --name=expert --weight=3 --project= uv run kindgi judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` - **Java** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --project= ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` - **Scala** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --project= ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` * `--name` names the class, and `--weight=3` counts its corrections three times. * `--project=` is the project it applies to. [Build the test cases](https://docs.kindgi.com/v0.3.0/guides/evals/build-a-test-set.md#build-the-test-cases) shows how to find a project's id. * `--class=` is the class's `id`, which `add` printed. In the clients, it's `judgeClassId`. The correction now counts with the class's weight. ### Restrict who may correct as a class A restricted class says who may correct as it, so the people using your app can't move the figures that release a version. Add the class with a restriction, before your reviewers correct runs: - **TypeScript** ```sh pnpm exec kindgi judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` - **Python** ```sh uv run kindgi judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` - **Java** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` - **Scala** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` * `--tenant` makes it the tenant's class. * `--min-reviewer-role=senior` lets only a [reviewer](https://docs.kindgi.com/v0.3.0/guides/approvals/decide-an-approval.md#reviewers-and-roles) of that role or above correct as it: `standard`, `senior` or `admin`. * `--description` says who it is for. * `--principal-kind=user|service` and `--principal-id=`, which repeat, restrict it to people or service tokens, or to named ones. A caller must meet each restriction you set. One who doesn't is refused with `403`, and nothing is recorded: ```text Error [auth]: You can't judge as "expert": it needs a senior reviewer or above; you're a standard reviewer. Error [auth]: You can't judge as "tokens": only service tokens may assert it. ``` A correction made under a restriction carries `"restricted": true`. It's decided when the correction is made, so corrections recorded before the restriction don't count as restricted. [Judge classes](https://docs.kindgi.com/v0.3.0/guides/evals/judging-reference.md#judge-classes) has the rest. ## Queue the runs to correct **Judging rules** pick the project's runs that need a person's correction, so your experts don't pick runs one at a time. As a run of the project ends, each rule checks it. A run that any rule matches joins the project's queue of answers waiting for a correction, **To correct** in the console, once. To make a rule, ask your coding agent: "Queue a sample of my agent's live runs for people to correct." For reference, the code looks like this: - **TypeScript** `judging-rule.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); const project = await kindgi.projects.getDefault(); const rule = await kindgi.projects.judgingRules.create(project.id, { name: 'Live refund runs', when: { agentIds: ['acme.refunds'], versions: ['live'] }, sample: 0.05, // 5% of them maxOpen: 20, // at most 20 waiting at once }); console.log(rule.ruleId, rule.version); ``` - **Python** `judging_rule.py` ```python from kindgi.client import Kindgi kindgi = Kindgi() project = kindgi.projects.get_default() rule = kindgi.projects.judging_rules.create( project.id, name="Live refund runs", when={"agent_ids": ["acme.refunds"], "versions": ["live"]}, sample=0.05, # 5% of them max_open=20, # at most 20 waiting at once ) print(rule.rule_id, rule.version) ``` - **Java** `src/main/java/com/acme/app/JudgingRule.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.JudgingRuleSpec; import com.kindgi.client.models.JudgingRuleWhen; import com.kindgi.client.models.Project; import java.util.List; public final class JudgingRule { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev Project project = kindgi.projects().getDefault(); var rule = kindgi.projects().judgingRules().create(project.id(), JudgingRuleSpec.builder() .name("Live refund runs") .when(JudgingRuleWhen.builder().agentIds(List.of("acme.refunds")).versions(List.of("live")).build()) .sample(0.05) // 5% of them .maxOpen(20L) // at most 20 waiting at once .build()); System.out.println(rule.ruleId() + " " + rule.version()); } } ``` ```text 0867c21c-c911-4d68-9fca-bf3234768931 1 ``` - **Scala** `src/main/scala/acme/app/JudgingRule.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.models.{JudgingRuleSpec, JudgingRuleWhen} import scala.jdk.CollectionConverters._ object JudgingRule { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val project = kindgi.projects().getDefault() val rule = kindgi.projects().judgingRules().create(project.id, JudgingRuleSpec.builder() .name("Live refund runs") .when(JudgingRuleWhen.builder().agentIds(List("acme.refunds").asJava).versions(List("live").asJava).build()) .sample(0.05) // 5% of them .maxOpen(20L) // at most 20 waiting at once .build()) println(s"${rule.ruleId} ${rule.version}") } } ``` ```text d57204bd-e0d2-4a2c-ade9-d87b83761e41 1 ``` * `when` is which runs it matches: here, runs of `acme.refunds`'s live version. * `sample` is the share of them to queue, `0.05` for 5%. * `maxOpen` stops the rule adding runs while that many of its runs wait. It prints the rule's id and its version, `1`. A rule as the runtime returns it, here one made with `sample: 1` and `maxOpen: 2`: ```json { "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53", "projectId": "153a0091-613e-4c34-9535-da8194f9068e", "version": 1, "name": "Live refund runs", "when": { "agentIds": [ "acme.refunds" ], "versions": [ "live" ] }, "sample": 1, "maxOpen": 2, "enabled": true, "createdBy": "user:6161fd71-761c-4b92-be40-153923bad755", "createdAt": "2026-10-10T02:44:01.075Z" } ``` From now on, the runs it matches join the queue as they end. It never runs a model or spends anything. [Judging rules](https://docs.kindgi.com/v0.3.0/guides/evals/judging-reference.md#judging-rules) has every field, and how to try a rule before you make it. ### Work through the queue The queue lists the runs waiting for a correction, oldest first. To list the ones you can close, ask your coding agent: "Show me the runs waiting for my correction." For reference, the code looks like this: - **TypeScript** `judging-queue.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); const project = await kindgi.projects.getDefault(); const waiting = await kindgi.projects.judgingQueue.list(project.id, { forMe: true }); console.log(waiting.total, waiting.data.length); ``` - **Python** `judging_queue.py` ```python from kindgi.client import Kindgi kindgi = Kindgi() project = kindgi.projects.get_default() waiting = kindgi.projects.judging_queue.list(project.id, for_me=True) print(waiting.total, len(waiting.data)) ``` - **Java** `src/main/java/com/acme/app/JudgingQueue.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.JudgingQueuePage; import com.kindgi.client.models.Project; import com.kindgi.client.resources.ProjectsJudgingQueueListParams; public final class JudgingQueue { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev Project project = kindgi.projects().getDefault(); JudgingQueuePage waiting = kindgi.projects().judgingQueue() .list(project.id(), ProjectsJudgingQueueListParams.builder().forMe(true).build()); System.out.println(waiting.total() + " " + waiting.data().size()); } } ``` ```text 0 0 ``` - **Scala** `src/main/scala/acme/app/JudgingQueue.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.resources.ProjectsJudgingQueueListParams object JudgingQueue { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val project = kindgi.projects().getDefault() val waiting = kindgi.projects().judgingQueue() .list(project.id, ProjectsJudgingQueueListParams.builder().forMe(true).build()) println(s"${waiting.total} ${waiting.data.size}") } } ``` ```text 0 0 ``` `forMe: true` keeps the items you can close: those that want a class you may correct as, or any correction. It prints how many such items wait in all, and how many are on this page. One item, as that rule queued it: ```json { "runId": "4d5fb3c0-6835-436d-ba29-fd182661d189", "projectId": "153a0091-613e-4c34-9535-da8194f9068e", "agentId": "acme.refunds", "agentVersion": "2.1.0", "flowId": "agent.turn", "runStatus": "completed", "completedAt": "2026-10-10T02:44:01.450Z", "rules": [ { "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53", "version": 1 } ], "wantedClassIds": [], "anyJudgment": true, "progress": { "total": 0, "byClass": [] }, "addedAt": "2026-10-10T02:44:01.447Z", "state": "open", "can": { "dismiss": true, "reopen": false } } ``` Correct each item's run, as [Correct an item](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md#correct-an-item) shows. An item closes as `judged` once each rule that queued it has the correction it wants. `judgingQueue.dismiss`, with an optional reason, takes a run not worth correcting out of the queue. [Answers waiting for a correction](https://docs.kindgi.com/v0.3.0/guides/evals/judging-reference.md#answers-waiting-for-a-correction) has every field and count. Reading the rules and the queue takes read access to the project. Changing a rule, and dismissing or reopening an item, take write access, as correcting a run does. ## Next - [Build test cases from corrections](https://docs.kindgi.com/v0.3.0/guides/evals/build-a-test-set.md) - [Try a version on the test cases](https://docs.kindgi.com/v0.3.0/guides/evals/compare-an-agent-version.md) - [Corrections reference](https://docs.kindgi.com/v0.3.0/guides/evals/judging-reference.md)