# Judge a run's output > Record a yes or no, with a reason, on each item of a finished run's output, from your app, the CLI or the console, and weight judges by class. This guide records people's yes or no on what a run returned, weights some judges more than others, and queues the runs that need judging. A **judgment** is a yes or no, with an optional reason, about one item of a finished run's output: the whole answer, or one element of a list it returned. People who know the work, your users or your experts, judge. The judgments then tell you how an agent or flow version does: you [build a test set](https://docs.kindgi.com/v0.1.10/guides/evals/build-a-test-set.md) from them, and [compare a new version](https://docs.kindgi.com/v0.1.10/guides/evals/compare-an-agent-version.md) on it. [Judging reference](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md) has every field, rule and console page. ## Before you start - **A finished run to judge,** and its id, on a runtime such as `kindgi dev`, with Kindgi 0.1's CLI. [List runs](https://docs.kindgi.com/v0.1.10/guides/runs/list-runs.md) finds one. - **Write access to the run's project,** when authorization is on. - **For the app's half:** Node 22.12 or later, Python 3.11 or later, or Java 17 or later, with Kindgi 0.1's client, and `KINDGI_API_URL` and `KINDGI_API_TOKEN` set. ## Judge an item A judgment records whether one item of the run's output was right. From the CLI, judge the run's answer: - **TypeScript** ```sh pnpm exec kindgi judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` - **Python** ```sh uv run kindgi judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` - **Java** ```sh ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` - **Scala** ```sh ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --reason="Echoes the message back" ``` * `--run=` is the run's id. * `--item=answer` is your id for the item: `answer` for the whole answer. * `--pointer=/response/content` is where the item is in the run's output, as a JSON Pointer. * `--yes` is the verdict, or `--no`. * `--reason` (optional) says why. It prints the judgment: ```json excerpt { "id": "512df8f6-6e27-4c4a-934e-1b345dfc510a", "projectId": "d4910d76-7355-4022-8f3c-514361cfa986", "runId": "bc6d1b53-3562-48c9-9f29-719779e3d110", "subject": { "kind": "agent", "id": "acme-ops.echo-agent", "version": "0.1.0" }, "item": { "key": "answer", "pointer": "/response/content" }, "verdict": "yes", "reason": "Echoes the message back", "assertedBy": { "kind": "user", "id": "26fc414a-b49f-4e1f-a022-162865770178" }, "createdAt": "2026-10-06T04:08:27.667Z", … } ``` From your app, when your users judge in your own UI, ask your coding agent: "I want to record my user's yes or no on a run's answer." For reference, the script looks like this, with the run's id as its argument: - **TypeScript** `judge.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const judgment = await kindgi.judgments.create({ runId: process.argv[2], item: { key: 'answer', pointer: '/response/content' }, verdict: 'yes', reason: 'Echoes the message back', participantId: 'customer-user-17', // your id for the user who judged, if you judge for your users }); console.log(judgment.id, judgment.verdict, judgment.assertedBy); ``` - **Python** `judge.py` ```python import sys from kindgi.client import Kindgi kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev judgment = kindgi.judgments.create( run_id=sys.argv[1], item={"key": "answer", "pointer": "/response/content"}, verdict="yes", reason="Echoes the message back", participant_id="customer-user-18", # your id for the user who judged, if you judge for your users ) print(judgment.id, judgment.verdict, judgment.asserted_by) ``` - **Java** `src/main/java/com/acme/app/Judge.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.CreateJudgmentBody; import com.kindgi.client.models.JudgedItem; import com.kindgi.client.models.Judgment; public final class Judge { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev Judgment judgment = kindgi.judgments().create(CreateJudgmentBody.builder() .runId(args[0]) .item(JudgedItem.builder().key("answer").pointer("/response/content").build()) .verdict(CreateJudgmentBody.Verdict.YES) .reason("Echoes the message back") .participantId("customer-user-19") // your id for the user who judged, if you judge for your users .build()); System.out.println(judgment.id() + " " + judgment.verdict() + " " + judgment.assertedBy()); } } ``` ```text 42a5cdac-a5b4-4bf1-a777-d46ead4ec743 yes JudgmentAssertedBy[kind=user, id=ebdba929-5422-4cb2-b6bb-2c046109feb8] ``` - **Scala** `src/main/scala/acme/app/Judge.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.models.{CreateJudgmentBody, JudgedItem} object Judge { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val judgment = kindgi.judgments().create(CreateJudgmentBody.builder() .runId(args(0)) .item(JudgedItem.builder().key("answer").pointer("/response/content").build()) .verdict(CreateJudgmentBody.Verdict.YES) .reason("Echoes the message back") .participantId("customer-user-20") // your id for the user who judged, if you judge for your users .build()) println(s"${judgment.id} ${judgment.verdict} ${judgment.assertedBy}") } } ``` ```text 339dbf38-f4b0-4a9a-a1f6-0dc4585dae93 yes JudgmentAssertedBy[kind=user, id=ebdba929-5422-4cb2-b6bb-2c046109feb8] ``` * **`participantId`** is your app's own opaque id for the user who judged. Who called comes from the token, as `assertedBy`. * **Judging the same item again,** as the same caller and participant, replaces the earlier judgment, and the earlier one stays in the history. The run now has the judgment. Only a finished run can be judged, and the pointer must find something. [Judgments](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md#judgments) has the rules and the refusals. In the console, a finished run's page has **Judge this run**, or a **Judgments** tab once it has judgments. It lists the answer and each element of a list in the output, each with **Yes** and **No** and the counts so far. It lists each judgment with who made it, its class and its reason. ## See the judgments A run's judgments say how its output was judged, and by whom: - **TypeScript** ```sh pnpm exec kindgi judgments list --run= # newest first; replaced and removed ones are left out pnpm exec kindgi judgments show # with copies of the run's input and output, and the item's value pnpm exec kindgi judgments remove ``` - **Python** ```sh uv run kindgi judgments list --run= # newest first; replaced and removed ones are left out uv run kindgi judgments show # with copies of the run's input and output, and the item's value uv run kindgi judgments remove ``` - **Java** ```sh ./kindgiw judgments list --run= # newest first; replaced and removed ones are left out ./kindgiw judgments show # with copies of the run's input and output, and the item's value ./kindgiw judgments remove ``` - **Scala** ```sh ./kindgiw judgments list --run= # newest first; replaced and removed ones are left out ./kindgiw judgments show # with copies of the run's input and output, and the item's value ./kindgiw judgments remove ``` * `` is the run's id, and `` a judgment's `id`, from the list. * `list` also filters by agent and version, flow, verdict, judge class, participant and project. `list` prints a page of judgments as JSON: `data`, `hasMore`, and `nextCursor` when there's another page. `show` prints one judgment, and `remove` prints its id with `"removed": true`. A judgment keeps copies of the run's input and output, so it outlives the run's own retention. The clients have the same: `judgments.list`, `judgments.get` and `judgments.unregister`. ## Weight judges by class Without a class, every judgment counts once. A **judge class** counts some judges more, for the tenant, a project, or an agent in a project. Add a class, then judge as it: - **TypeScript** ```sh pnpm exec kindgi judge-classes add --name=expert --weight=3 --project= pnpm exec kindgi judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` - **Python** ```sh uv run kindgi judge-classes add --name=expert --weight=3 --project= uv run kindgi judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` - **Java** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --project= ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` - **Scala** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --project= ./kindgiw judgments add --run= --item=answer --pointer=/response/content --yes --class= ``` * `--name` names the class, and `--weight=3` counts its judgments three times. * `--project=` is the project it applies to. [Build the test set](https://docs.kindgi.com/v0.1.10/guides/evals/build-a-test-set.md#build-the-test-set) shows how to find a project's id. * `--class=` is the class's `id`, which `add` printed. In the clients, it's `judgeClassId`. The judgment now counts with the class's weight. ### Restrict who may judge as a class A restricted class says who may judge as it, so the people using your app can't move the figures that release a version. Add the class with a restriction, before your reviewers judge: - **TypeScript** ```sh pnpm exec kindgi judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` - **Python** ```sh uv run kindgi judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` - **Java** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` - **Scala** ```sh ./kindgiw judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads" ``` * `--tenant` makes it the tenant's class. * `--min-reviewer-role=senior` lets only a [reviewer](https://docs.kindgi.com/v0.1.10/guides/approvals/decide-an-approval.md#reviewers-and-roles) of that role or above judge as it: `standard`, `senior` or `admin`. * `--description` says who it is for. * `--principal-kind=user|service` and `--principal-id=`, which repeat, restrict it to people or service tokens, or to named ones. A caller must meet each restriction you set. One who doesn't is refused with `403`, and nothing is recorded: ```text Error [auth]: You can't judge as "expert": it needs a senior reviewer or above; you're a standard reviewer. Error [auth]: You can't judge as "tokens": only service tokens may assert it. ``` A judgment made under a restriction carries `"restricted": true`. It's decided when the judgment is made, so judgments recorded before the restriction don't count as restricted. [Judge classes](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md#judge-classes) has the rest. ## Queue the runs to judge **Judging rules** pick the project's runs that need a person's judgment, so your experts don't pick runs one at a time. As a run of the project ends, each rule checks it. A run that any rule matches joins the project's **judging queue**, once. To make a rule, ask your coding agent: "Queue a sample of my agent's live runs for people to judge." For reference, the code looks like this: - **TypeScript** `judging-rule.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); const project = await kindgi.projects.getDefault(); const rule = await kindgi.projects.judgingRules.create(project.id, { name: 'Live refund runs', when: { agentIds: ['acme.refunds'], versions: ['live'] }, sample: 0.05, // 5% of them maxOpen: 20, // at most 20 waiting at once }); console.log(rule.ruleId, rule.version); ``` - **Python** `judging_rule.py` ```python from kindgi.client import Kindgi kindgi = Kindgi() project = kindgi.projects.get_default() rule = kindgi.projects.judging_rules.create( project.id, name="Live refund runs", when={"agent_ids": ["acme.refunds"], "versions": ["live"]}, sample=0.05, # 5% of them max_open=20, # at most 20 waiting at once ) print(rule.rule_id, rule.version) ``` - **Java** `src/main/java/com/acme/app/JudgingRule.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.JudgingRuleSpec; import com.kindgi.client.models.JudgingRuleWhen; import com.kindgi.client.models.Project; import java.util.List; public final class JudgingRule { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev Project project = kindgi.projects().getDefault(); var rule = kindgi.projects().judgingRules().create(project.id(), JudgingRuleSpec.builder() .name("Live refund runs") .when(JudgingRuleWhen.builder().agentIds(List.of("acme.refunds")).versions(List.of("live")).build()) .sample(0.05) // 5% of them .maxOpen(20L) // at most 20 waiting at once .build()); System.out.println(rule.ruleId() + " " + rule.version()); } } ``` ```text 0867c21c-c911-4d68-9fca-bf3234768931 1 ``` - **Scala** `src/main/scala/acme/app/JudgingRule.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.models.{JudgingRuleSpec, JudgingRuleWhen} import scala.jdk.CollectionConverters._ object JudgingRule { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val project = kindgi.projects().getDefault() val rule = kindgi.projects().judgingRules().create(project.id, JudgingRuleSpec.builder() .name("Live refund runs") .when(JudgingRuleWhen.builder().agentIds(List("acme.refunds").asJava).versions(List("live").asJava).build()) .sample(0.05) // 5% of them .maxOpen(20L) // at most 20 waiting at once .build()) println(s"${rule.ruleId} ${rule.version}") } } ``` ```text d57204bd-e0d2-4a2c-ade9-d87b83761e41 1 ``` * `when` is which runs it matches: here, runs of `acme.refunds`'s live version. * `sample` is the share of them to queue, `0.05` for 5%. * `maxOpen` stops the rule adding runs while that many of its runs wait. It prints the rule's id and its version, `1`. A rule as the runtime returns it, here one made with `sample: 1` and `maxOpen: 2`: ```json { "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53", "projectId": "153a0091-613e-4c34-9535-da8194f9068e", "version": 1, "name": "Live refund runs", "when": { "agentIds": [ "acme.refunds" ], "versions": [ "live" ] }, "sample": 1, "maxOpen": 2, "enabled": true, "createdBy": "user:6161fd71-761c-4b92-be40-153923bad755", "createdAt": "2026-10-10T02:44:01.075Z" } ``` From now on, the runs it matches join the queue as they end. It never runs a model or spends anything. [Judging rules](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md#judging-rules) has every field, and how to try a rule before you make it. ### Work through the queue The queue lists the runs waiting for a judgment, oldest first. To list the ones you can close, ask your coding agent: "Show me the runs waiting for my judgment." For reference, the code looks like this: - **TypeScript** `judging-queue.ts` ```ts import { createClient } from '@kindgi/sdk/client'; const kindgi = createClient(); const project = await kindgi.projects.getDefault(); const waiting = await kindgi.projects.judgingQueue.list(project.id, { forMe: true }); console.log(waiting.total, waiting.data.length); ``` - **Python** `judging_queue.py` ```python from kindgi.client import Kindgi kindgi = Kindgi() project = kindgi.projects.get_default() waiting = kindgi.projects.judging_queue.list(project.id, for_me=True) print(waiting.total, len(waiting.data)) ``` - **Java** `src/main/java/com/acme/app/JudgingQueue.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.JudgingQueuePage; import com.kindgi.client.models.Project; import com.kindgi.client.resources.ProjectsJudgingQueueListParams; public final class JudgingQueue { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev Project project = kindgi.projects().getDefault(); JudgingQueuePage waiting = kindgi.projects().judgingQueue() .list(project.id(), ProjectsJudgingQueueListParams.builder().forMe(true).build()); System.out.println(waiting.total() + " " + waiting.data().size()); } } ``` ```text 0 0 ``` - **Scala** `src/main/scala/acme/app/JudgingQueue.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.resources.ProjectsJudgingQueueListParams object JudgingQueue { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val project = kindgi.projects().getDefault() val waiting = kindgi.projects().judgingQueue() .list(project.id, ProjectsJudgingQueueListParams.builder().forMe(true).build()) println(s"${waiting.total} ${waiting.data.size}") } } ``` ```text 0 0 ``` `forMe: true` keeps the items you can close: those that want a class you may judge as, or any judgment. It prints how many such items wait in all, and how many are on this page. One item, as that rule queued it: ```json { "runId": "4d5fb3c0-6835-436d-ba29-fd182661d189", "projectId": "153a0091-613e-4c34-9535-da8194f9068e", "agentId": "acme.refunds", "agentVersion": "2.1.0", "flowId": "agent.turn", "runStatus": "completed", "completedAt": "2026-10-10T02:44:01.450Z", "rules": [ { "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53", "version": 1 } ], "wantedClassIds": [], "anyJudgment": true, "progress": { "total": 0, "byClass": [] }, "addedAt": "2026-10-10T02:44:01.447Z", "state": "open", "can": { "dismiss": true, "reopen": false } } ``` Judge each item's run, as [Judge an item](https://docs.kindgi.com/v0.1.10/guides/evals/judge-a-runs-output.md#judge-an-item) shows. An item closes as `judged` once each rule that queued it has the judgment it wants. `judgingQueue.dismiss`, with an optional reason, takes a run not worth judging out of the queue. [The judging queue](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md#the-judging-queue) has every field and count. Reading the rules and the queue takes read access to the project. Changing a rule, and dismissing or reopening an item, take write access, as judging a run does. ## Next - [Build a test set](https://docs.kindgi.com/v0.1.10/guides/evals/build-a-test-set.md) - [Compare a version on a test set](https://docs.kindgi.com/v0.1.10/guides/evals/compare-an-agent-version.md) - [Judging reference](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md)