Skip to content
Kindgi

Judge a run's output

View as Markdown

This guide records people's yes or no on what a run returned, weights some judges more than others, and queues the runs that need judging.

A judgment is a yes or no, with an optional reason, about one item of a finished run's output: the whole answer, or one element of a list it returned. People who know the work, your users or your experts, judge. The judgments then tell you how an agent or flow version does: you build a test set from them, and compare a new version on it. Judging reference has every field, rule and console page.

Before you start

Section titled “Before you start”
  • A finished run to judge, and its id, on a runtime such as kindgi dev, with Kindgi 0.1's CLI. List runs finds one.
  • Write access to the run's project, when authorization is on.
  • For the app's half: Node 22.12 or later, Python 3.11 or later, or Java 17 or later, with Kindgi 0.1's client, and KINDGI_API_URL and KINDGI_API_TOKEN set.

Judge an item

Section titled “Judge an item”

A judgment records whether one item of the run's output was right.

From the CLI, judge the run's answer:

Terminal window
pnpm exec kindgi judgments add --run=<run-id> --item=answer --pointer=/response/content --yes --reason="Echoes the message back"
  • --run=<run-id> is the run's id.
  • --item=answer is your id for the item: answer for the whole answer.
  • --pointer=/response/content is where the item is in the run's output, as a JSON Pointer.
  • --yes is the verdict, or --no.
  • --reason (optional) says why.

It prints the judgment:

{
  "id": "512df8f6-6e27-4c4a-934e-1b345dfc510a",
  "projectId": "d4910d76-7355-4022-8f3c-514361cfa986",
  "runId": "bc6d1b53-3562-48c9-9f29-719779e3d110",
  "subject": { "kind": "agent", "id": "acme-ops.echo-agent", "version": "0.1.0" },
  "item": { "key": "answer", "pointer": "/response/content" },
  "verdict": "yes",
  "reason": "Echoes the message back",
  "assertedBy": { "kind": "user", "id": "26fc414a-b49f-4e1f-a022-162865770178" },
  "createdAt": "2026-10-06T04:08:27.667Z",
  …
}
Excerpt

From your app, when your users judge in your own UI, ask your coding agent: "I want to record my user's yes or no on a run's answer." For reference, the script looks like this, with the run's id as its argument:

judge.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev


const judgment = await kindgi.judgments.create({
  runId: process.argv[2],
  item: { key: 'answer', pointer: '/response/content' },
  verdict: 'yes',
  reason: 'Echoes the message back',
  participantId: 'customer-user-17', // your id for the user who judged, if you judge for your users
});
console.log(judgment.id, judgment.verdict, judgment.assertedBy);
  • participantId is your app's own opaque id for the user who judged. Who called comes from the token, as assertedBy.
  • Judging the same item again, as the same caller and participant, replaces the earlier judgment, and the earlier one stays in the history.

The run now has the judgment. Only a finished run can be judged, and the pointer must find something. Judgments has the rules and the refusals.

In the console, a finished run's page has Judge this run, or a Judgments tab once it has judgments. It lists the answer and each element of a list in the output, each with Yes and No and the counts so far. It lists each judgment with who made it, its class and its reason.

See the judgments

Section titled “See the judgments”

A run's judgments say how its output was judged, and by whom:

Terminal window
pnpm exec kindgi judgments list --run=<run-id>      # newest first; replaced and removed ones are left out
pnpm exec kindgi judgments show <judgment-id>       # with copies of the run's input and output, and the item's value
pnpm exec kindgi judgments remove <judgment-id>
  • <run-id> is the run's id, and <judgment-id> a judgment's id, from the list.
  • list also filters by agent and version, flow, verdict, judge class, participant and project.

list prints a page of judgments as JSON: data, hasMore, and nextCursor when there's another page. show prints one judgment, and remove prints its id with "removed": true.

A judgment keeps copies of the run's input and output, so it outlives the run's own retention. The clients have the same: judgments.list, judgments.get and judgments.unregister.

Weight judges by class

Section titled “Weight judges by class”

Without a class, every judgment counts once. A judge class counts some judges more, for the tenant, a project, or an agent in a project.

Add a class, then judge as it:

Terminal window
pnpm exec kindgi judge-classes add --name=expert --weight=3 --project=<project-id>
pnpm exec kindgi judgments add --run=<run-id> --item=answer --pointer=/response/content --yes --class=<judge-class-id>
  • --name names the class, and --weight=3 counts its judgments three times.
  • --project=<project-id> is the project it applies to. Build the test set shows how to find a project's id.
  • --class=<judge-class-id> is the class's id, which add printed. In the clients, it's judgeClassId.

The judgment now counts with the class's weight.

Restrict who may judge as a class

Section titled “Restrict who may judge as a class”

A restricted class says who may judge as it, so the people using your app can't move the figures that release a version.

Add the class with a restriction, before your reviewers judge:

Terminal window
pnpm exec kindgi judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads"
  • --tenant makes it the tenant's class.
  • --min-reviewer-role=senior lets only a reviewer of that role or above judge as it: standard, senior or admin.
  • --description says who it is for.
  • --principal-kind=user|service and --principal-id=<id>, which repeat, restrict it to people or service tokens, or to named ones.

A caller must meet each restriction you set. One who doesn't is refused with 403, and nothing is recorded:

Error [auth]: You can't judge as "expert": it needs a senior reviewer or above; you're a standard reviewer.
Error [auth]: You can't judge as "tokens": only service tokens may assert it.

A judgment made under a restriction carries "restricted": true. It's decided when the judgment is made, so judgments recorded before the restriction don't count as restricted. Judge classes has the rest.

Queue the runs to judge

Section titled “Queue the runs to judge”

Judging rules pick the project's runs that need a person's judgment, so your experts don't pick runs one at a time. As a run of the project ends, each rule checks it. A run that any rule matches joins the project's judging queue, once.

To make a rule, ask your coding agent: "Queue a sample of my agent's live runs for people to judge." For reference, the code looks like this:

judging-rule.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient();
const project = await kindgi.projects.getDefault();


const rule = await kindgi.projects.judgingRules.create(project.id, {
  name: 'Live refund runs',
  when: { agentIds: ['acme.refunds'], versions: ['live'] },
  sample: 0.05, // 5% of them
  maxOpen: 20, // at most 20 waiting at once
});
console.log(rule.ruleId, rule.version);
  • when is which runs it matches: here, runs of acme.refunds's live version.
  • sample is the share of them to queue, 0.05 for 5%.
  • maxOpen stops the rule adding runs while that many of its runs wait.

It prints the rule's id and its version, 1. A rule as the runtime returns it, here one made with sample: 1 and maxOpen: 2:

{
  "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "version": 1,
  "name": "Live refund runs",
  "when": {
    "agentIds": [
      "acme.refunds"
    ],
    "versions": [
      "live"
    ]
  },
  "sample": 1,
  "maxOpen": 2,
  "enabled": true,
  "createdBy": "user:6161fd71-761c-4b92-be40-153923bad755",
  "createdAt": "2026-10-10T02:44:01.075Z"
}

From now on, the runs it matches join the queue as they end. It never runs a model or spends anything. Judging rules has every field, and how to try a rule before you make it.

Work through the queue

Section titled “Work through the queue”

The queue lists the runs waiting for a judgment, oldest first. To list the ones you can close, ask your coding agent: "Show me the runs waiting for my judgment." For reference, the code looks like this:

judging-queue.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient();
const project = await kindgi.projects.getDefault();


const waiting = await kindgi.projects.judgingQueue.list(project.id, { forMe: true });
console.log(waiting.total, waiting.data.length);

forMe: true keeps the items you can close: those that want a class you may judge as, or any judgment. It prints how many such items wait in all, and how many are on this page. One item, as that rule queued it:

{
  "runId": "4d5fb3c0-6835-436d-ba29-fd182661d189",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "agentId": "acme.refunds",
  "agentVersion": "2.1.0",
  "flowId": "agent.turn",
  "runStatus": "completed",
  "completedAt": "2026-10-10T02:44:01.450Z",
  "rules": [
    {
      "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
      "version": 1
    }
  ],
  "wantedClassIds": [],
  "anyJudgment": true,
  "progress": {
    "total": 0,
    "byClass": []
  },
  "addedAt": "2026-10-10T02:44:01.447Z",
  "state": "open",
  "can": {
    "dismiss": true,
    "reopen": false
  }
}

Judge each item's run, as Judge an item shows. An item closes as judged once each rule that queued it has the judgment it wants. judgingQueue.dismiss, with an optional reason, takes a run not worth judging out of the queue. The judging queue has every field and count.

Reading the rules and the queue takes read access to the project. Changing a rule, and dismissing or reopening an item, take write access, as judging a run does.

Next

Section titled “Next”