Judge a run's output

A judgment is a yes or no, with an optional reason, about one item of a finished run's output: the whole answer, or one element of a list it returned. People who know the work, your users or your experts, judge; the judgments then tell you how an agent or flow version does: build a test set from them, and compare a new version on it.

Judge an item

From the CLI, on the run's answer:

kindgi judgments add --run=<run-id> --item=answer --pointer=/response/content --yes --reason="Echoes the message back"

{
  "id": "512df8f6-6e27-4c4a-934e-1b345dfc510a",
  "projectId": "d4910d76-7355-4022-8f3c-514361cfa986",
  "runId": "bc6d1b53-3562-48c9-9f29-719779e3d110",
  "subject": { "kind": "agent", "id": "acme-ops.echo-agent", "version": "0.1.0" },
  "item": { "key": "answer", "pointer": "/response/content" },
  "verdict": "yes",
  "reason": "Echoes the message back",
  "assertedBy": { "kind": "user", "id": "26fc414a-b49f-4e1f-a022-162865770178" },
  "createdAt": "2026-10-06T04:08:27.667Z",
  …
}

From your app, when your users judge in your own UI:

TypeScript

judge.ts

import { createClient } from '@kindgi/sdk/client';

const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev

const judgment = await kindgi.judgments.create({
  runId: process.argv[2],
  item: { key: 'answer', pointer: '/response/content' },
  verdict: 'yes',
  reason: 'Echoes the message back',
  participantId: 'customer-user-17', // your id for the user who judged, if you judge for your users
});
console.log(judgment.id, judgment.verdict, judgment.assertedBy);

Python

judge.py

import sys

from kindgi.client import Kindgi

kindgi = Kindgi()  # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev

judgment = kindgi.judgments.create(
    run_id=sys.argv[1],
    item={"key": "answer", "pointer": "/response/content"},
    verdict="yes",
    reason="Echoes the message back",
    participant_id="customer-user-18",  # your id for the user who judged, if you judge for your users
)
print(judgment.id, judgment.verdict, judgment.asserted_by)

• item.key is your stable id for the item: answer for the whole answer, or, for a list, the id of the element (a matched case's id, say).
• item.pointer (optional) is where the item is in the run's output, as a JSON Pointer such as /matches/2. Its value is kept with the judgment. item.rank (optional) is its place in a ranked list, 0 first.
• Who judged comes from the token: assertedBy. An app that judges for its own users adds participantId, its own opaque id for the user.
• Judging the same item again, as the same caller and participant, under any class or none, replaces the earlier judgment; the earlier one stays in the history. So each person has one verdict per item, and its class only says how much it counts.

Only a finished run can be judged, and the pointer must find something:

Run "fc26b436-611b-4962-9913-2f2669ade7e1" has no output to judge yet (status suspended).
Nothing at "/no/such" in run "bc6d1b53-3562-48c9-9f29-719779e3d110"'s output.

Judging a run takes write access to its project, as cancelling it does.

In the console, a finished run's page has Judge this run (a Judgments tab once it has judgments). It lists the answer and each element of a list in the output, each with Yes and No, the counts so far, and each judgment with who made it, its class and its reason.

See the judgments

kindgi judgments list --run=<run-id>      # newest first; replaced and removed ones are left out
kindgi judgments show <judgment-id>       # with copies of the run's input and output, and the item's value
kindgi judgments remove <judgment-id>

A judgment keeps copies of the run's input and output, so it outlives the run's own retention. list also filters by agent and version, flow, verdict, judge class and participant, and by project. The clients have the same: judgments.list, judgments.get and judgments.unregister.

Weight judges by class

Without a class, every judgment counts once. To count some judges more, add a judge class, for the tenant, a project, or an agent in a project:

kindgi judge-classes add --name=expert --weight=3 --project=<project-id>
kindgi judgments add --run=<run-id> --item=answer --pointer=/response/content --yes --class=<judge-class-id>

A judgment made with --class (judgeClassId in the clients) counts with that class's weight. The project's id: see Build a test set.

Restrict who may judge as a class

A class can say who may judge as it, so the people using your app can't move the figures that release a version:

kindgi judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads"

• --min-reviewer-role: standard, senior or admin, a reviewer of that role or above;
• --principal-kind: user (people) or service (service tokens);
• --principal-id: named users or tokens, by id.

The last two repeat; a caller must meet each one you set (assertableBy in the API and the clients). One who doesn't is refused with 403, and nothing is recorded:

Error [auth]: You can't judge as "expert": it needs a senior reviewer or above; you're a standard reviewer.
Error [auth]: You can't judge as "tokens": only service tokens may assert it.

Whom a class names is for the admins of its scope: anyone else who reads the class gets how many it names (principalCount), not their ids.

A judgment made under a restriction carries "restricted": true (in kindgi judgments list too). It's decided when the judgment is made, so:

• Restrict the class before your reviewers judge. Judgments recorded before the restriction don't count as restricted.
• Lifting a restriction (kindgi judge-classes set <judge-class-id> --unrestricted) keeps the earlier judgments restricted; new ones aren't.

kindgi judge-classes set with restriction flags replaces the whole restriction. A test set counts restricted judgments apart, and a comparison can count only them: see Compare a version on a test set.

Queue the runs to judge

Rather than have your experts pick runs one at a time, give the project judging rules: which of its runs need a person's judgment. As a run of the project ends, each rule checks it, and a run that any rule matches joins the project's judging queue, once.

TypeScript

judging-rule.ts

import { createClient } from '@kindgi/sdk/client';

const kindgi = createClient();
const project = await kindgi.projects.getDefault();

const rule = await kindgi.projects.judgingRules.create(project.id, {
  name: 'Live refund runs',
  when: { agentIds: ['acme.refunds'], versions: ['live'] },
  sample: 0.05, // 5% of them
  maxOpen: 20, // at most 20 waiting at once
});
console.log(rule.ruleId, rule.version);

Python

judging_rule.py

from kindgi.client import Kindgi

kindgi = Kindgi()
project = kindgi.projects.get_default()

rule = kindgi.projects.judging_rules.create(
    project.id,
    name="Live refund runs",
    when={"agent_ids": ["acme.refunds"], "versions": ["live"]},
    sample=0.05,  # 5% of them
    max_open=20,  # at most 20 waiting at once
)
print(rule.rule_id, rule.version)

A rule as the runtime returns it, its version 1, here one made with sample: 1 and maxOpen: 2:

{
  "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "version": 1,
  "name": "Live refund runs",
  "when": {
    "agentIds": [
      "acme.refunds"
    ],
    "versions": [
      "live"
    ]
  },
  "sample": 1,
  "maxOpen": 2,
  "enabled": true,
  "createdBy": "user:6161fd71-761c-4b92-be40-153923bad755",
  "createdAt": "2026-10-10T02:44:01.075Z"
}

What a rule matches (when, each field narrowing it):

• agentIds or flowIds: runs of these agents or flows (one or the other, not both). A rule naming an agent matches the agent's own runs, not its turns as a step of a flow.
• versions: these versions exactly, or live: runs that ran the agent's live version rather than one the caller named, so the rule follows each release.
• status: how the run ended. Only completed for now, since only a completed run can be judged; it's also the default.
• includeDryRuns: dry runs are left out unless this is true. A comparison's runs on a test set are never queued.

And how much of it to queue:

• sample: the share of matching runs, 0.05 for 5%. The same run and rule always give the same answer, across the rule's versions too: raising it later keeps the runs it took before, and adds more.
• maxOpen: the rule adds nothing while this many of its runs wait, and starts again as they're judged.
• judgeClassId: whose judgment the rule wants, a judge class. Without it, any judgment does.

A rule applies to runs that end after it's made. To try one before you make it, judgingRules.preview takes the same fields and answers how many of the project's last 100 runs (last, up to 500) it would have queued, with its sample but not its maxOpen. Give an existing rule's ruleId to sample as it does; without one, the count is an estimate. Runs from before the runtime recorded how their version was chosen never match live.

A rule only lists runs: it never runs a model or spends anything.

Work through the queue

judgingQueue.list gives the queue, oldest first:

TypeScript

const waiting = await kindgi.projects.judgingQueue.list(project.id, { forMe: true });

Python

waiting = kindgi.projects.judging_queue.list(project.id, for_me=True)

One item, as that rule queued it:

{
  "runId": "4d5fb3c0-6835-436d-ba29-fd182661d189",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "agentId": "acme.refunds",
  "agentVersion": "2.1.0",
  "flowId": "agent.turn",
  "runStatus": "completed",
  "completedAt": "2026-10-10T02:44:01.450Z",
  "rules": [
    {
      "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
      "version": 1
    }
  ],
  "wantedClassIds": [],
  "anyJudgment": true,
  "progress": {
    "total": 0,
    "byClass": []
  },
  "addedAt": "2026-10-10T02:44:01.447Z",
  "state": "open",
  "can": {
    "dismiss": true,
    "reopen": false
  }
}

• Each item is a run: its agent or flow and version, how it ended, the rules that queued it (each at the version that did), the judge classes they want, and the judgments on it so far, by class. It carries none of the run's content: open the run to judge it.
• forMe: true keeps the items you can close: those that want a class you may judge as, or any judgment.
• An item closes as judged once each rule that queued it has the judgment it wants: one of its class, or any judgment for a rule without a class.
• judgingQueue.dismiss, with an optional reason, takes a run not worth judging out of the queue, and judgingQueue.reopen puts it back. Each item's can says whether you may.
• Counts: limit: 0 answers only the total. With state, addedAfter and closedAfter, it gives "N waiting, M judged this week".
• When a run's content is gone, erased or purged by retention, its waiting item closes as erased and shows nothing of the run.

Reading the rules and the queue takes read access to the project. Changing a rule, and dismissing or reopening an item, take write access, as judging a run does.

Change a rule, and read its results

• judgingRules.update writes a new version of the rule, and judgingRules.versions lists them all, newest first, with who wrote each and when. Runs queued from then on count toward the new version. null removes maxOpen or judgeClassId.
• judgingRules.unregister stops a rule. It queues nothing more; its versions and the runs it queued stay.
• judgingRules.results gives a group for each version of the rule and each agent version: the runs it queued (waiting, judged, dismissed, erased), the judgments on them with their weighted yes share, and the judgments by class.
  • The yes share is per judgment: one person's verdict on one item of a run's output.
  • skippedByCap counts the runs the rule matched and sampled but didn't queue because maxOpen were waiting, even when another rule queued them, so added + skippedByCap is every run it matched and sampled. When it isn't 0, the queued runs lean toward quiet times.
  • Never pooled: each version of each rule is a sample of its own when, so its results are never added to another's.

That rule's results (capped at 2 waiting) after three live runs: one judged yes, one dismissed, and one skipped while both waited.

{
  "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
  "groups": [
    {
      "ruleVersion": 1,
      "agentVersion": "2.1.0",
      "added": 2,
      "open": 0,
      "judged": 1,
      "dismissed": 1,
      "erased": 0,
      "skippedByCap": 1,
      "judgments": 1,
      "yesShare": 1,
      "byClass": [
        {
          "judgeClassId": null,
          "judgments": 1,
          "yes": 1
        }
      ]
    }
  ]
}

In the console

A project's Quality section is the judging queue and its rules in the console. Its headline says where judging stands:

5 runs wait for a judgment (3 for you) · 1 judged this week · 1 dismissed this week · 7 added this week · 0 proposals to review.

When a proposal waits for a reviewer, "· 1 waiting for a reviewer" follows. The section has five tabs: Waiting, Rules, Trends, Judge classes and Proposals.

Waiting

Each row is a run in the queue:

• What ran: the agent or flow and its version, and when it finished. It links the run's Judgments tab, where you judge it (see Judge an item).
• Why it's here: the rules that queued it.
• Where it stands: what it still needs ("Needs an expert judgment (1 so far, unclassified)"), or how a closed one ended.

Filter it by state (Waiting, Judged, Dismissed, Erased), by For me or Everyone's ("2 more wait for other judges."), by rule and by agent. Dismiss takes a run out of the queue with a reason, and Reopen puts a dismissed one back.

Rules

Each rule is in words:

• Which runs: "Completed runs of Dev-Echo Agent", and how many it takes ("Takes every matching run", or "Takes 1 in 10").
• What it wants: "An expert judgment", "Any judgment" or "A bot judgment".
• Its cap: "No cap", or no new items while that many are open.
• What it has open now: for example "3 open", linking Waiting.

A rule matches an agent's or a flow's own runs, not an agent's turns as a flow's step; replays never match.

Project editors see New rule, and Change on each rule. The editor shows what the rule would have queued from the recent runs, and saving makes a new version of the rule. History lists what each version changed, who changed it and when. Unregister stops a rule; the runs it queued stay in the queue.

Trends

Trends shows, per rule and by its latest version, the share of yes among people's judgments of each agent version, over all time or the last 90 or 30 days. These are judgingRules.results' numbers:

• A version shows from 10 judgments. Below that it says so: "1.0.0: too few judgments to say (2 so far)".
• Each line names its judgments and its range: "1.1.0: about 95% yes, from 12 judgments (12 runs judged so far); range 64–100%".
• A comparison only when the ranges are apart: "1.1.0 is judged better than 1.0.0."
• The weights, in words: "Weighted by judge class: An expert's judgment counts 3× an unclassified one; a bot's judgment counts the same as an unclassified one."
• What the numbers leave out: an earlier version of the rule is "shown apart, never added in", and runs the cap kept out are counted ("1 matched run wasn't queued because the rule already had its most open runs waiting, so the queued ones lean toward quieter times.").

Every count opens the counts behind a rule's trend.

Judge classes

Each judge class that applies to the project's runs, in words:

• how much it counts ("Counts 3× an unclassified one");
• where it applies ("This project");
• who may give it ("Only people", "Only services"); who may give a restricted class is counted here, never listed;
• the rules that want it.

Project admins add, change and remove a project's classes (a tenant's classes: tenant admins). Removing a class names the rules that want it.

Proposals

The project's agents' proposals, grouped by what they wait for: To review, Waiting for a reviewer, Being evaluated, Need a look and Closed. Each says where it stands ("Not better on the test set", with the judged yes share it was evaluated on) and links its page: "Each proposal's page holds its actions: evaluate, request, roll back, withdraw."