# Judging reference

> Judgments and their items, judge classes and restrictions, judging rules, the judging queue, rule results, and the console's Quality section.

This page lists everything about judging runs: what a judgment holds, how judge classes weigh it, what judging rules match, what the judging queue holds, how a rule's results count, and what the console's Quality section shows. To judge a run, follow [Judge a run's output](https://docs.kindgi.com/v0.1.10/guides/evals/judge-a-runs-output.md).

## Judgments

- **`item.key`** is your stable id for the item: `answer` for the whole answer, or, for a list, the id of the element, such as a matched case's id.
- **`item.pointer`** (optional) is where the item is in the run's output, as a JSON Pointer such as `/matches/2`. Its value is kept with the judgment.
- **`item.rank`** (optional) is its place in a ranked list, 0 first.
- **Who judged** comes from the token, as `assertedBy`. An app that judges for its own users adds `participantId`, its own opaque id for the user.
- **Judging the same item again,** as the same caller and participant, under any class or none, replaces the earlier judgment. The earlier one stays in the history. So each person has one verdict per item, and its class only says how much it counts.
- **A judgment keeps copies** of the run's input and output, so it outlives the run's own retention.
- **Judging a run takes write access** to its project, as cancelling it does.

Only a finished run can be judged, and the pointer must find something:

```text
Run "fc26b436-611b-4962-9913-2f2669ade7e1" has no output to judge yet (status suspended).
Nothing at "/no/such" in run "bc6d1b53-3562-48c9-9f29-719779e3d110"'s output.
```

## Judge classes

- **A class weighs its judgments:** `--weight=3` counts each three times. Without a class, a judgment counts once.

- **A class applies** to the tenant, a project, or an agent in a project.

- **A restriction says who may judge as the class:**

  - `--min-reviewer-role`: `standard`, `senior` or `admin`, a reviewer of that role or above;
  - `--principal-kind`: `user` for people, or `service` for service tokens;
  - `--principal-id`: named users or tokens, by id.

  The last two repeat. A caller must meet each one you set. In the API and the clients, it's `assertableBy`.

- **Whom a class names is for the admins of its scope.** Anyone else who reads the class gets how many it names, as `principalCount`, not their ids.

- **A judgment made under a restriction carries `"restricted": true`,** in `kindgi judgments list` too. It's decided when the judgment is made: judgments recorded before the restriction don't count as restricted.

- **Lifting a restriction,** with `kindgi judge-classes set <judge-class-id> --unrestricted`, keeps the earlier judgments restricted. New ones aren't.

- **`kindgi judge-classes set` with restriction flags replaces the whole restriction.**

A test set counts restricted judgments apart, and a comparison can count only them, as [Compare a version on a test set](https://docs.kindgi.com/v0.1.10/guides/evals/compare-an-agent-version.md#count-only-restricted-judges) shows.

## Judging rules

What a rule matches, in `when`, each field narrowing it:

- **`agentIds`** or **`flowIds`**: runs of these agents or flows, one or the other, not both. A rule naming an agent matches the agent's own runs, not its turns as a step of a flow.
- **`versions`**: these versions exactly, or `live`: runs that ran the agent's live version rather than one the caller named. So the rule follows each release.
- **`status`**: how the run ended. Only `completed`, since only a completed run can be judged. It's also the default.
- **`includeDryRuns`**: dry runs are left out unless this is `true`. A comparison's runs on a test set are never queued.

How much of it to queue:

- **`sample`**: the share of matching runs, `0.05` for 5%. The same run and rule always give the same answer, across the rule's versions too. So raising it later keeps the runs it took before, and adds more.
- **`maxOpen`**: the rule adds nothing while this many of its runs wait, and starts again as they're judged.
- **`judgeClassId`**: whose judgment the rule wants, a [judge class](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md#judge-classes). Without it, any judgment does.

A rule applies to runs that end after it's made. To try one before you make it, `judgingRules.preview` takes the same fields. It answers how many of the project's last 100 runs it would have queued, with its `sample` but not its `maxOpen`. `last` sets how many runs, up to 500. Give an existing rule's `ruleId` to sample as it does. Without one, the count is an estimate. A run that doesn't record how its version was chosen never matches `live`.

A rule only lists runs: it never runs a model or spends anything.

### Change a rule

- **`judgingRules.update`** writes a new version of the rule. Runs queued from then on count toward the new version. `null` removes `maxOpen` or `judgeClassId`.
- **`judgingRules.versions`** lists every version, newest first, with who wrote each and when.
- **`judgingRules.unregister`** stops a rule. It queues nothing more. Its versions and the runs it queued stay.

## The judging queue

`judgingQueue.list` gives the queue, oldest first.

- **Each item** is a run: its agent or flow and version, how it ended, the rules that queued it, each at the version that did, the judge classes they want, and the judgments on it so far, by class. It carries none of the run's content: open the run to judge it.
- **`forMe: true`** keeps the items you can close: those that want a class you may judge as, or any judgment.
- **An item closes** as `judged` once each rule that queued it has the judgment it wants: one of its class, or any judgment for a rule without a class.
- **`judgingQueue.dismiss`,** with an optional reason, takes a run not worth judging out of the queue. `judgingQueue.reopen` puts it back. Each item's `can` says whether you may.
- **Counts:** `limit: 0` answers only the `total`. With `state`, `addedAfter` and `closedAfter`, it gives "N waiting, M judged this week".
- **When a run's content is gone,** erased or purged by [retention](https://docs.kindgi.com/v0.1.10/deploy/retention.md), its waiting item closes as `erased` and shows nothing of the run.

Reading the rules and the queue takes read access to the project. Changing a rule, and dismissing or reopening an item, take write access, as judging a run does.

## A rule's results

`judgingRules.results` gives a group for each version of the rule and each agent version:

- **The runs it queued:** `added`, and of them `open`, `judged`, `dismissed` and `erased`.
- **The judgments on them,** with their weighted `yes` share, and the judgments by class.
- **The `yes` share is per judgment:** one person's verdict on one item of a run's output.
- **`runsWithJudgments`** counts the runs those judgments come from: the queued runs with at least one live judgment, whatever their item's state now. That's still open, closed as `judged`, or dismissed after someone judged it. `judgments` counts verdicts, so several people judging one run make several judgments but one run here. To see how much a share rests on, count runs, not judgments.
- **`skippedByCap`** counts the runs the rule matched and sampled but didn't queue because `maxOpen` were waiting, even when another rule queued them. So `added` + `skippedByCap` is every run it matched and sampled. When it isn't 0, the queued runs lean toward quiet times.
- **Never pooled:** each version of each rule is a sample of its own `when`, so its results are never added to another's.

Here a rule that wants an expert's judgment queued four runs. An expert, with weight 3, and a peer judged the first. A peer judged the second `no`, and it's still open. A peer judged the third, which was then dismissed. The fourth was never judged. So `judgments` is 4, `runsWithJudgments` 3, and `yesShare` 0.83: 5 of 6 weighted verdicts.

```json
{
  "ruleId": "fd7a1785-717b-404e-bcdc-3b819806d784",
  "groups": [
    {
      "ruleVersion": 1,
      "agentVersion": "1.0.0",
      "added": 4,
      "open": 2,
      "judged": 1,
      "dismissed": 1,
      "erased": 0,
      "skippedByCap": 0,
      "judgments": 4,
      "runsWithJudgments": 3,
      "yesShare": 0.8333333333333334,
      "byClass": [
        {
          "judgeClassId": "2da17046-eb37-4157-b175-3381a926545a",
          "judgments": 3,
          "yes": 2
        },
        {
          "judgeClassId": "787b6f6f-30cf-4bac-bf27-ac5ef32cb6d3",
          "judgments": 1,
          "yes": 1
        }
      ]
    }
  ]
}
```

## In the console

A project's **Quality** section is the judging queue and its rules in the console. Its headline says where judging stands:

> 5 runs wait for a judgment (3 for you) · 1 judged this week · 1 dismissed this week · 7 added this week · 0 proposals to review.

When a proposal waits for a reviewer, "· 1 waiting for a reviewer" follows. The section has five tabs: **Waiting**, **Rules**, **Trends**, **Judge classes** and **Proposals**. Their tables sort by a column's header, and **Columns** chooses which columns each shows. The tab, the filters and each table's view go in the page's address, such as `?tab=rules`, so a link opens the same view.

### Waiting

![The Quality section's Waiting tab: the headline, the filters, "2 more wait for other judges.", the Columns menu, and three runs of demo.echo-agent 1.0.0 queued by the Everything and Support sample rules, each needing an expert judgment, with Dismiss.](https://docs.kindgi.com/v0.1.10/_astro/quality-waiting.pei968gA_Z1Isgxo.webp)

Each row is a run in the queue:

- **What ran:** the agent or flow and its version, and when it finished. It links the run's **Judgments** tab, where you judge it, as [Judge an item](https://docs.kindgi.com/v0.1.10/guides/evals/judge-a-runs-output.md#judge-an-item) shows.
- **Why it's here:** the rules that queued it.
- **Where it stands:** what it still needs, such as "Needs an expert judgment (1 so far, unclassified)", or how a closed one ended.

Filter it by state, **Waiting**, **Judged**, **Dismissed** or **Erased**. Or by **For me** or **Everyone's** ("2 more wait for other judges."), by rule and by agent. **Dismiss** takes a run out of the queue with a reason, and **Reopen** puts a dismissed one back.

![The Waiting tab filtered to Dismissed and Everyone's: a run of demo.echo-agent 1.0.0 queued by the Everything and Support sample rules, "Dismissed by a person (11111111…) 2026-10-11 12 UTC: A test order", with Reopen.](https://docs.kindgi.com/v0.1.10/_astro/quality-dismissed.B0nXqL3g_kQFEX.webp)

### Rules

![The Rules tab: "3 judging rules." with New rule, a search box and the Columns menu, and the Support sample, Everything and Bot review rules, each with which runs it takes, what it wants, its cap, its open items, and History, Change and Unregister.](https://docs.kindgi.com/v0.1.10/_astro/quality-rules.Bgod2gHL_Z1KDGwq.webp)

Each rule is in words:

- **Which runs:** "Completed runs of Dev-Echo Agent", and how many it takes: "Takes every matching run", or "Takes 1 in 10".
- **What it wants:** "An expert judgment", "Any judgment" or "A bot judgment".
- **Its cap:** "No cap", or no new items while that many are open.
- **What it has open now:** for example "3 open", linking **Waiting**.

A rule matches an agent's or a flow's own runs, not an agent's turns as a flow's step. Replays never match.

Project editors see **New rule**, and **Change** on each rule. The editor shows what the rule would have queued from the recent runs, and saving makes a new version of the rule. **History** lists what each version changed, who changed it and when. **Unregister** stops a rule, and the runs it queued stay in the queue.

![The New judging rule dialog, over the Rules tab.](https://docs.kindgi.com/v0.1.10/_astro/quality-rule-editor.CKqkZZ30_Z17R8iU.webp)

![Echo half: its versions, over the Rules tab: "Version 2 · a person (11111111…) · 2026-10-11 12 UTC" changed "Cap: No new items while 5 are open → No cap" and "Wants: any judgment → an expert judgment"; Version 1, by the same person at the same time, "Created".](https://docs.kindgi.com/v0.1.10/_astro/quality-rule-history.I_Ri9-B2_Z25zHk1.webp)

### Trends

![The Trends tab: each rule by its latest version, with the share of yes for each agent version. The Trend check rule's chart shows 1.1.0 at about 95% yes and 1.0.0 at about 25%, "1.1.0 is judged better than 1.0.0."](https://docs.kindgi.com/v0.1.10/_astro/quality-trends.BUdVAGxX_Z2oKPCQ.webp)

**Trends** shows, per rule and by its latest version, the share of yes among people's judgments of each agent version, over all time or the last 90 or 30 days. These are [a rule's results](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md#a-rules-results):

- **A version shows from 10 judgments.** With fewer, it says so: "1.0.0: too few judgments to say (2 so far)".
- **Each line names its judgments and its range:** "1.1.0: about 95% yes, from 12 judgments (12 runs judged so far); range 64–100%".
- **A comparison only when the ranges are apart:** "1.1.0 is judged better than 1.0.0."
- **The weights, in words:** "Weighted by judge class: An expert's judgment counts 3× an unclassified one; a bot's judgment counts the same as an unclassified one."
- **What the numbers leave out:** an earlier version of the rule is "shown apart, never added in". Runs the cap kept out are counted: "1 matched run wasn't queued because the rule already had its most open runs waiting, so the queued ones lean toward quieter times."

**Every count** opens the counts behind a rule's trend.

![A rule's trend after the rule changed, now at rule version 2: too few judgments to chart yet, 1.2.0 with too few judgments to say, the one matched run the cap kept out, and rule version 1 shown apart, never added in.](https://docs.kindgi.com/v0.1.10/_astro/quality-trends-changed.BAMLhEHi_lPkVm.webp)

### Judge classes

![The Judge classes tab: "2 judge classes apply to this project's runs. An unclassified judgment counts 1." The expert class counts 3×, applies to this project, may be given only by people and is wanted by the Echo half and Support sample rules; the bot class counts the same, only services.](https://docs.kindgi.com/v0.1.10/_astro/quality-classes.DAi2J6os_22Gxn.webp)

Each [judge class](https://docs.kindgi.com/v0.1.10/guides/evals/judging-reference.md#judge-classes) that applies to the project's runs, in words:

- how much it counts, such as "Counts 3× an unclassified one";
- where it applies, such as "This project";
- who may give it, such as "Only people" or "Only services". Who may give a restricted class is counted here, never listed;
- the rules that want it.

Project admins add, change and remove a project's classes, and tenant admins a tenant's. Removing a class names the rules that want it.

![The Remove expert? dialog, over the Judge classes tab: "No new judgment can name it. Judgments that named it keep it, and still count with its weight in trends." and "2 rules want its judgment (Echo half, Support sample): change them first, or their items wait for a class nobody can give."](https://docs.kindgi.com/v0.1.10/_astro/quality-class-remove.G60SD57O_Z1SR43P.webp)

### Proposals

![The Proposals tab: "0 to review · 0 waiting for a reviewer · 1 being evaluated · 1 needs a look." Tuned agent's proposals: one drafted and not evaluated yet, one "Not better on the test set", and a withdrawn one under Closed.](https://docs.kindgi.com/v0.1.10/_astro/quality-proposals.CoqknEZJ_Z2gmOMC.webp)

The project's agents' [proposals](https://docs.kindgi.com/v0.1.10/guides/evals/improve-an-agent.md), grouped by what they wait for: **To review**, **Waiting for a reviewer**, **Being evaluated**, **Need a look** and **Closed**. Each says where it stands, such as "Not better on the test set", with the judged yes share it was evaluated on. Each links its page: "Each proposal's page holds its actions: evaluate, request, roll back, withdraw."