Skip to content
Kindgi

Correct a run's output

View as Markdown

This guide records people's yes or no on what a run returned, counts some people's corrections more than others, and queues the runs that need a correction.

A correction is a yes or no, with an optional reason, about one item of a finished run's output: the whole answer, or one element of a list it returned. People who know the work, your users or your experts, correct it. The corrections then tell you how an agent or flow version does: you build test cases from them, and try a new version on them. Corrections reference has every field, rule and console page.

Before you start

Section titled “Before you start”
  • A finished run to correct, and its id, on a runtime such as kindgi dev, with Kindgi 0.2's CLI. List runs finds one.
  • Write access to the run's project, when authorization is on.
  • For the app's half: Node 22.12 or later, Python 3.11 or later, or Java 17 or later, with Kindgi 0.2's client, and KINDGI_API_URL and KINDGI_API_TOKEN set.

Correct an item

Section titled “Correct an item”

A correction records whether one item of the run's output was right.

From the CLI, correct the run's answer:

Terminal window
pnpm exec kindgi judgments add --run=<run-id> --item=answer --pointer=/response/content --yes --reason="Echoes the message back"
  • --run=<run-id> is the run's id.
  • --item=answer is your id for the item: answer for the whole answer.
  • --pointer=/response/content is where the item is in the run's output, as a JSON Pointer.
  • --yes is the verdict, or --no.
  • --reason (optional) says why.

It prints the correction:

{
  "id": "512df8f6-6e27-4c4a-934e-1b345dfc510a",
  "projectId": "d4910d76-7355-4022-8f3c-514361cfa986",
  "runId": "bc6d1b53-3562-48c9-9f29-719779e3d110",
  "subject": { "kind": "agent", "id": "acme-ops.echo-agent", "version": "0.1.0" },
  "item": { "key": "answer", "pointer": "/response/content" },
  "verdict": "yes",
  "reason": "Echoes the message back",
  "assertedBy": { "kind": "user", "id": "26fc414a-b49f-4e1f-a022-162865770178" },
  "createdAt": "2026-10-06T04:08:27.667Z",
  …
}
Excerpt

From your app, when your users correct runs in your own UI, ask your coding agent: "I want to record my user's yes or no on a run's answer." For reference, the script looks like this, with the run's id as its argument:

judge.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev


const judgment = await kindgi.judgments.create({
  runId: process.argv[2],
  item: { key: 'answer', pointer: '/response/content' },
  verdict: 'yes',
  reason: 'Echoes the message back',
  participantId: 'customer-user-17', // your id for the user who judged, if you judge for your users
});
console.log(judgment.id, judgment.verdict, judgment.assertedBy);
  • participantId is your app's own opaque id for the user who made the correction. Who called comes from the token, as assertedBy.
  • Correcting the same item again, as the same caller and participant, replaces the earlier correction, and the earlier one stays in the history.

The run now has the correction. Only a finished run can be corrected, and the pointer must find something. Corrections has the rules and the refusals.

In the console, a finished run's page has Correct this run, or a Corrections tab once it has corrections. It lists the answer and each element of a list in the output, each with Yes and No and the counts so far. It lists each correction with who made it, its class and its reason.

Say what it should have done

Section titled “Say what it should have done”

A correction of an agent's run can also say what the run should have done (runtime 0.2.0 or later): the tool calls it should have made, with the arguments that matter, and the tools it shouldn't have called. It's about the whole run, not one item of its output. When you try a version on test cases built from the run, it checks what that version does against it.

Write what it should have done in a file, such as refund-fix.json:

{
  "calls": [
    { "tool": "acme-store.offer-store-credit", "args": { "orderId": "A-1042" } }
  ],
  "notCalled": ["acme-store.refund"]
}
  • calls are the calls it should have made. args holds only the top-level arguments that matter, each compared whole. Arguments left out aren't compared, and {} asks only that the tool was called.
  • notCalled are the tools it shouldn't have called at all.

Then record it with your mark:

Terminal window
pnpm exec kindgi judgments add --run=<run-id> --no --reason="Final sale: store credit, never a refund" --correction=@refund-fix.json
  • --correction is what it should have done, as JSON or @<file>.
  • --no marks the run's decisions wrong. --yes marks them right, and the calls you record then hold a later version to them.
  • It's about the whole run, so leave out --item and --pointer.

It prints the correction, with what it should have done as its correction:

CAPTURE: the printed judgment, with "item": { "key": "kindgi:correction", "pointer": "" } and its "correction"
  • One per run per person. Saving it again for the same run, as the same caller and participant, replaces your last one.
  • Only the agent's own tools. A tool the agent version that ran doesn't have is refused: The judged version has no tool "<tool>".

From your app, ask your coding agent: "I want to record what my expert says the run should have done." The clients' judgments.create takes it as correction, under the item { key: 'kindgi:correction', pointer: '' }. Its types are Correction and ExpectedCall, and in Python the field is not_called.

In the console, a completed agent run's Correct this run, or its Corrections tab once it has corrections, has What should it have done?:

  1. It lists the run's calls. Keep the ones that were right, and edit or untick the argument fields that matter.
  2. Flag the calls that shouldn't have happened, and add any the run missed with Add a call….
  3. Choose The decisions were wrong or The decisions were right, add Why (optional), and Save correction.

See the corrections

Section titled “See the corrections”

A run's corrections say how its output was corrected, and by whom:

Terminal window
pnpm exec kindgi judgments list --run=<run-id>      # newest first; replaced and removed ones are left out
pnpm exec kindgi judgments show <judgment-id>       # with copies of the run's input and output, and the item's value
pnpm exec kindgi judgments remove <judgment-id>
  • <run-id> is the run's id, and <judgment-id> a correction's id, from the list.
  • list also filters by agent and version, flow, verdict, judge class, participant and project.

list prints a page of corrections as JSON: data, hasMore, and nextCursor when there's another page. show prints one correction, and remove prints its id with "removed": true.

A correction keeps copies of the run's input and output, so it outlives the run's own retention. The clients have the same: judgments.list, judgments.get and judgments.unregister.

Weight corrections by judge class

Section titled “Weight corrections by judge class”

Without a class, every correction counts once. A judge class counts some people's corrections more, for the tenant, a project, or an agent in a project.

Add a class, then correct as it:

Terminal window
pnpm exec kindgi judge-classes add --name=expert --weight=3 --project=<project-id>
pnpm exec kindgi judgments add --run=<run-id> --item=answer --pointer=/response/content --yes --class=<judge-class-id>
  • --name names the class, and --weight=3 counts its corrections three times.
  • --project=<project-id> is the project it applies to. Build the test cases shows how to find a project's id.
  • --class=<judge-class-id> is the class's id, which add printed. In the clients, it's judgeClassId.

The correction now counts with the class's weight.

Restrict who may correct as a class

Section titled “Restrict who may correct as a class”

A restricted class says who may correct as it, so the people using your app can't move the figures that release a version.

Add the class with a restriction, before your reviewers correct runs:

Terminal window
pnpm exec kindgi judge-classes add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads"
  • --tenant makes it the tenant's class.
  • --min-reviewer-role=senior lets only a reviewer of that role or above correct as it: standard, senior or admin.
  • --description says who it is for.
  • --principal-kind=user|service and --principal-id=<id>, which repeat, restrict it to people or service tokens, or to named ones.

A caller must meet each restriction you set. One who doesn't is refused with 403, and nothing is recorded:

Error [auth]: You can't judge as "expert": it needs a senior reviewer or above; you're a standard reviewer.
Error [auth]: You can't judge as "tokens": only service tokens may assert it.

A correction made under a restriction carries "restricted": true. It's decided when the correction is made, so corrections recorded before the restriction don't count as restricted. Judge classes has the rest.

Queue the runs to correct

Section titled “Queue the runs to correct”

Judging rules pick the project's runs that need a person's correction, so your experts don't pick runs one at a time. As a run of the project ends, each rule checks it. A run that any rule matches joins the project's queue of answers waiting for a correction, To correct in the console, once.

To make a rule, ask your coding agent: "Queue a sample of my agent's live runs for people to correct." For reference, the code looks like this:

judging-rule.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient();
const project = await kindgi.projects.getDefault();


const rule = await kindgi.projects.judgingRules.create(project.id, {
  name: 'Live refund runs',
  when: { agentIds: ['acme.refunds'], versions: ['live'] },
  sample: 0.05, // 5% of them
  maxOpen: 20, // at most 20 waiting at once
});
console.log(rule.ruleId, rule.version);
  • when is which runs it matches: here, runs of acme.refunds's live version.
  • sample is the share of them to queue, 0.05 for 5%.
  • maxOpen stops the rule adding runs while that many of its runs wait.

It prints the rule's id and its version, 1. A rule as the runtime returns it, here one made with sample: 1 and maxOpen: 2:

{
  "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "version": 1,
  "name": "Live refund runs",
  "when": {
    "agentIds": [
      "acme.refunds"
    ],
    "versions": [
      "live"
    ]
  },
  "sample": 1,
  "maxOpen": 2,
  "enabled": true,
  "createdBy": "user:6161fd71-761c-4b92-be40-153923bad755",
  "createdAt": "2026-10-10T02:44:01.075Z"
}

From now on, the runs it matches join the queue as they end. It never runs a model or spends anything. Judging rules has every field, and how to try a rule before you make it.

Work through the queue

Section titled “Work through the queue”

The queue lists the runs waiting for a correction, oldest first. To list the ones you can close, ask your coding agent: "Show me the runs waiting for my correction." For reference, the code looks like this:

judging-queue.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient();
const project = await kindgi.projects.getDefault();


const waiting = await kindgi.projects.judgingQueue.list(project.id, { forMe: true });
console.log(waiting.total, waiting.data.length);

forMe: true keeps the items you can close: those that want a class you may correct as, or any correction. It prints how many such items wait in all, and how many are on this page. One item, as that rule queued it:

{
  "runId": "4d5fb3c0-6835-436d-ba29-fd182661d189",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "agentId": "acme.refunds",
  "agentVersion": "2.1.0",
  "flowId": "agent.turn",
  "runStatus": "completed",
  "completedAt": "2026-10-10T02:44:01.450Z",
  "rules": [
    {
      "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
      "version": 1
    }
  ],
  "wantedClassIds": [],
  "anyJudgment": true,
  "progress": {
    "total": 0,
    "byClass": []
  },
  "addedAt": "2026-10-10T02:44:01.447Z",
  "state": "open",
  "can": {
    "dismiss": true,
    "reopen": false
  }
}

Correct each item's run, as Correct an item shows. An item closes as judged once each rule that queued it has the correction it wants. judgingQueue.dismiss, with an optional reason, takes a run not worth correcting out of the queue. Answers waiting for a correction has every field and count.

Reading the rules and the queue takes read access to the project. Changing a rule, and dismissing or reopening an item, take write access, as correcting a run does.

Next

Section titled “Next”