Skip to content
Kindgi

Correct a run's output

View as Markdown

This guide records whether what a run did and returned was right, counts some people's corrections more than others, and queues the runs that need a correction.

A correction says whether a finished run's decisions, the tools it called, were right, whether its answer was right, and what was right instead. For an answer that is a ranked list, it says which items were right. People who know the work, your users or your experts, correct it. Each correction goes to whoever fixes it, and moves on until it's fixed. The corrections also tell you how an agent or flow version does: they're the cases of the agent's test set, and you try a new version on them. Corrections reference has every field, rule and console page.

Before you start

Section titled “Before you start”
  • A finished run to correct, and its id, on a runtime such as kindgi dev, with Kindgi 0.2's CLI. List runs finds one.
  • Write access to the run's project, when authorization is on.
  • For the app's half: Node 22.12 or later, Python 3.11 or later, or Java 17 or later, with Kindgi 0.2's client, and KINDGI_API_URL and KINDGI_API_TOKEN set.

Correct a run's answer

Section titled “Correct a run's answer”

A correction records whether the run's answer was right.

From the CLI, correct the run's answer:

Terminal window
pnpm exec kindgi corrections add --run=<run-id> --answer=right --why="Echoes the message back"
  • --run=<run-id> is the run's id.
  • --answer=right says the run's answer was right, or --answer=wrong that it was wrong. For a flow run, the answer is its whole output.
  • --why says why. It's optional, but a wrong answer needs it, or --should-say with what the answer should have said.

It prints the correction:

{
  "id": "512df8f6-6e27-4c4a-934e-1b345dfc510a",
  "projectId": "d4910d76-7355-4022-8f3c-514361cfa986",
  "runId": "bc6d1b53-3562-48c9-9f29-719779e3d110",
  "subject": { "kind": "agent", "id": "acme-ops.echo-agent", "version": "0.1.0" },
  "answer": "right",
  "why": "Echoes the message back",
  "madeBy": { "kind": "user", "id": "26fc414a-b49f-4e1f-a022-162865770178" },
  "status": "fixed",
  "fixedIn": {
    "how": "recorded-right",
    "by": { "kind": "user", "id": "26fc414a-b49f-4e1f-a022-162865770178" },
    "at": "2026-10-06T04:08:27.667Z"
  },
  "createdAt": "2026-10-06T04:08:27.667Z",
  …
}
Excerpt

Nothing in it is wrong, so it's fixed as soon as it's saved, as recorded-right. Where a correction goes has the other cases.

From your app, when your users correct runs in your own UI, ask your coding agent: "I want to record whether my user says a run's answer was right." For reference, the script looks like this, with the run's id as its argument:

correct.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev


const correction = await kindgi.corrections.create({
  runId: process.argv[2],
  answer: 'right',
  why: 'Echoes the message back',
  participantId: 'customer-user-17', // your id for the user who corrected it, if you correct for your users
});
console.log(correction.id, correction.answer, correction.madeBy);
  • answer is right or wrong. A wrong answer needs what it should have said, shouldSay, or why it was wrong, why.
  • participantId is your app's own opaque id for the user who made the correction. Who called comes from the token, as madeBy. Your users' corrections stay open and unsorted until a person sorts them, unless the project sorts them. Record your users' thumbs puts a thumbs up and down under your app's answers.
  • For an answer that is a ranked list, items says which items were right, as An agent's test set shows. Each item's key is an id, such as the matched case's id, never the item's text.
  • One correction per run and person. Correcting the run again, as the same caller and participant, replaces your earlier correction, and the earlier one stays in the history. From a client, send the whole correction each time. The CLI does that for you: each kindgi corrections add replaces the parts it gives and keeps the rest. Saying a part was right drops what should have happened there, and an empty --why= clears the why. A correction that was being fixed or with the owner starts again, sorted afresh, and the CLI says which it replaced.

The run now has the correction. Only a finished run can be corrected, and each item's pointer must find something. Corrections has the rules and the refusals.

In the console, a finished run you may correct has a Corrections page, one of the run's pages in the ▾ at the end of the breadcrumbs, and Correct this run opens it. It lists the answer and each element of a list in the output, each with Yes and No and the counts so far. It lists each correction with who made it, its group and its reason.

Once yours is saved, it shows under the form: "Saved as" its id, its status, who fixes it, with Hand it to someone else…, and links to what comes next: In Corrections › To fix while it isn't fixed, Run Improve on the agent when Improve fixes it, A case in its test set when it's one, and Next run to review, the next run waiting in To review. The run's header has Next run to review too.

Say what it should have done

Section titled “Say what it should have done”

A correction of an agent's run can also say what the run should have done (runtime 0.2.0 or later): the tool calls it should have made, with the arguments that matter, and the tools it shouldn't have called. It's about the run's decisions, not its answer. When you try a version on test cases built from the run, it checks what that version does against it.

Record it with your mark:

Terminal window
pnpm exec kindgi corrections add --run=<run-id> --decisions=wrong --should-call='acme-store.offer-store-credit:{"orderId":"A-1042"}' --must-not-call=acme-store.refund --why="Final sale: store credit, never a refund"
  • --decisions=wrong marks the run's decisions wrong.
  • --should-call is a call it should have made: the tool, then, after a :, the top-level arguments that matter, as JSON. Each is compared whole. Arguments left out aren't compared, and the tool alone asks only that it was called. Repeat it for each call.
  • --must-not-call is a tool it shouldn't have called at all. Repeat it for each tool.

It prints the correction, with what it should have done as its shouldCall and mustNotCall:

CAPTURE: the printed correction, with "decisions": "wrong", its "shouldCall" and "mustNotCall", "why", "status": "with-owner" and "sortedTo": "owner"
  • Part of your one correction of the run. Saving it again replaces what you said about the decisions, and keeps what you said about the answer.
  • It goes to the agent's owner. Wrong decisions are with-owner as soon as they're saved: a change to what the agent does may loosen a limit, and only the owner decides that.
  • Only the agent's own tools. A tool the agent version that ran doesn't have is refused: The corrected version has no tool "<tool>".

From your app, ask your coding agent: "I want to record what my expert says the run should have done." The clients' corrections.create takes it as decisions: 'wrong', with shouldCall, the calls it should have made ([{ tool, args? }], with args only when the inputs matter), and mustNotCall, the tool ids it shouldn't have called. In Python, they're should_call and must_not_call. To say its decisions were right, send decisions: 'right' alone.

In the console, a completed agent run's Corrections page, which Correct this run opens, has What should it have done?:

  1. It lists the run's calls. Keep the ones that were right, and edit or untick the argument fields that matter.
  2. Flag the calls that shouldn't have happened, and add any the run missed with Add a call….
  3. Choose The decisions were wrong or The decisions were right, add Why (optional), and Save correction.

See the corrections

Section titled “See the corrections”

A run's corrections say how its output was corrected, and by whom:

Terminal window
pnpm exec kindgi corrections list --run=<run-id>      # newest first; replaced and removed ones are left out
pnpm exec kindgi corrections show <correction-id>       # with copies of the run's input and output
pnpm exec kindgi corrections remove <correction-id>
  • <run-id> is the run's id, and <correction-id> a correction's id, from the list.
  • list also filters by agent and version, flow, corrector group, participant and project.
  • remove is for the correction's author, or an admin of its project. The correction becomes removed, and it can't come back.

list prints a page of corrections as JSON: data, hasMore, and nextCursor when there's another page. show prints one correction, and remove prints its id with "removed": true.

A correction keeps copies of the run's input and output, so it outlives the run's own retention. The clients have the same: corrections.list, corrections.get and corrections.remove. Their corrections.list also filters by status, sorting, who made it and when, as Where a correction goes shows.

Export the corrections

Section titled “Export the corrections”

An agent's corrections export as a CSV file, for a spreadsheet, or as JSON lines, for a script:

Terminal window
pnpm exec kindgi corrections export --agent=<agent-id> --since=2026-10-01 --until=2026-11-01 --out=october.csv
pnpm exec kindgi corrections export --agent=<agent-id> --format=jsonl --status=not-fixed --out=not-fixed.jsonl
  • <agent-id> is the agent whose corrections you export. It filters as the clients' corrections.list does: --project, --status and --sorted-to (each repeatable), --from (experts, end-users or all), --since and --until. A date, such as 2026-10-01, is midnight UTC.
  • Like list, it leaves out replaced corrections, and removed ones unless --status=removed asks for them.
  • --out names the file. On macOS and Linux, only you can read it. It takes that name only once the whole export has arrived, and an existing file is kept unless you add --force. Then it prints the agent, the format, the file and its size in bytes, as JSON. Without --out, the export is printed.
  • Each correction comes only when you may read its project.

The CSV has a header row, then one correction per line: id, projectId, runId, agentVersion, madeBy (user:<id> or service:<id>), participantId, expert, decisions, shouldCall and mustNotCall (these two as JSON), answer, shouldSay, why, sortedTo, status, fixedIn (the version that fixed it), reopenedIn (the version whose test run reopened it) and createdAt. A field that starts with =, +, -, @, a tab or a carriage return gets a leading ', so a spreadsheet shows it rather than running it as a formula. Where a spreadsheet's list separator is ;, import the file with a comma as the separator rather than opening it directly. JSON lines have one correction per line, as each one in list's data. Neither has the run's own input or output, only the run's id.

Every export is in the audit log, with who exported, the filters and how many rows went out. In the TypeScript and Python clients it's corrections.export: TypeScript's returns the body as a stream, and Python's yields it in chunks. The Java client has no method for it: use the CLI, or call GET /v1/corrections/export, as Export shows.

Where a correction goes

Section titled “Where a correction goes”

Each correction has a status, and sortedTo says who fixes it: improve, a developer, or the agent's owner. It's sorted when it's saved:

  • Your users' corrections, made with a participantId, stay open and unsorted until a person sorts them, and never start Improve, unless the project sorts them, as below. One under a restricted corrector group is an expert's, and is sorted like any other.
  • Wrong decisions go to the agent's owner: with-owner, sorted to owner. This is so even when the answer was wrong too.
  • A wrong answer, or a wrong item of it, goes to Improve: open, sorted to improve.
  • Nothing wrong is fixed, as recorded-right.

To sort your users' corrections like the others, turn on the project's sortEndUserCorrections, with admin on the project:

Terminal window
curl -X PATCH "$KINDGI_API_URL/v1/projects/<project-id>" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") -H 'content-type: application/json' \
  -d '{"sortEndUserCorrections":true}'

In TypeScript, that's kindgi.projects.update(projectId, { sortEndUserCorrections: true }), and in Python, kindgi.projects.update(project_id, sort_end_user_corrections=True). A project reads it back as sortEndUserCorrections, false until it's on.

Sorting applies to the corrections saved after you turn it on: one saved before stays unsorted until a person sorts it. A schedule that counts everyone's corrections (from: everyone) counts your users' corrections sorted to improve only while it's on. Once it's off, the schedule leaves them out, even ones sorted while it was on or by a person, though an Improve run that starts still takes them in.

A person then moves it on. With publish on the agent, corrections.change re-sorts it, changes its status, or both. Here a developer says they're fixing it, with a link to the work:

Terminal window
curl -X PATCH "$KINDGI_API_URL/v1/corrections/<correction-id>" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") -H 'content-type: application/json' \
  -d '{"status":"being-fixed","link":"https://code.acme.example/store/pull/42"}'

In TypeScript, that's kindgi.corrections.change(correctionId, { status: 'being-fixed', link }). It answers the changed correction. A change its status doesn't allow is refused with 409 correction-status-change-not-allowed. Status and sorting has every status, and Change a correction the changes each allows.

To see how many of a project's corrections are in each status, or one agent's, count them:

Terminal window
curl -s -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") \
  "$KINDGI_API_URL/v1/corrections/counts?scopeKind=project&scopeId=<project-id>&agentId=acme-store.refunds"
{ "open": 3, "beingFixed": 1, "withOwner": 1, "fixed": 12, "wontFix": 0, "removed": 0 }

In TypeScript, that's kindgi.corrections.counts({ projectId, agentId }). To list the ones behind a count, corrections.list takes status, such as not-fixed, short for open, being-fixed and with-owner, and sortedTo.

Weight corrections by corrector group

Section titled “Weight corrections by corrector group”

Without a group, every correction counts once. A corrector group counts some people's corrections more, for the workspace, a project, or an agent in a project.

Add a group, then correct as it:

Terminal window
pnpm exec kindgi corrector-groups add --name=expert --weight=3 --project=<project-id>
pnpm exec kindgi corrections add --run=<run-id> --answer=right --group=<corrector-group-id>
  • --name names the group, and --weight=3 counts its corrections three times.
  • --project=<project-id> is the project it applies to. Read the test set shows how to find a project's id.
  • --group=<corrector-group-id> is the group's id, which add printed. In the clients, it's correctorGroupId.

The correction now counts with the group's weight.

Restrict who may correct as a group

Section titled “Restrict who may correct as a group”

An expert group says who may correct as it, so the people using your app can't move the figures that release a version.

Add the group with a restriction, before your reviewers correct runs:

Terminal window
pnpm exec kindgi corrector-groups add --name=expert --weight=3 --tenant --min-reviewer-role=senior --description="Order desk leads"
  • --tenant makes it the workspace's group.
  • --min-reviewer-role=senior lets only a reviewer of that role or above correct as it: standard, senior or admin.
  • --description says who it is for.
  • --principal-kind=user|service and --principal-id=<id>, which repeat, restrict it to people or service tokens, or to named ones.

A caller must meet each restriction you set. One who doesn't is refused with 403, and nothing is recorded:

Error [auth]: You can't correct as "expert": it needs a senior reviewer or above; you're a standard reviewer.
Error [auth]: You can't correct as "tokens": only service tokens may assert it.

A correction made under a restriction carries "expert": true. It's decided when the correction is made, so corrections recorded before the restriction don't count as expert. Corrector groups has the rest.

Queue the runs to correct

Section titled “Queue the runs to correct”

Queue rules pick the project's runs that need a person's correction, so your experts don't pick runs one at a time. As a run of the project ends, each rule checks it. A run that any rule matches joins the project's queue of answers waiting for a correction, To review in the console's Corrections, once.

To make a rule, ask your coding agent: "Queue a sample of my agent's pinned-version runs for people to correct." For reference, the code looks like this:

queue-rule.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient();
const project = await kindgi.projects.getDefault();


const rule = await kindgi.projects.queueRules.create(project.id, {
  name: 'Pinned refund runs',
  when: { agentIds: ['acme.refunds'], versions: ['pinned'] },
  sample: 0.05, // 5% of them
  maxOpen: 20, // at most 20 waiting at once
});
console.log(rule.ruleId, rule.version);
  • when is which runs it matches: here, runs of acme.refunds's pinned version.
  • sample is the share of them to queue, 0.05 for 5%.
  • maxOpen stops the rule adding runs while that many of its runs wait.

It prints the rule's id and its version, 1. A rule as the runtime returns it, here one made with sample: 1 and maxOpen: 2:

{
  "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "version": 1,
  "name": "Pinned refund runs",
  "when": {
    "agentIds": [
      "acme.refunds"
    ],
    "versions": [
      "pinned"
    ]
  },
  "sample": 1,
  "maxOpen": 2,
  "enabled": true,
  "createdBy": "user:6161fd71-761c-4b92-be40-153923bad755",
  "createdAt": "2026-10-10T02:44:01.075Z"
}

From now on, the runs it matches join the queue as they end. It never runs a model or spends anything. Queue rules has every field, and how to try a rule before you make it.

Work through the queue

Section titled “Work through the queue”

The queue lists the runs waiting for a correction, oldest first. To list the ones you can close, ask your coding agent: "Show me the runs waiting for my correction." For reference, the code looks like this:

expert-queue.ts
import { createClient } from '@kindgi/sdk/client';


const kindgi = createClient();
const project = await kindgi.projects.getDefault();


const waiting = await kindgi.projects.expertQueue.list(project.id, { forMe: true });
console.log(waiting.total, waiting.data.length);

forMe: true keeps the items you can close: those that want a group you may correct as, or any correction. It prints how many such items wait in all, and how many are on this page. One item, as that rule queued it:

{
  "runId": "4d5fb3c0-6835-436d-ba29-fd182661d189",
  "projectId": "153a0091-613e-4c34-9535-da8194f9068e",
  "agentId": "acme.refunds",
  "agentVersion": "2.1.0",
  "flowId": "agent.turn",
  "runStatus": "completed",
  "completedAt": "2026-10-10T02:44:01.450Z",
  "rules": [
    {
      "ruleId": "19191620-85c3-4e2a-bd87-a0270d15ab53",
      "version": 1
    }
  ],
  "wantedGroupIds": [],
  "anyCorrection": true,
  "progress": {
    "total": 0,
    "byGroup": []
  },
  "addedAt": "2026-10-10T02:44:01.447Z",
  "state": "open",
  "can": {
    "dismiss": true,
    "reopen": false
  }
}

Correct each item's run, as Correct a run's answer shows. An item closes as corrected once each rule that queued it has the correction it wants. expertQueue.dismiss, with an optional reason, takes a run not worth correcting out of the queue. Answers waiting for a correction has every field and count.

Reading the rules and the queue takes read access to the project. Changing a rule, and dismissing or reopening an item, take write access, as correcting a run does.

Next

Section titled “Next”