kindgi judgments
kindgi judgments
Section titled “kindgi judgments”Synopsis
kindgi judgments <command>Signature
Correct items of a run's output, marking them right or wrong (add / list / show / remove).
Commands
| Command | Description |
|---|---|
kindgi judgments add |
Correct one item of a finished run's output: mark the answer right (--yes) or wrong (--no), with an optional reason. |
kindgi judgments list |
List corrections, newest first (replaced and removed ones are left out). |
kindgi judgments show |
Show a correction with the stored copies of the run's input and output, and the corrected item's value. |
kindgi judgments remove |
Remove a correction. |
See also
- Build test cases from corrections
- Correct a run's output
- Suggest a change, and test it
- Fix what the experts corrected
kindgi judgments add
Section titled “kindgi judgments add”Synopsis
kindgi judgments add --run=<run-id> --item=<key> [--pointer=<json-pointer>] [--rank=<n>] (--yes | --no) [--reason=<text>] [--correction=<json-or-@file>] [--class=<judge-class-id>] [--participant=<id>]Signature
Correct one item of a finished run's output: mark the answer right (--yes) or wrong (--no), with an optional reason. Correcting the same item again as the same caller replaces the earlier correction.
Options
| Option | Default | Description |
|---|---|---|
--class <value> |
none (the correction is unclassified) | The judge class id to record the correction under. Without one, the correction is unclassified. |
--correction <value> |
none (an ordinary correction, without what the agent should have done) | What the agent should have done, as JSON or @<file>: { "calls": [{ "tool": "<tool-id>", "args": { … } }], "notCalled": ["<tool-id>"] }. args are the partial top-level fields to compare (those left out are not compared); notCalled lists tools that should not have been called at all. Graded on a replay. Recorded against the whole output, so a later correction of the same run by the same caller replaces it; --item/--pointer/--rank aren't needed. |
--item <value> |
required | Your stable id for the corrected item, e.g. a matched case id. Required. |
--no |
off (exactly one of --yes or --no is required) |
Mark the item wrong (the verdict no). Give --yes or --no. |
--participant <value> |
none | Your opaque id for the end user who corrected, when you correct on their behalf. |
--pointer <value> |
none (item recorded without a pointer) | Where the item is in the run's output, as a JSON Pointer such as /matches/2. Its value is kept with the correction. |
--rank <value> |
none | The item's position in a ranked list (0 = first). |
--reason <value> |
none | Why, recorded with the correction. |
--run <value> |
required | The run whose output is corrected. Required. |
--yes |
off (exactly one of --yes or --no is required) |
Mark the item right (the verdict yes). Give --yes or --no. |
Every command also takes the global flags.
Examples
From Build test cases from corrections:
pnpm exec kindgi judgments add --run=<run-id> --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password"pnpm exec kindgi judgments add --run=<run-id> --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in"See also
- Build test cases from corrections
- Correct a run's output
- Suggest a change, and test it
kindgi judgments
kindgi judgments list
Section titled “kindgi judgments list”Synopsis
kindgi judgments list [--run=<run-id>] [--agent=<id> [--agent-version=<v>]] [--flow=<id>] [--verdict=yes|no] [--class=<judge-class-id>] [--participant=<id>] [--project=<id>] [--limit=<n>] [--cursor=<c>]Signature
List corrections, newest first (replaced and removed ones are left out).
Options
| Option | Default | Description |
|---|---|---|
--agent <value> |
none | Only corrections of runs of this agent. |
--agent-version <value> |
none | Only corrections of runs of this agent version. Needs --agent. |
--class <value> |
none | Only corrections under this judge class id. |
--cursor <value> |
none | Resume after this cursor, from the previous page's nextCursor. |
--flow <value> |
none | Only corrections of runs of this flow. |
--limit <value> |
25 (at most 100) |
The most corrections to return (default 25, at most 100). |
--participant <value> |
none | Only corrections made for this end user's opaque id. |
--project <value> |
none | Only corrections in this project. |
--run <value> |
none | Only corrections of this run. |
--verdict <value> |
none | Only this verdict: yes or no. |
Every command also takes the global flags.
Examples
From Fix what the experts corrected:
pnpm exec kindgi judgments list --agent=acme-store.refunds --agent-version=1.0.3 --verdict=noFrom Correct a run's output:
pnpm exec kindgi judgments list --run=<run-id> # newest first; replaced and removed ones are left outSee also
kindgi judgments show
Section titled “kindgi judgments show”Synopsis
kindgi judgments show <judgment-id>Signature
Show a correction with the stored copies of the run's input and output, and the corrected item's value.
Options
Every command also takes the global flags.
Examples
From Fix what the experts corrected:
pnpm exec kindgi judgments show <judgment-id>From Correct a run's output:
pnpm exec kindgi judgments show <judgment-id> # with copies of the run's input and output, and the item's valueSee also
kindgi judgments remove
Section titled “kindgi judgments remove”Synopsis
kindgi judgments remove <judgment-id>Signature
Remove a correction.
Options
Every command also takes the global flags.
Examples
From Correct a run's output:
pnpm exec kindgi judgments remove <judgment-id>See also