# kindgi judgments > Correct items of a run's output, marking them right or wrong (add / list / show / remove) ## `kindgi judgments` **Synopsis** ```sh signature kindgi judgments ``` Correct items of a run's output, marking them right or wrong (add / list / show / remove). **Commands** | Command | Description | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | [`kindgi judgments add`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments-add) | Correct one item of a finished run's output: mark the answer right (`--yes`) or wrong (`--no`), with an optional reason. | | [`kindgi judgments list`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments-list) | List corrections, newest first (replaced and removed ones are left out). | | [`kindgi judgments show`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments-show) | Show a correction with the stored copies of the run's input and output, and the corrected item's value. | | [`kindgi judgments remove`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments-remove) | Remove a correction. | **See also** - [Build test cases from corrections](https://docs.kindgi.com/v0.3.0/guides/evals/build-a-test-set.md) - [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md) - [Suggest a change, and test it](https://docs.kindgi.com/v0.3.0/guides/evals/propose-a-change.md) - [Fix what the experts corrected](https://docs.kindgi.com/v0.3.0/guides/evals/fix-what-experts-corrected.md) ### `kindgi judgments add` **Synopsis** ```sh signature kindgi judgments add --run= --item= [--pointer=] [--rank=] (--yes | --no) [--reason=] [--correction=] [--class=] [--participant=] ``` Correct one item of a finished run's output: mark the answer right (`--yes`) or wrong (`--no`), with an optional reason. Correcting the same item again as the same caller replaces the earlier correction. **Options** | Option | Default | Description | | ----------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--class ` | none (the correction is unclassified) | The judge class id to record the correction under. Without one, the correction is unclassified. | | `--correction ` | none (an ordinary correction, without what the agent should have done) | What the agent should have done, as JSON or `@`: `{ "calls": [{ "tool": "", "args": { … } }], "notCalled": [""] }`. `args` are the partial top-level fields to compare (those left out are not compared); `notCalled` lists tools that should not have been called at all. Graded on a replay. Recorded against the whole output, so a later correction of the same run by the same caller replaces it; `--item`/`--pointer`/`--rank` aren't needed. | | `--item ` | required | Your stable id for the corrected item, e.g. a matched case id. Required. | | `--no` | off (exactly one of `--yes` or `--no` is required) | Mark the item wrong (the verdict `no`). Give `--yes` or `--no`. | | `--participant ` | none | Your opaque id for the end user who corrected, when you correct on their behalf. | | `--pointer ` | none (item recorded without a pointer) | Where the item is in the run's output, as a JSON Pointer such as `/matches/2`. Its value is kept with the correction. | | `--rank ` | none | The item's position in a ranked list (0 = first). | | `--reason ` | none | Why, recorded with the correction. | | `--run ` | required | The run whose output is corrected. Required. | | `--yes` | off (exactly one of `--yes` or `--no` is required) | Mark the item right (the verdict `yes`). Give `--yes` or `--no`. | Every command also takes the [global flags](https://docs.kindgi.com/v0.3.0/reference/cli.md#global-flags). **Examples** From [Build test cases from corrections](https://docs.kindgi.com/v0.3.0/guides/evals/build-a-test-set.md): ```sh pnpm exec kindgi judgments add --run= --item=kb-101 --pointer=/output/articles/0 --rank=0 --yes --reason="Resets the password" pnpm exec kindgi judgments add --run= --item=kb-102 --pointer=/output/articles/1 --rank=1 --no --reason="Not about signing in" ``` **See also** - [Build test cases from corrections](https://docs.kindgi.com/v0.3.0/guides/evals/build-a-test-set.md) - [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md) - [Suggest a change, and test it](https://docs.kindgi.com/v0.3.0/guides/evals/propose-a-change.md) - [`kindgi judgments`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments) ### `kindgi judgments list` **Synopsis** ```sh signature kindgi judgments list [--run=] [--agent= [--agent-version=]] [--flow=] [--verdict=yes|no] [--class=] [--participant=] [--project=] [--limit=] [--cursor=] ``` List corrections, newest first (replaced and removed ones are left out). **Options** | Option | Default | Description | | ------------------------- | -------------------- | ---------------------------------------------------------------- | | `--agent ` | none | Only corrections of runs of this agent. | | `--agent-version ` | none | Only corrections of runs of this agent version. Needs `--agent`. | | `--class ` | none | Only corrections under this judge class id. | | `--cursor ` | none | Resume after this cursor, from the previous page's `nextCursor`. | | `--flow ` | none | Only corrections of runs of this flow. | | `--limit ` | `25` (at most `100`) | The most corrections to return (default 25, at most 100). | | `--participant ` | none | Only corrections made for this end user's opaque id. | | `--project ` | none | Only corrections in this project. | | `--run ` | none | Only corrections of this run. | | `--verdict ` | none | Only this verdict: `yes` or `no`. | Every command also takes the [global flags](https://docs.kindgi.com/v0.3.0/reference/cli.md#global-flags). **Examples** From [Fix what the experts corrected](https://docs.kindgi.com/v0.3.0/guides/evals/fix-what-experts-corrected.md): ```sh pnpm exec kindgi judgments list --agent=acme-store.refunds --agent-version=1.0.3 --verdict=no ``` From [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md): ```sh pnpm exec kindgi judgments list --run= # newest first; replaced and removed ones are left out ``` **See also** - [Fix what the experts corrected](https://docs.kindgi.com/v0.3.0/guides/evals/fix-what-experts-corrected.md) - [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md) - [`kindgi judgments`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments) ### `kindgi judgments show` **Synopsis** ```sh signature kindgi judgments show ``` Show a correction with the stored copies of the run's input and output, and the corrected item's value. **Options** Every command also takes the [global flags](https://docs.kindgi.com/v0.3.0/reference/cli.md#global-flags). **Examples** From [Fix what the experts corrected](https://docs.kindgi.com/v0.3.0/guides/evals/fix-what-experts-corrected.md): ```sh pnpm exec kindgi judgments show ``` From [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md): ```sh pnpm exec kindgi judgments show # with copies of the run's input and output, and the item's value ``` **See also** - [Fix what the experts corrected](https://docs.kindgi.com/v0.3.0/guides/evals/fix-what-experts-corrected.md) - [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md) - [`kindgi judgments`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments) ### `kindgi judgments remove` **Synopsis** ```sh signature kindgi judgments remove ``` Remove a correction. **Options** Every command also takes the [global flags](https://docs.kindgi.com/v0.3.0/reference/cli.md#global-flags). **Examples** From [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md): ```sh pnpm exec kindgi judgments remove ``` **See also** - [Correct a run's output](https://docs.kindgi.com/v0.3.0/guides/evals/judge-a-runs-output.md) - [`kindgi judgments`](https://docs.kindgi.com/v0.3.0/reference/cli/judgments.md#kindgi-judgments)