Fix what the experts corrected

View as Markdown

An expert — a support lead, a claims reviewer, an editor — marks a run's answer wrong and says why. This guide turns those corrections into changes to the agent, each at the layer where it belongs, so the agent stops making the mistake and you can see the mistake is gone.

A correction is not always a prompt change. A store's "never refund shipping" belongs in the refund tool's code, where it's checked every time, not in instructions a model may or may not follow. The work is to sort each correction to the right layer and change that layer. Every correction is a case in the agent's test set, so each fix is checked against it, and so is every version after it. A coding agent does most of the work, led by its kindgi-fix-corrections skill, which walks it through these same steps.

Before you start

an agent whose settings and instructions are blocks, so a setting or the wording changes without a code deploy;
corrections to work from: runs the experts marked wrong, with a reason;
a coding agent, or someone who runs the CLI, to make the changes and add the tests;
permission to change things: publish on the agent to suggest a change or mark a correction you take, and pin to pin a version.

The loop

Pull the open corrections, with what was wrong and the run, and mark the ones you take being-fixed.
Sort each to a rule in code, a setting, an approval rule, or the wording.
Fix it there. A setting or the wording becomes a suggested change, checked on the agent's test set. A rule in code goes through the repo, with a pack test per correction.
A person pins it. The release gate decides, and its approvals means a second person checks it. Each correction the pinned version passes is Fixed in that version.

Most corrections make the agent stricter or change how it talks, and those a coding agent can make. Some ask for a policy looser than the one in force — with a 14-day return window, "should have refunded — 20 days is fine for us" wants the window widened. A correction that would loosen anything (raise a limit, in code or in a setting; widen a window; drop a check; stop a tool asking) is not made here: it goes to the owner, with the correction quoted.

A refusal the current rule already allows is different: the agent applied the rule wrong — "should have refunded, it was inside the 14 days" — and nothing about the policy changes. Fix it like any other correction, in code with a test or in the wording.

Pull the corrections

Start from the version the project runs now, so you don't re-fix something an earlier version already fixed, then list the corrections waiting for a developer:

pnpm exec kindgi agents pinned acme-store.refunds --project=<project-id>
pnpm exec kindgi corrections list --agent=acme-store.refunds --status=open --sorted-to=developer

Each correction has a status: open, being-fixed, with-owner, fixed or wont-fix. --status=not-fixed lists the first three. Each also says who fixes it, in sortedTo. A wrong answer is sorted to improve. Wrong decisions wait with the agent's owner, as with-owner, since a change to what the agent does may loosen a limit; the owner can re-sort one to developer for a change in code. Where a correction goes has the rules.

Read each one for what was wrong (why, shouldSay) and the run it's on:

pnpm exec kindgi corrections show <correction-id>
pnpm exec kindgi runs get <run-id>
pnpm exec kindgi runs journal <run-id>

runs get is the run's snapshot; runs journal is its durable record, step by step, where the agent's tool calls and their arguments are. Between the correction and the run you have what the agent did and what it should have done.

An expert can also have said exactly what the run should have done: the correction then says decisions: "wrong", with shouldCall and mustNotCall, the calls it should have made and the tools it shouldn't have called. Take those as what a test checks.

Mark the ones you take

Kindgi doesn't hand out corrections: two people's coding agents can pull the same ones. When you start on one, set it being-fixed with where the fix is, so nobody else takes it:

pnpm exec kindgi corrections set <correction-id> --status=being-fixed \
  --link=https://github.com/acme/store-agents/pull/219

--link takes a pull request's URL, a suggested change as sc- and its id (--link=sc-<suggested-change-id>), or an Improve run as ir- and its id.
It needs publish on the agent.
If you stop without a fix, set it back with --status=open.
Nobody sets fixed by hand: a pin does.

The runs hold real customer conversations. Keep them out of your repository, and write every test with invented data — made-up order ids, amounts and names — never a customer's words or details.

Sort each correction

The expert wrote	Where it belongs	Who changes it
"Never refund shipping", "final sale: no refund", "over 50 on a first order, ask a person"	a rule in code — the refund tool checks it every time; a cap sends the case to a tool that always asks	the coding agent, reviewed as a code change
"Our return window is 14 days, not 30"	a setting	a suggested change to the settings block, or the owner in the console
"Always have a person approve a refund"	an approval rule — a policy that makes a tool always ask; it keys on the tool, not its arguments	the owner or the approver
"Store credit before cash", "too stiff", "say when the money arrives"	the agent's wording	a suggested change to the prompt block when the expert said what to say; otherwise Improve

A correction that must hold every time is a code rule; one that's a matter of judgment or wording belongs in a block. A correction the owner would rather own (a setting) or the approver owns (an approval rule) is theirs to make: give them the command or the console page, and don't make the change for them.

A setting or the wording: suggest a change

A setting's value, or the wording when the expert said what it should say, is fixed with a suggested change: a new version of one block, made for one project on the version it runs, and checked before anyone pins it. No code changes.

pnpm exec kindgi suggested-changes create --agent=acme-store.refunds --project=<project-id> \
  --block=acme.store-policy --values=@store-policy.json \
  --why="Our return window is 14 days" --fixes=<correction-id>

--block names a block the project's version already pins. A new key, or a block the agent doesn't use yet, is a code change.
--values is a settings block's new values, checked against its schema; --template is a prompt block's new text instead.
Each --fixes names a correction it fixes. Mark each one being-fixed with --link=sc-<suggested-change-id>.
It makes the block's new version and the agent version that pins it, and starts its check: read the new version from the output.

The check runs the new version and the one it's built on, on the project's part of the agent's test set. Read it before asking for a pin: the corrections it fixes should pass on the new version and fail on the old one, and nothing should newly fail. Then ask to pin it with kindgi suggested-changes pin <id> --reason=…, or, for a project that runs its Latest, add it as Latest with kindgi suggested-changes add-as-latest. Suggest a change, and test it has the check's fields and both paths.

When the expert said what was wrong with the wording but not what to say, leave it to Improve. It tries rewrites on the corrections sorted to it, and makes a suggested change only when one does better on the test set. A person decides whether to take it, and nothing is applied on its own.

A rule in code: fix it through the repo

Change the tool, with one pack test per correction: the check, and a test that the corrected case is now handled right — an order delivered 20 days ago is refused, a final-sale item isn't refunded. Take what the test asserts from the correction (shouldCall, mustNotCall, shouldSay). A Node pack runs its tests with kindgi test (a wrapper over vitest); a Python, Java or Scala pack runs its own test runner.
Open a pull request, and mark each correction it fixes being-fixed with the pull request's URL. A person reviews and merges it.
The deploy makes the new version. A project that runs its Latest runs it at once. A project that runs a pin gets it as a suggested change, made by the deploy and checked like any other: ask to pin it.

When the fix needs a setting that doesn't exist yet, both parts are in the pack. The settings block's file in blocks/ gains the key in its schema and values, with a new version, and the code reads it from the agent's settings. kindgi deploy publishes the pack's blocks before the agents that use them. Mark a key Improve may change with "x-kindgi-tunable": true, and leave it off a key that guards a rule: a limit, a window, a cap.

The pin, and Fixed

Ask what would happen first, then ask for the pin:

pnpm exec kindgi agents pin acme-store.refunds 1.0.4 --project=<project-id> --check
pnpm exec kindgi agents pin acme-store.refunds 1.0.4 --project=<project-id> \
  --reason="Fixes 1 correction: our return window is 14 days"

Where it is pinned: --tenant (every project), --org=<org-id>, or --project=<project-id> (one project).
--check changes nothing and answers whether it would be pinned, needs an approval, or fails a release gate.
--reason is kept on the record: write what the corrections were, so the approver reads the change in plain words.
The outcome is pinned, or pending-approval when the release gate there has approvals and a second person must approve. The release gate's approvals is how you get that second person.
--check and the request exit 3 when the answer is no or the request waits for an approval. Read the printed outcome: exit 3 isn't an error.

When a project starts running another version, Kindgi checks that version on the project's part of the test set. Each correction the version passes becomes fixed, with the version it was fixed in; one that was fixed and now fails is reopened. The pin's answer lists both, under fixed and reopened.

There's no rollback: going back is a pin of the earlier version, through the same release gate, and the corrections that version fails reopen.

A worked example: the return window

An expert corrects a refund: "Shouldn't have refunded — delivered 20 days ago, our window is 14." The correction says the decisions were wrong, so it went to the agent's owner, who passed it to a developer. You pull it, and see the window is a store policy, not a one-off.

If the agent already reads a returnWindowDays setting, suggest a change to the block with returnWindowDays: 14, naming the correction in --fixes. Its check should show the correction passing on the new version and failing on the old one, with nothing newly failing. Or tell the owner to change it in the console.
If it doesn't read one yet, it's a code change. Add returnWindowDays to the settings block's file in the pack, with this schema and the value 14, and a new version; make the refund tool read it and refuse an order delivered more than that many days ago (offering a person); and add a test that an order delivered 20 days ago is refused. Leave x-kindgi-tunable off: the window is a rule.

{
  "type": "object",
  "properties": {
    "returnWindowDays": { "type": "integer", "minimum": 0, "maximum": 365 }
  },
  "required": ["returnWindowDays"]
}

Either way, ask to pin it with --reason="Fixes 1 correction: our return window is 14 days", and the pin marks the correction Fixed. The refund limit — the most a refund can be without a person — is never raised here: the cap stays in the tool's code, and the large-refund tool still always asks.

To try the loop, kindgi init <pack-name> --template=returns-helper scaffolds a returns helper with its instructions, its return window and its model settings as blocks. Its README walks through it on kindgi dev: correcting a run, then fixing it in the pack yourself, or running Improve and pinning its suggested change, which writes it into the pack's files and marks the correction Fixed (Under kindgi dev). On a deployed runtime, pinning it leaves the pack as it is: bring the pack up to it through a pull request.