Improve an agent

View as Markdown

This guide has an Improve run look for better wording or better settings for an agent in a project, within a budget, and write a change that does better as a suggested change a person reviews.

An Improve run does the search a suggested change starts from. It works in one project: it starts from the version the project runs, learns from the project's corrections, and tries candidates on the project's test cases. It keeps a change only when the change passes more of the cases and fails none that passed, and writes it as a suggested change. A person still decides: the suggested change waits for a reviewer.

Before you start

an agent whose wording or settings are data blocks: its instructions in a prompt block, or a settings block with keys Improve may change;
test cases in the project, built from corrections;
a version of the agent running in the project, pinned there or its Latest;
Kindgi 0.4's runtime and CLI, deployed or kindgi dev. Under kindgi dev, the run's suggested change is for the whole workspace, and pinning it writes it into the pack's files (Under kindgi dev). On a deployed runtime, pinning it leaves the pack as it is: bring the pack up to it through a pull request;
publish on the agent. For an agent more than one project serves, also write where its versions are made.

Check it's ready says what's missing, if anything, and how to fix it.

Mark what Improve may change

An Improve run changes only what you let it. On settings, it changes only the keys a settings block's schema marks with "x-kindgi-tunable": true: a number or an integer with a minimum below its maximum, or an enum. Any other mark is refused when the block is published.

Ask your coding agent: "Let Improve change my reply length and tone, but not the refund limit." For reference, it marks them in the settings block's schema like this:

{
  "type": "object",
  "properties": {
    "maxSentences": { "type": "integer", "minimum": 1, "maximum": 6, "x-kindgi-tunable": true },
    "tone": { "enum": ["friendly", "formal"], "x-kindgi-tunable": true },
    "refundLimit": { "type": "number", "minimum": 0 }
  }
}

Here Improve may try other sentence limits and tones, and never touches the refund limit. Publish the schema with the block's next version, with kindgi blocks publish … --schema=@reply-style.schema.json, as Publish the blocks shows. A version with no marked key has no settings for Improve to try: the readiness check lists settings as missing, and says how to fix it.

Let it try another model or temperature

Improve changes the agent's model-settings block (the one it names as modelSettings) the same way: mark its keys. It takes temperature (0 to 2), maxOutputTokens, how much the model thinks, thinking, and the model the turn prefers, model (with providerId for its provider). List the models an Improve run may try as the model key's enum, and the thinking levels as thinking's:

{
  "type": "object",
  "properties": {
    "temperature": { "type": "number", "minimum": 0, "maximum": 1, "x-kindgi-tunable": true },
    "thinking": { "enum": ["lowest", "low", "medium"], "x-kindgi-tunable": true },
    "model": { "enum": ["claude-sonnet-5-5", "claude-haiku-4-5"], "x-kindgi-tunable": true }
  }
}

An Improve run tries each marked key within these bounds, kept inside what model settings allow. A level the model doesn't take is rounded, or not sent to a model that doesn't think, as Temperature and thinking describes. Its replays count toward its budget at the models' registered prices, as a budget counts them, so list models registered with their prices. List ones your registered providers serve, too: a preference never excludes, so a model none of them serves replays on the next one in line.

Name the model in one place. When the block sets model or providerId, it's the turn's preference instead of the agent's preferredModel and preferredProvider. A version that names the model in both places is refused when it's published or deployed, and so is a version derived to pin such a block (a suggested change's candidate). When the agent sets preferredModel or preferredProvider, an Improve run doesn't try model or providerId.
Kept by the same rule. A new model or temperature is kept only when it does better, as any change is (How it decides). There's no cheaper path: a model that does as well for less isn't kept. To move to one, change the block yourself and test it.

Check it's ready

Before you start, --check asks whether an Improve run is ready, and what it's estimated to cost. It starts nothing:

pnpm exec kindgi improve runs start --agent=acme.rank.scorer --project=<project-id> --check --table

ITEM             STATUS   FIX
───────────────  ───────  ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
wording          missing  Improve rewrites a prompt block's wording with a model. If the instructions are written in code, not in a prompt block, ask your coding agent to move them into one, then deploy. If no model is named to write it, give the run a provider and model, or name them in the version's model-settings block. (in the agent's code, then a deploy)
settings         ok
test-cases (25)  ok
permission       ok
one-at-a-time    ok
project          ok

Ready: yes
Starts from: 1.0.0 (pinned in the project)
Estimate: $0.84, its search stops at $5.00 (200 replays, 60 of them the proof's)
This month: $3.10 of the Improve schedule's $20.00

The items, in this order: wording (a prompt block to reword, and a model to draft it), settings (settings marked Improve may tune), test-cases (the project's active cases for the agent, and how many), permission (publish on the agent), one-at-a-time (no Improve run of the agent already running in the project) and project (a version of the agent runs in the project, and the project isn't suspended). A missing one says where it's fixed, in the agent's code (then a deploy) or in Kindgi, and how.
Ready when wording or settings is there, and every other item is. With --tier, each tier named needs its own: --tier=prompt needs wording.
Starts from: the version the project runs, pinned there or the agent's latest.
The estimate: the replays the Improve run would make, times the version's mean cost over its recent completed runs. The search's share is at most --max-cost, and the proof's comes on top, as the budget says. None when the version has no cost history. Here, of 25 cases it sets 5 aside and searches on 20. The settings marked in the example above, at maxSentences 3 and tone friendly, give 5 candidates to try first (maxSentences at 1, 2, 4 and 6, and tone at formal). The search replays the 20 cases twice for the version it starts from and once for each candidate: 20 × (2 + 5) = 140. The proof replays every case twice for the best change, and the 5 set aside twice for the version it starts from: (25 + 5) × 2 = 60, the answer's basis.proofReplays. That's 200 replays; at a mean of $0.0042 a run, $0.84. When it tries new wording too, each draft is a candidate.
Treat it as an approximation: a candidate can cost more or less per run than that mean, drafting new wording costs more, and a second round of settings or a retried draft adds replays the estimate can't know of, so an Improve run can cost more than its estimate. The budget stops its search; the proof runs after it and isn't capped by it.
This month: with an Improve schedule on the agent and project, what their Improve runs spent this month (UTC), by hand or by the schedule, against the schedule's month cap. Only the schedule stops at the cap.

--check exits 3 when it isn't ready. Without --table, it prints the answer as JSON: ready, startsFrom, items, estimate and month. In the API, it's POST /v1/improve-runs/check, with the body an Improve run would start from.

Start an Improve run

pnpm exec kindgi improve runs start --agent=acme.rank.scorer --project=<project-id> --max-candidates=4 --wait

The agent and the project: --agent, and --project, the project it works in. In the API and the clients, they're agentId and projectId (project_id in Python).
The version is the one the project runs: pinned there, else the agent's latest. The Improve run's startsFrom says which.
The corrections: when it starts, it takes the project's open corrections sorted to Improve, and takes lists their ids. Whose corrections it learns from says whose count.
The test cases: the project's test cases for the agent, at the version it freezes when it starts (testSetVersion).
What it tries: --tier=prompt, new wording for the prompt block, or --tier=settings, new values for the settings it may tune. Repeat --tier for both. Left out, it tries every tier that's ready.
The budget: --max-cost in US dollars, 5 by default and at most 100, and --max-candidates, 30 by default and at most 200. They cap the search, which stops at whichever it reaches first. It checks its cost before each run on the test cases, so the one already running can take it a little over --max-cost. Its cost counts every run on the test cases it starts, the proof of its best change too. The proof runs after the search, so the budget doesn't stop it, and it can take an Improve run past --max-cost.
How candidates that pass equally are ranked: --objective, passRate by default or weightedPrecisionAtK, as in a run on the test cases. The candidate that passes the most cases is the best; between ones that pass as many, the better score wins, then the one that changes fewer settings.

A start that isn't ready is refused with 409 improve-not-ready, listing every missing item with its fix, as --check does. While the agent has an Improve run going in the project, another is refused with 409 improve-run-running, naming it: one runs at a time.

How it decides

It sets about a fifth of the test cases aside (setAside says how many, of how many), and searches on the rest: each candidate runs on those cases, and the best one is the change it may suggest. Then it proves that change on every case of the test set, the ones set aside too, as many times each as the version it started from ran. With fewer than five cases, it sets none aside, and searches and proves on all of them. The change is kept only when its proof passes more cases than the version it started from, and fails none that passed. That's the only rule: a change that does as well for less, or does better on some cases and worse on others, isn't kept.

What it found

It answers at once with the Improve run. --wait waits for its outcome, at most 30 minutes, and prints it. An Improve run is running, then completed, failed or cancelled, and its outcome is one of:

suggested, with the suggested change it wrote (suggestedChangeId), which a person decides: the Improve run asks for its review itself, in the name of whoever started it, so the suggested change waits in reviewers' approvals. An Improve run's suggested change always waits for a reviewer, because Improve, not a person, drafted it. Who may approve it depends on the project's release gate:
- No approvals: someone other than you, when there's another active reviewer who can see the approval. If there isn't, you can approve it yourself, as a registered reviewer.
- approvals: someone other than you, with its role or a higher one. With "separateApprover": false, that holds only while there's such a reviewer; if there isn't, you can approve it yourself.
nothing-better, with why (reason). No change it tried was kept. When one got as far as its proof, holdOut has the proof's numbers: the version it started from (baseline), the change (candidate), and the difference (delta).
cost-cap, with what it spent (spentUsd). It reached --max-cost before it kept a change.
failed, with why (message).

Here one tried four settings. Setting fit to 0 scored best on the cases it searched, and its proof passed more cases than version 1.0.0 and failed none that passed, so it wrote a suggested change. The cost is 0 on dev-echo, which costs nothing:

{
  "id": "46209449-8d69-42aa-8cdd-78a514d9c848",
  "agentId": "acme.rank.scorer",
  "projectId": "8f1e8a52-3c5d-4a5e-9b8e-0d6c2a7f4e11",
  "startsFrom": { "version": "1.0.0", "follows": "pin" },
  "testSetVersion": "1.0.0",
  "setAside": { "cases": 5, "of": 25 },
  "tiers": ["settings"],
  …
  "status": "completed",
  "tried": 4,
  "costUsd": "0",
  "outcome": {
    "kind": "suggested",
    "suggestedChangeId": "19b0f480-146a-4545-bd6b-37742b592dfe"
  },
  "comparisons": [
    {
      "role": "reference",
      "part": "search",
      "testRunId": "6c4d6ce7-6838-4eb1-b2ec-d7992ed86278",
      "score": 0.55
    },
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.weights",
      "changed": {
        "fit": 0
      },
      "testRunId": "7a0ba425-57dc-43aa-83ce-0e37b9d5b9f4",
      "score": 0.9
    },
    …
  ],
  …
}

Excerpt

comparisons are its runs on the test cases, each a test run you can open: the version it started from on each part (reference), each candidate on the cases it searched (candidate), and the proof on all of them (proof, part all). kindgi improve runs get <improve-run-id> reads the same Improve run back later.

Whose corrections it learns from

By default, once the project, its tenant, or the agent in the project has an expert group, an Improve run counts only the corrections recorded under one (groupWeights: "experts-only"), so who may steer Improve is a choice you make. Until then it counts every correction, each by its group's weight (groupWeights: "as-recorded"), since counting only expert groups would count nothing. A group restricted for a different agent, or in a different project, doesn't count. Set it with the --group-weights flag, or groupWeights in the API or the clients, to choose either, whatever groups exist.

The suggested change an Improve run writes names the corrections on the cases it learned from, never on the rest: the ones its runs on the test cases counted and, for an Improve run on the instructions, the ones its drafter was shown, including the ones with calls. Its fixes lists up to 100 of them: first the ones the drafter was shown, case by case, each case's corrections with calls right after its items marked wrong; then the ones behind items marked wrong, then the others. Its correctionCount says how many in all. An older runtime leaves it out, and there the length of fixes is the count. Its runs on the test cases also grade a correction's calls on each replay and show the result. An Improve run on test cases built before correction ids (correctionIds) names none.

Draft the agent's instructions instead

An Improve run can also draft the agent's instructions. The agent version must take its instructions from a prompt block it pins, and a model must draft them: the run's --provider and --model, or the provider and model the version's model-settings block names (providerId and model). If either is missing, the readiness check lists wording as missing. Start an Improve run on the instructions with --tier=prompt:

pnpm exec kindgi improve runs start --agent=acme-ops.reply-drafter --project=<project-id> \
  --tier=prompt --provider=anthropic --model=claude-sonnet-5-5 --drafts=3

--tier=prompt drafts the agent's instructions.
--provider and --model are the provider and model of yours that draft them, given together. Left out, the provider and model the version's model-settings block names draft them; with neither, the prompt tier isn't ready.
--drafts is how many drafts of the instructions it makes: 1 to 5, 3 by default.

In the API and the clients, they're tiers: ["prompt"], model and candidates:

curl -X POST "$KINDGI_API_URL/v1/improve-runs" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") -H 'content-type: application/json' \
  -d '{"agentId":"acme-ops.reply-drafter","projectId":"<project-id>","tiers":["prompt"],"model":{"providerId":"anthropic","model":"claude-sonnet-5-5"},"candidates":3}'

$KINDGI_API_URL and $KINDGI_API_TOKEN are your runtime's URL and API token.
<project-id> is the project it works in.

It answers at once (202) with the Improve run, as the CLI does without --wait.

The drafter sees the cases marked wrong, their reasons, what an answer marked wrong should have said or why it was wrong (shouldSay, why), and what the corrections on them say the agent should have done: the calls it should have made and the tools it shouldn't have called, as an example for that case, not a rule. It's shown only the corrections the Improve run counts. A draft it writes is tried only if it stays within what the agent already has:

it parses as Liquid;
it reads only the instructions' declared parameters, the variables the current instructions read, today, now, agent, conversation, and settings["<id>"] of a settings block the version pins;
it names no tool, agent or block id the agent doesn't already use;
it isn't empty or the current instructions, and it's at most twice as long as the current ones: at least 2,000 characters, never over 20,000;
it quotes no test case: a run of 8 or more words in a row from what the drafter was shown is refused, and cut from the why it keeps.

A refused draft shows in the Improve run's comparisons with its issues, as refused, and each drafted one with its why. Here a local model drafted instructions reading a variable the agent doesn't have. The drafts are that model's words, and yours will differ:

pnpm exec kindgi improve runs get <improve-run-id>

{
  "id": "7d56260c-0b5a-43ef-9c56-164c59bd76f7",
  …
  "model": {
    "model": "qwen2.5:7b",
    "providerId": "ollama-qwen"
  },
  …
  "comparisons": [
    …
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.prompt",
      "changed": {
        "template": "Rank the candidates from most to least preferred. Consider the following weights for each pair: {{score}}. Report the ranking."
      },
      "why": "This template emphasizes ranking from most to least preferred and uses {{score}} to include specific weights, aiming to provide more context for accurate ranking.",
      "refused": [
        {
          "path": "/template",
          "message": "it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)"
        }
      ],
      "failed": "The drafted instructions were refused: it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)."
    }
  ],
  …
}

Excerpt

An Improve run's drafts and reasons live on in it, the suggested change and the published block version, which erasing a person's words doesn't reach. That's why a draft may not quote a test case, and why a person reviews it before it is pinned.

Read Improve runs back, or stop one

pnpm exec kindgi improve runs list --project=<project-id> --table
pnpm exec kindgi improve runs get <improve-run-id>
pnpm exec kindgi improve runs cancel <improve-run-id>

list --table prints each Improve run's agent and the version it started from, its project, status, the candidates it tried, its cost, what it found and when it started. --agent, --project and --status (running, completed, failed or cancelled) narrow it.
get adds its runs on the test cases, each a test run you can open. <improve-run-id> is the Improve run's id.
cancel stops a running Improve run: it ends cancelled and writes no suggested change. An Improve run that already ended answers 409 improve-run-finished.

From your app: improveRuns.start, check, list, get and cancel in TypeScript; improve_runs in Python. An endpoint subscribed to improve-run.finished gets each Improve run when it ends (Webhook events and deliveries).

Run Improve on a schedule

A schedule can start an Improve run on an agent in a project when there are enough new corrections since the schedule's last Improve run, and stop for the month at its cap, as Improve on a schedule shows. Under kindgi dev, a schedule's fire starts no Improve run: kindgi dev runs while you work, not around the clock, so a schedule is for a deployed runtime. Start Improve by hand there.

In the console

The same loop runs in the console, on the agent's pages: Overview, Runs, Corrections, Test set, Suggested changes and Pinned versions. The breadcrumbs at the top end on the page you're on, such as Agents › Returns helper › Overview ▾, and its ▾ lists the others:

Overview has the card How it gets better: "People correct its runs. Each correction becomes a case in its test set. Improve tries changes on the test set and suggests the best, and a pin puts one in use." Each of its four lines opens its page, such as "Corrections: 1 open" or "Test set: 1 case".
Runs has the agent's runs in the project, as a project's Runs page lists them, with the same filters. Each run opens under the agent.
Corrections has what people said about the agent's runs, and who fixes each one, the ones not fixed yet to start with. The runs waiting for a person to review them are in the menu's Corrections, under To review.
Test set has its cases, with Add a case, Retire and Run Improve, which opens Suggested changes at its Improve runs, and its latest test runs, each opening its page. It says what's in it: "What a new version is tested on: this agent's corrections with something to check, and the cases added by hand. Left out: Won't fix and Removed corrections, corrections made on a test run's replays, and retired cases."
Suggested changes has its suggested changes, and its Improve runs, each with how it ended, such as Found nothing better, and its reason. Run Improve starts an Improve run, and the Improve run's page follows it. The suggested change it writes shows its verdict and the figures from its run on the test cases, under Evidence the corrections it learned from, each with its verdict and reason and a link to its run, and an Ask to pin it button to send it for a decision. Schedule… makes a schedule that starts Improve runs for the agent in a project, as Schedule Improve in the console shows.
Pinned versions has where each version is pinned, as In the console shows.

A flow's pages are Overview, Runs and Corrections, as Write a flow shows.