Improve an agent

This guide has an Improve run search for better settings or better instructions for an agent version, within a budget, and write its best candidate as a suggested change a person reviews.

An Improve run does the search a suggested change starts from. It tries candidates on part of the test cases, within a budget, proves the best one on the rest, and writes it as a suggested change only when it's better. A person still decides: the suggested change waits for a reviewer.

Before you start

An Improve run needs what a suggested change does, as Suggest a change, and test it lists:

• an agent whose instructions or settings are data blocks;
• test cases built from corrections;
• a live version of the agent for the whole tenant. Without one, the Improve run is refused with proposal-needs-pin;
• a runtime: kindgi dev, or a deployed one, with Kindgi 0.2's CLI. Under kindgi dev, making a suggested change live writes it into the pack's files, as Under kindgi dev shows;
• publish on the agent, to start one, and write on the project it works in when that isn't the agent version's.

Mark what Improve may change

An Improve run changes only what you let it. On settings, it changes only the keys a settings block's schema marks with "x-kindgi-tunable": true: a number or an integer with a minimum below its maximum, or an enum. Any other mark is refused when the block is published.

Ask your coding agent: "Let Improve change my reply length and tone, but not the refund limit." For reference, it marks them in the settings block's schema like this:

{
  "type": "object",
  "properties": {
    "maxSentences": { "type": "integer", "minimum": 1, "maximum": 6, "x-kindgi-tunable": true },
    "tone": { "enum": ["friendly", "formal"], "x-kindgi-tunable": true },
    "refundLimit": { "type": "number", "minimum": 0 }
  }
}

Here Improve may try other sentence limits and tones, and never touches the refund limit. Publish the schema with the block's next version, with kindgi blocks publish … --schema=@reply-style.schema.json, as Publish the blocks shows. A version that pins no block with a marked key has nothing Improve may change, and the Improve run is refused, saying so: validation-failed, with details.reason no-tunable-keys.

Let it try a cheaper model or another temperature

Improve changes the agent's model-settings block (the one it names as modelSettings) the same way: mark its keys. It takes temperature (0 to 2), maxOutputTokens, how much the model thinks, thinking, and the model the turn prefers, model (with providerId for its provider). List the models an Improve run may try as the model key's enum, and the thinking levels as thinking's:

{
  "type": "object",
  "properties": {
    "temperature": { "type": "number", "minimum": 0, "maximum": 1, "x-kindgi-tunable": true },
    "thinking": { "enum": ["lowest", "low", "medium"], "x-kindgi-tunable": true },
    "model": { "enum": ["claude-sonnet-5-5", "claude-haiku-4-5"], "x-kindgi-tunable": true }
  }
}

An Improve run tries each marked key within these bounds, kept inside what model settings allow. A level the model doesn't take is rounded, or not sent to a model that doesn't think, as Temperature and thinking describes. It compares what the replays cost at the models' registered prices, as a budget counts them, so list models registered with their prices. List ones your registered providers serve, too: a preference never excludes, so a model none of them serves replays on the next one in line.

• Name the model in one place. When the block sets model or providerId, it's the turn's preference instead of the agent's preferredModel and preferredProvider. A version that names the model in both places is refused when it's published or deployed, and so is a version derived to pin such a block (a suggested change's candidate). When the agent sets preferredModel or preferredProvider, an Improve run doesn't try model or providerId, and says why.
• The decisions must stay right. A change to this block is kept only when no correction the current version gets right goes wrong: the tool calls an expert said the agent should have made, graded on each replay. The Improve run checks one on the part it searches and again on the part that proves its best change, and the split puts cases that carry a correction in both parts only when there are two or more. So at least two of the test cases need a correction that names the tool calls; without them, nothing is kept, and the Improve run says why.
• Better, or no worse and cheaper. Such a change is kept when it's better, as any candidate is. Or when it's no worse than the current version (beyond the spread of the current version's repeated runs) and costs less, by more than the spread of what those runs cost. A temperature change is usually the first kind, a cheaper model the second.

To hold its best change to that, the Improve run also replays the current version on the cases it didn't learn from, twice for a spread. That counts in the Improve run's cost, like the best change's own run on those cases, but the budget doesn't stop it (see the budget). The suggested change's evaluation says which rule kept it ("rule": "cheaper"), what each side cost (costUsd), and how the corrections went (decisions):

"evaluation": {
  "rule": "cheaper",
  "costUsd": { "current": 0.42, "candidate": 0.26 },
  "decisions": { "checked": 15, "held": 12, "broken": 0, "stillWrong": 3 },
  …
}

Excerpt

Start an Improve run

An Improve run searches for a better version of the settings Improve may change:

pnpm exec kindgi proposals improve --agent=acme.rank.scorer --tenant --test-set=acme.rank-prompt-52080 --max-candidates=4 --wait

• The agent and scope: --agent and --tenant, or another scope flag.
• The version is the one serving the scope, unless --from-version names another.
• The project: an Improve run works in one project and learns from its runs. Its version and test cases must be ones that project's runs can use, and its eval runs run there. With --project, it's that project. With --tenant or --org, it's --pass-project=<project-id>, else the agent version's. A project other than the version's needs write there. In the API and the clients, it's projectId (project_id in Python). The Improve run's project needs runtime 0.2.0 or later: an earlier one refuses --pass-project, and its Improve run works in the agent version's project.
• The test cases: --test-set, built from corrections.
• The budget: --max-cost in US dollars, 5 by default and at most 100, and --max-candidates, 30 by default and at most 200. An Improve run stops at whichever it reaches first. It checks the cost between runs on the test cases, so the one already running can take it a little over --max-cost. Its cost counts every run on the test cases it starts, the ones that prove its best candidate on the cases it didn't search too. Those run after the search, so the budget doesn't stop them, and they can take an Improve run past --max-cost.
• What better means: --objective, weightedYesShare by default or weightedPrecisionAtK, as in a run on the test cases.
• How it decides: it searches on 70% of the test cases and proves its best candidate on the other 30% (the cases it didn't learn from), a split fixed for the Improve run. On the 70%, a candidate has to beat the current version by more than the spread between the current version's own repeated runs. On the 30%, the best one has to beat the recorded outputs, and only then does it become a suggested change, which carries both figures: the recorded outputs' score and the candidate's. A change to the model-settings block is also held to the current version replayed on each part, and can instead be kept as no worse and cheaper: Let it try a cheaper model or another temperature.

Before you start, you can ask what an Improve run would cost: POST /v1/proposals/improve/estimate, with the body an Improve run would start from, answers estimatedCostUsd — the replay count times the version's mean cost over its recent completed runs, capped at --max-cost — and starts nothing. Treat it as an approximation: a candidate can cost more or less per run than that mean, and an Improve run on the instructions also pays to draft them, so it can cost more than it estimates; what stops it is the budget above. An estimate equal to --max-cost means the Improve run is expected to spend its whole budget.

It answers at once with the Improve run. --wait waits for its outcome, at most 30 minutes, and prints it:

• proposed, with the suggested change it wrote, which a person decides: the Improve run asks for its review itself, in the name of whoever started it, so the suggested change waits in reviewers' approvals. An Improve run's suggested change always waits for a reviewer, because Improve, not a person, drafted it. Who may approve it depends on the scope's release gate policy:
  • No approvals: someone other than you, when there's another active reviewer who can see the approval. If there isn't, you can approve it yourself, as a registered reviewer.
  • approvals: someone other than you, with its role or a higher one. With "separateApprover": false, that holds only while there's such a reviewer; if there isn't, you can approve it yourself.
• nothing-found, with why. Nothing Improve tried beat the current version by more than the spread of the current version's repeated runs on the 70%. Or the best one didn't beat the recorded outputs on the 30%, and the outcome's run on those cases gives both figures there. Or a model-settings change wasn't kept under its rule. Or the best one is a change an existing suggested change already holds. Or nothing could be measured: an Improve run scores only the items of an output that corrections cover (a ranking's entries, say). So a changed free-text answer is a new answer no one has corrected yet and has nothing to score, and then you correct the new answers, then rescore. The reason also says if the Improve run ran out of budget before trying the rest.
• failed, with the reason.

An Improve run is running, then completed, failed or cancelled. Here one tried four settings. On the cases it learned from, setting fit to 0 scored best, but an earlier suggested change already held that change, so it wrote none. The cost is 0 on dev-echo, which costs nothing:

{
  "id": "46209449-8d69-42aa-8cdd-78a514d9c848",
  …
  "budget": {
    "maxCostUsd": 5,
    "maxCandidates": 4
  },
  "requestedBy": "user:62e1ffe2-30f4-4dff-948e-ae774107a7db",
  "status": "completed",
  "candidatesEvaluated": 4,
  "costUsd": "0",
  "outcome": {
    "kind": "nothing-found",
    "reason": "The best change Improve tried is already suggested change 19b0f480-146a-4545-bd6b-37742b592dfe, which is evaluated."
  },
  "comparisons": [
    {
      "role": "reference",
      "part": "search",
      "evalRunId": "6c4d6ce7-6838-4eb1-b2ec-d7992ed86278",
      "score": 0.5714285714285714
    },
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.weights",
      "changed": {
        "fit": 0
      },
      "evalRunId": "7a0ba425-57dc-43aa-83ce-0e37b9d5b9f4",
      "score": 1
    },
    …
  ],
  …
}

Excerpt

kindgi proposals passes get <pass-id> reads the same Improve run back later.

Whose corrections it learns from

By default, once your test cases' project, its tenant, or the agent you're improving there has a restricted judge class, an Improve run counts only the corrections recorded under one (classWeights: "restricted-only"), so who may steer Improve is a choice you make. That's the test cases' project, which can differ from the one the agent is made in. Until then it counts every correction, each by its class's weight (classWeights: "as-recorded"), since counting only restricted classes would count nothing. A class restricted for a different agent, or in a different project, doesn't count. Set it with the --class-weights flag, or classWeights in the API or the clients, to choose either, whatever classes exist.

The suggested change an Improve run writes names the corrections on the cases it learned from, never on the rest: the ones its runs on the test cases counted and, for an Improve run on the instructions, the ones its drafter was shown, including the ones with calls. Its evidence.judgmentIds lists up to 100 of them: first the ones the drafter was shown, case by case, each case's corrections with calls right after its items marked wrong; then the ones behind items marked wrong, then the others. When there are more, evidence.judgmentCount gives how many. Its runs on the test cases also grade a correction's calls on each replay and show the result, and that result decides only whether a change to the model settings is kept (the decisions must stay right). An Improve run on test cases built before correction ids (judgmentIds) names none.

Draft the agent's instructions instead

An Improve run can also draft the agent's instructions. The agent version must take its instructions from a prompt block it pins. Otherwise the Improve run is refused with details.reason no-pinned-prompt-block. Start an Improve run on the instructions with --tier=prompt:

pnpm exec kindgi proposals improve --agent=acme-ops.reply-drafter --tenant \
  --test-set=acme.reply-cases --tier=prompt \
  --provider=anthropic --model=claude-sonnet-5-5 --drafts=3

• --tier=prompt drafts the agent's instructions.
• --provider and --model are the provider and model of yours that draft them, given together.
• --drafts is how many drafts of the instructions it makes: 1 to 5, 3 by default.

In the API and the clients, they're tiers: ["prompt"], model and candidates:

curl -X POST "$KINDGI_API_URL/v1/proposals/improve" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") -H 'content-type: application/json' \
  -d '{"agentId":"acme-ops.reply-drafter","scope":{"kind":"tenant"},"suiteId":"acme.reply-cases","tiers":["prompt"],"model":{"providerId":"anthropic","model":"claude-sonnet-5-5"},"candidates":3}'

• $KINDGI_API_URL and $KINDGI_API_TOKEN are your runtime's URL and API token.

It answers at once (202) with the Improve run, as the CLI does without --wait.

The drafter sees the cases marked wrong, their reasons, and what the corrections on them say the agent should have done: the calls it should have made and the tools it shouldn't have called, as an example for that case, not a rule. It's shown only the corrections the Improve run counts. A draft it writes is tried only if it stays within what the agent already has:

• it parses as Liquid;
• it reads only the instructions' declared parameters, the variables the current instructions read, today, now, agent, conversation, and settings["<id>"] of a settings block the version pins;
• it names no tool, agent or block id the agent doesn't already use;
• it isn't empty or the current instructions, and it's at most twice as long as the current ones: at least 2,000 characters, never over 20,000;
• it quotes no test case: a run of 8 or more words in a row from what the drafter was shown is refused, and cut from the hypothesis it keeps.

A refused draft shows in the Improve run's comparisons with its issues, as refused, and each drafted one with its hypothesis. Here a local model drafted instructions reading a variable the agent doesn't have. The drafts are that model's words, and yours will differ:

pnpm exec kindgi proposals passes get <pass-id>

{
  "id": "7d56260c-0b5a-43ef-9c56-164c59bd76f7",
  …
  "model": {
    "model": "qwen2.5:7b",
    "providerId": "ollama-qwen"
  },
  …
  "comparisons": [
    …
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.prompt",
      "changed": {
        "template": "Rank the candidates from most to least preferred. Consider the following weights for each pair: {{score}}. Report the ranking."
      },
      "hypothesis": "This template emphasizes ranking from most to least preferred and uses {{score}} to include specific weights, aiming to provide more context for accurate ranking.",
      "refused": [
        {
          "path": "/template",
          "message": "it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)"
        }
      ],
      "failed": "The drafted instructions were refused: it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)."
    }
  ],
  …
}

Excerpt

An Improve run's drafts and hypotheses live on in it, the suggested change and the published block version, which erasing a person's words doesn't reach. That's why a draft may not quote a test case, and why a person reviews it before it goes live.

Read Improve runs back, or stop one

pnpm exec kindgi proposals passes list --table
pnpm exec kindgi proposals passes get <pass-id>
pnpm exec kindgi proposals passes cancel <pass-id>

• list --table prints each Improve run's agent version, scope, status, the candidates tried, the cost and the outcome.
• get adds its runs on the test cases, each an eval run you can open. <pass-id> is the Improve run's id.
• cancel stops a running Improve run: it ends cancelled and writes no suggested change. An Improve run that already ended answers 409 improvement-pass-finished.

From your app: proposals.improve and improvementPasses.list, get and cancel in TypeScript; proposals.improve and improvement_passes in Python. An endpoint subscribed to improvement-pass.finished gets each Improve run when it ends (Webhook events and deliveries).

Run Improve on a schedule

A schedule can start an Improve run when enough new corrections have come in, counted the way the Improve run will count them, as Improve on a schedule shows.

In the console

The same loop runs in the console. On an agent's Suggested changes page, Improve starts an Improve run, and the Improve run's page follows it. The suggested change it writes shows its verdict and the figures from its run on the test cases, under Evidence the corrections it learned from, each with its verdict and reason and a link to its run, and an Ask to pin it button to send it for a decision. Schedule Improve, on the same page, makes a schedule that starts Improve runs for the agent, as Schedule Improve in the console shows.

An agent's own page shows the loop for that agent after its Recent runs, each section with its newest five: Corrected runs, the runs people corrected, with To correct for its answers waiting in the Quality section; Test cases, gathered from them, with All test cases; Improve runs; and Suggested changes, with All suggested changes and Improve runs, which opens the page above.

Next

• Suggest a change, and test it
• Correct a run's output
• Run on a schedule