# Improve an agent

> An Improve run tries the settings Improve may change on an agent version, or drafts the agent's instructions, on the test cases within a budget, and writes its best candidate as a suggested change a person reviews.

This guide has an Improve run search for better settings or better instructions for an agent version, within a budget, and write its best candidate as a suggested change a person reviews.

An **Improve run** does the search a [suggested change](https://docs.kindgi.com/v0.3.1/guides/evals/propose-a-change.md) starts from. It tries candidates on part of the test cases, within a budget, proves the best one on the rest, and writes it as a suggested change only when it's better. A person still decides: the suggested change waits for a reviewer.

## Before you start

An Improve run needs what a suggested change does, as [Suggest a change, and test it](https://docs.kindgi.com/v0.3.1/guides/evals/propose-a-change.md#before-you-start) lists:

- **an agent whose instructions or settings are data blocks;**
- **test cases built from corrections;**
- **a live version of the agent for the whole tenant.** Without one, the Improve run is refused with `proposal-needs-pin`;
- **a runtime: `kindgi dev`, or a deployed one,** with Kindgi 0.2's CLI. Under `kindgi dev`, making a suggested change live writes it into the pack's files, as [Under `kindgi dev`](https://docs.kindgi.com/v0.3.1/guides/evals/propose-a-change.md#under-kindgi-dev) shows;
- **`publish` on the agent,** to start one, and `write` on the project it works in when that isn't the agent version's.

## Mark what Improve may change

An Improve run changes only what you let it. On settings, it changes only the keys a settings block's schema marks with `"x-kindgi-tunable": true`: a number or an integer with a `minimum` below its `maximum`, or an `enum`. Any other mark is refused when the block is published.

Ask your coding agent: "Let Improve change my reply length and tone, but not the refund limit." For reference, it marks them in the settings block's schema like this:

```json
{
  "type": "object",
  "properties": {
    "maxSentences": { "type": "integer", "minimum": 1, "maximum": 6, "x-kindgi-tunable": true },
    "tone": { "enum": ["friendly", "formal"], "x-kindgi-tunable": true },
    "refundLimit": { "type": "number", "minimum": 0 }
  }
}
```

Here Improve may try other sentence limits and tones, and never touches the refund limit. Publish the schema with the block's next version, with `kindgi blocks publish … --schema=@reply-style.schema.json`, as [Publish the blocks](https://docs.kindgi.com/v0.3.1/guides/evals/change-a-prompt-or-setting.md#publish-the-blocks) shows. A version that pins no block with a marked key has nothing Improve may change, and the Improve run is refused, saying so: `validation-failed`, with `details.reason` `no-tunable-keys`.

### Let it try a cheaper model or another temperature

Improve changes the agent's model-settings block (the one it names as `modelSettings`) the same way: mark its keys. It takes `temperature` (0 to 2), `maxOutputTokens`, how much the model thinks, `thinking`, and the model the turn prefers, `model` (with `providerId` for its provider). List the models an Improve run may try as the `model` key's `enum`, and the thinking levels as `thinking`'s:

```json
{
  "type": "object",
  "properties": {
    "temperature": { "type": "number", "minimum": 0, "maximum": 1, "x-kindgi-tunable": true },
    "thinking": { "enum": ["lowest", "low", "medium"], "x-kindgi-tunable": true },
    "model": { "enum": ["claude-sonnet-5-5", "claude-haiku-4-5"], "x-kindgi-tunable": true }
  }
}
```

An Improve run tries each marked key within these bounds, kept inside what model settings allow. A level the model doesn't take is rounded, or not sent to a model that doesn't think, as [Temperature and thinking](https://docs.kindgi.com/v0.3.1/guides/models.md#temperature-and-thinking) describes. It compares what the replays cost at the models' registered prices, as a [budget](https://docs.kindgi.com/v0.3.1/guides/agents/budgets.md) counts them, so list models registered with their prices. List ones your registered providers serve, too: a preference never excludes, so a model none of them serves replays on the next one in line.

- **Name the model in one place.** When the block sets `model` or `providerId`, it's the turn's preference instead of the agent's `preferredModel` and `preferredProvider`. A version that names the model in both places is refused when it's published or deployed, and so is a version derived to pin such a block (a suggested change's candidate). When the agent sets `preferredModel` or `preferredProvider`, an Improve run doesn't try `model` or `providerId`, and says why.
- **The decisions must stay right.** A change to this block is kept only when no [correction](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md) the current version gets right goes wrong: the tool calls an expert said the agent should have made, graded on each replay. The Improve run checks one on the part it searches and again on the part that proves its best change, and the split puts cases that carry a correction in both parts only when there are two or more. So at least two of the test cases need a correction that names the tool calls; without them, nothing is kept, and the Improve run says why.
- **Better, or no worse and cheaper.** Such a change is kept when it's better, as any candidate is. Or when it's no worse than the current version (beyond the spread of the current version's repeated runs) and costs less, by more than the spread of what those runs cost. A temperature change is usually the first kind, a cheaper model the second.

To hold its best change to that, the Improve run also replays the current version on the cases it didn't learn from, twice for a spread. That counts in the Improve run's cost, like the best change's own run on those cases, but the budget doesn't stop it (see [the budget](https://docs.kindgi.com/v0.3.1/guides/evals/improve-an-agent.md#start-an-improve-run)). The suggested change's `evaluation` says which rule kept it (`"rule": "cheaper"`), what each side cost (`costUsd`), and how the corrections went (`decisions`):

```json excerpt
"evaluation": {
  "rule": "cheaper",
  "costUsd": { "current": 0.42, "candidate": 0.26 },
  "decisions": { "checked": 15, "held": 12, "broken": 0, "stillWrong": 3 },
  …
}
```

## Start an Improve run

An Improve run searches for a better version of the settings Improve may change:

```sh
pnpm exec kindgi proposals improve --agent=acme.rank.scorer --tenant --test-set=acme.rank-prompt-52080 --max-candidates=4 --wait
```

- **The agent and scope:** `--agent` and `--tenant`, or another scope flag.
- **The version** is the one serving the scope, unless `--from-version` names another.
- **The project:** an Improve run works in one project and learns from its runs. Its version and test cases must be ones [that project's runs can use](https://docs.kindgi.com/v0.3.1/guides/projects/organize-by-org-and-project.md#what-a-projects-runs-can-use), and its eval runs run there. With `--project`, it's that project. With `--tenant` or `--org`, it's `--pass-project=<project-id>`, else the agent version's. A project other than the version's needs `write` there. In the API and the clients, it's `projectId` (`project_id` in Python). The Improve run's project needs runtime 0.2.0 or later: an earlier one refuses `--pass-project`, and its Improve run works in the agent version's project.
- **The test cases:** `--test-set`, built from corrections.
- **The budget:** `--max-cost` in US dollars, 5 by default and at most 100, and `--max-candidates`, 30 by default and at most 200. An Improve run stops at whichever it reaches first. It checks the cost between runs on the test cases, so the one already running can take it a little over `--max-cost`. Its cost counts every run on the test cases it starts, the ones that prove its best candidate on the cases it didn't search too. Those run after the search, so the budget doesn't stop them, and they can take an Improve run past `--max-cost`.
- **What better means:** `--objective`, `weightedYesShare` by default or `weightedPrecisionAtK`, as in a [run on the test cases](https://docs.kindgi.com/v0.3.1/guides/evals/compare-an-agent-version.md#read-the-result).
- **How it decides:** it searches on 70% of the test cases and proves its best candidate on the other 30% (the cases it didn't learn from), a split fixed for the Improve run. On the 70%, a candidate has to beat the current version by more than the spread between the current version's own repeated runs. On the 30%, the best one has to beat the recorded outputs, and only then does it become a suggested change, which carries both figures: the recorded outputs' score and the candidate's. A change to the model-settings block is also held to the current version replayed on each part, and can instead be kept as no worse and cheaper: [Let it try a cheaper model or another temperature](https://docs.kindgi.com/v0.3.1/guides/evals/improve-an-agent.md#let-it-try-a-cheaper-model-or-another-temperature).

Before you start, you can ask what an Improve run would cost: `POST /v1/proposals/improve/estimate`, with the body an Improve run would start from, answers `estimatedCostUsd` — the replay count times the version's mean cost over its recent completed runs, capped at `--max-cost` — and starts nothing. Treat it as an approximation: a candidate can cost more or less per run than that mean, and an Improve run on the instructions also pays to draft them, so it can cost more than it estimates; what stops it is the budget above. An estimate equal to `--max-cost` means the Improve run is expected to spend its whole budget.

It answers at once with the Improve run. `--wait` waits for its outcome, at most 30 minutes, and prints it:

- **`proposed`**, with the suggested change it wrote, which a person decides: the Improve run asks for its review itself, in the name of whoever started it, so the suggested change waits in reviewers' approvals. An Improve run's suggested change always waits for a reviewer, because Improve, not a person, drafted it. Who may approve it depends on the scope's [release gate policy](https://docs.kindgi.com/v0.3.1/guides/evals/release-an-agent-version.md#gate-a-release):

  - **No `approvals`:** someone other than you, when there's another active reviewer who can see the approval. If there isn't, you can approve it yourself, as a registered reviewer.
  - **`approvals`:** someone other than you, with its role or a higher one. With `"separateApprover": false`, that holds only while there's such a reviewer; if there isn't, you can approve it yourself.

- **`nothing-found`**, with why. Nothing Improve tried beat the current version by more than the spread of the current version's repeated runs on the 70%. Or the best one didn't beat the recorded outputs on the 30%, and the outcome's run on those cases gives both figures there. Or a model-settings change wasn't [kept under its rule](https://docs.kindgi.com/v0.3.1/guides/evals/improve-an-agent.md#let-it-try-a-cheaper-model-or-another-temperature). Or the best one is a change an existing suggested change already holds. Or nothing could be measured: an Improve run scores only the items of an output that corrections cover (a ranking's entries, say). So a changed free-text answer is a new answer no one has corrected yet and has nothing to score, and then you [correct the new answers, then rescore](https://docs.kindgi.com/v0.3.1/guides/evals/propose-a-change.md#correct-the-new-answers-then-rescore). The reason also says if the Improve run ran out of budget before trying the rest.

- **`failed`**, with the reason.

An Improve run is `running`, then `completed`, `failed` or `cancelled`. Here one tried four settings. On the cases it learned from, setting `fit` to 0 scored best, but an earlier suggested change already held that change, so it wrote none. The cost is 0 on `dev-echo`, which costs nothing:

```json excerpt
{
  "id": "46209449-8d69-42aa-8cdd-78a514d9c848",
  …
  "budget": {
    "maxCostUsd": 5,
    "maxCandidates": 4
  },
  "requestedBy": "user:62e1ffe2-30f4-4dff-948e-ae774107a7db",
  "status": "completed",
  "candidatesEvaluated": 4,
  "costUsd": "0",
  "outcome": {
    "kind": "nothing-found",
    "reason": "The best change Improve tried is already suggested change 19b0f480-146a-4545-bd6b-37742b592dfe, which is evaluated."
  },
  "comparisons": [
    {
      "role": "reference",
      "part": "search",
      "evalRunId": "6c4d6ce7-6838-4eb1-b2ec-d7992ed86278",
      "score": 0.5714285714285714
    },
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.weights",
      "changed": {
        "fit": 0
      },
      "evalRunId": "7a0ba425-57dc-43aa-83ce-0e37b9d5b9f4",
      "score": 1
    },
    …
  ],
  …
}
```

`kindgi proposals passes get <pass-id>` reads the same Improve run back later.

### Whose corrections it learns from

By default, once your test cases' project, its tenant, or the agent you're improving there has a [restricted judge class](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md#restrict-who-may-correct-as-a-class), an Improve run counts only the corrections recorded under one (`classWeights: "restricted-only"`), so who may steer Improve is a choice you make. That's the test cases' project, which can differ from the one the agent is made in. Until then it counts every correction, each by its class's weight (`classWeights: "as-recorded"`), since counting only restricted classes would count nothing. A class restricted for a different agent, or in a different project, doesn't count. Set it with the `--class-weights` flag, or `classWeights` in the API or the clients, to choose either, whatever classes exist.

The suggested change an Improve run writes names the corrections on the cases it learned from, never on the rest: the ones its runs on the test cases counted and, for an Improve run on the instructions, the ones its drafter was shown, including the ones with calls. Its `evidence.judgmentIds` lists up to 100 of them: first the ones the drafter was shown, case by case, each case's corrections with calls right after its items marked wrong; then the ones behind items marked wrong, then the others. When there are more, `evidence.judgmentCount` gives how many. Its runs on the test cases also grade a correction's calls on each replay and show the result, and that result decides only whether a change to the model settings is kept ([the decisions must stay right](https://docs.kindgi.com/v0.3.1/guides/evals/improve-an-agent.md#let-it-try-a-cheaper-model-or-another-temperature)). An Improve run on test cases built before correction ids (`judgmentIds`) names none.

## Draft the agent's instructions instead

An Improve run can also draft the agent's instructions. The agent version must take its instructions from a [prompt block](https://docs.kindgi.com/v0.3.1/guides/evals/change-a-prompt-or-setting.md) it pins. Otherwise the Improve run is refused with `details.reason` `no-pinned-prompt-block`. Start an Improve run on the instructions with `--tier=prompt`:

```sh
pnpm exec kindgi proposals improve --agent=acme-ops.reply-drafter --tenant \
  --test-set=acme.reply-cases --tier=prompt \
  --provider=anthropic --model=claude-sonnet-5-5 --drafts=3
```

- `--tier=prompt` drafts the agent's instructions.
- `--provider` and `--model` are the provider and model of yours that draft them, given together.
- `--drafts` is how many drafts of the instructions it makes: 1 to 5, 3 by default.

In the API and the clients, they're `tiers: ["prompt"]`, `model` and `candidates`:

```sh
curl -X POST "$KINDGI_API_URL/v1/proposals/improve" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") -H 'content-type: application/json' \
  -d '{"agentId":"acme-ops.reply-drafter","scope":{"kind":"tenant"},"suiteId":"acme.reply-cases","tiers":["prompt"],"model":{"providerId":"anthropic","model":"claude-sonnet-5-5"},"candidates":3}'
```

- `$KINDGI_API_URL` and `$KINDGI_API_TOKEN` are your runtime's URL and API token.

It answers at once (`202`) with the Improve run, as the CLI does without `--wait`.

The drafter sees the cases marked wrong, their reasons, and what [the corrections on them](https://docs.kindgi.com/v0.3.1/guides/evals/fix-what-experts-corrected.md) say the agent should have done: the calls it should have made and the tools it shouldn't have called, as an example for that case, not a rule. It's shown only the corrections the Improve run [counts](https://docs.kindgi.com/v0.3.1/guides/evals/improve-an-agent.md#whose-corrections-it-learns-from). A draft it writes is tried only if it stays within what the agent already has:

- it parses as Liquid;
- it reads only the instructions' declared parameters, the variables the current instructions read, `today`, `now`, `agent`, `conversation`, and `settings["<id>"]` of a settings block the version pins;
- it names no tool, agent or block id the agent doesn't already use;
- it isn't empty or the current instructions, and it's at most twice as long as the current ones: at least 2,000 characters, never over 20,000;
- it quotes no test case: a run of 8 or more words in a row from what the drafter was shown is refused, and cut from the hypothesis it keeps.

A refused draft shows in the Improve run's `comparisons` with its issues, as `refused`, and each drafted one with its `hypothesis`. Here a local model drafted instructions reading a variable the agent doesn't have. The drafts are that model's words, and yours will differ:

```sh
pnpm exec kindgi proposals passes get <pass-id>
```

```json excerpt
{
  "id": "7d56260c-0b5a-43ef-9c56-164c59bd76f7",
  …
  "model": {
    "model": "qwen2.5:7b",
    "providerId": "ollama-qwen"
  },
  …
  "comparisons": [
    …
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.prompt",
      "changed": {
        "template": "Rank the candidates from most to least preferred. Consider the following weights for each pair: {{score}}. Report the ranking."
      },
      "hypothesis": "This template emphasizes ranking from most to least preferred and uses {{score}} to include specific weights, aiming to provide more context for accurate ranking.",
      "refused": [
        {
          "path": "/template",
          "message": "it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)"
        }
      ],
      "failed": "The drafted instructions were refused: it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)."
    }
  ],
  …
}
```

An Improve run's drafts and hypotheses live on in it, the suggested change and the published block version, which erasing a person's words doesn't reach. That's why a draft may not quote a test case, and why a person reviews it before it goes live.

## Read Improve runs back, or stop one

```sh
pnpm exec kindgi proposals passes list --table
pnpm exec kindgi proposals passes get <pass-id>
pnpm exec kindgi proposals passes cancel <pass-id>
```

- `list --table` prints each Improve run's agent version, scope, status, the candidates tried, the cost and the outcome.
- `get` adds its runs on the test cases, each an eval run you can open. `<pass-id>` is the Improve run's `id`.
- `cancel` stops a running Improve run: it ends `cancelled` and writes no suggested change. An Improve run that already ended answers `409 improvement-pass-finished`.

From your app: `proposals.improve` and `improvementPasses.list`, `get` and `cancel` in TypeScript; `proposals.improve` and `improvement_passes` in Python. An endpoint subscribed to `improvement-pass.finished` gets each Improve run when it ends ([Webhook events and deliveries](https://docs.kindgi.com/v0.3.1/guides/webhooks/webhook-events.md#improve-runs)).

## Run Improve on a schedule

A schedule can start an Improve run when enough new corrections have come in, counted the way the Improve run will count them, as [Improve on a schedule](https://docs.kindgi.com/v0.3.1/guides/runs/run-on-a-schedule.md#improve-on-a-schedule) shows.

## In the console

The same loop runs in the console. On an agent's **Suggested changes** page, **Improve** starts an Improve run, and the Improve run's page follows it. The suggested change it writes shows its verdict and the figures from its run on the test cases, under **Evidence** the corrections it learned from, each with its verdict and reason and a link to its run, and an **Ask to pin it** button to send it for a decision. **Schedule Improve**, on the same page, makes a schedule that starts Improve runs for the agent, as [Schedule Improve in the console](https://docs.kindgi.com/v0.3.1/guides/runs/run-on-a-schedule.md#schedule-improve-in-the-console) shows.

An agent's own page shows the loop for that agent after its **Recent runs**, each section with its newest five: **Corrected runs**, the runs people corrected, with **To correct** for its answers waiting in the [Quality section](https://docs.kindgi.com/v0.3.1/guides/evals/judging-reference.md#to-correct); **Test cases**, gathered from them, with **All test cases**; **Improve runs**; and **Suggested changes**, with **All suggested changes and Improve runs**, which opens the page above.

## Next

- [Suggest a change, and test it](https://docs.kindgi.com/v0.3.1/guides/evals/propose-a-change.md)
- [Correct a run's output](https://docs.kindgi.com/v0.3.1/guides/evals/judge-a-runs-output.md)
- [Run on a schedule](https://docs.kindgi.com/v0.3.1/guides/runs/run-on-a-schedule.md)