# Improve an agent > An Improve run tries new wording for an agent's prompt block, or new values for the settings Improve may tune, on a project's test cases within a budget, and writes a change that does better as a suggested change a person reviews. This guide has an Improve run look for better wording or better settings for an agent in a project, within a budget, and write a change that does better as a suggested change a person reviews. An **Improve run** does the search a [suggested change](https://docs.kindgi.com/v0.5.0/guides/improve/suggest-a-change.md) starts from. It works in one project: it starts from the version the project runs, learns from the project's corrections, and tries candidates on the project's test cases. It keeps a change only when the change passes more of the cases and fails none that passed, and writes it as a suggested change. A person still decides: the suggested change waits for a reviewer. ## Before you start - **an agent whose wording or settings are data blocks:** its instructions in a [prompt block](https://docs.kindgi.com/v0.5.0/guides/improve/change-a-prompt-or-setting.md), or a settings block with keys [Improve may change](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#mark-what-improve-may-change); - **test cases in the project,** [built from corrections](https://docs.kindgi.com/v0.5.0/guides/improve/build-a-test-set.md); - **a version of the agent running in the project,** pinned there or its Latest; - **Kindgi 0.4's runtime and CLI,** deployed or `kindgi dev`. Under `kindgi dev`, the run's suggested change is for the whole workspace, and pinning it writes it into the pack's files ([Under `kindgi dev`](https://docs.kindgi.com/v0.5.0/guides/improve/suggest-a-change.md#under-kindgi-dev)). On a deployed runtime, pinning it leaves the pack as it is: [bring the pack up to it](https://docs.kindgi.com/v0.5.0/guides/improve/suggest-a-change.md#bring-the-pack-up-to-it) through a pull request; - **`publish` on the agent.** For an agent more than one project serves, also `write` where its versions are made. [Check it's ready](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#check-its-ready) says what's missing, if anything, and how to fix it. ## Mark what Improve may change An Improve run changes only what you let it. On settings, it changes only the keys a settings block's schema marks with `"x-kindgi-tunable": true`: a number or an integer with a `minimum` below its `maximum`, or an `enum`. Any other mark is refused when the block is published. Ask your coding agent: "Let Improve change my reply length and tone, but not the refund limit." For reference, it marks them in the settings block's schema like this: ```json { "type": "object", "properties": { "maxSentences": { "type": "integer", "minimum": 1, "maximum": 6, "x-kindgi-tunable": true }, "tone": { "enum": ["friendly", "formal"], "x-kindgi-tunable": true }, "refundLimit": { "type": "number", "minimum": 0 } } } ``` Here Improve may try other sentence limits and tones, and never touches the refund limit. Publish the schema with the block's next version, with `kindgi blocks publish … --schema=@reply-style.schema.json`, as [Publish the blocks](https://docs.kindgi.com/v0.5.0/guides/improve/change-a-prompt-or-setting.md#publish-the-blocks) shows. A version with no marked key has no settings for Improve to try: the readiness check lists `settings` as missing, and says how to fix it. ### Let it try another model or temperature Improve changes the agent's model-settings block (the one it names as `modelSettings`) the same way: mark its keys. It takes `temperature` (0 to 2), `maxOutputTokens`, how much the model thinks, `thinking`, and the model the turn prefers, `model` (with `providerId` for its provider). List the models an Improve run may try as the `model` key's `enum`, and the thinking levels as `thinking`'s: ```json { "type": "object", "properties": { "temperature": { "type": "number", "minimum": 0, "maximum": 1, "x-kindgi-tunable": true }, "thinking": { "enum": ["lowest", "low", "medium"], "x-kindgi-tunable": true }, "model": { "enum": ["claude-sonnet-5-5", "claude-haiku-4-5"], "x-kindgi-tunable": true } } } ``` An Improve run tries each marked key within these bounds, kept inside what model settings allow. A level the model doesn't take is rounded, or not sent to a model that doesn't think, as [Temperature and thinking](https://docs.kindgi.com/v0.5.0/guides/models.md#temperature-and-thinking) describes. Its replays count toward its budget at the models' registered prices, as a [budget](https://docs.kindgi.com/v0.5.0/guides/agents/budgets.md) counts them, so list models registered with their prices. List ones your registered providers serve, too: a preference never excludes, so a model none of them serves replays on the next one in line. - **Name the model in one place.** When the block sets `model` or `providerId`, it's the turn's preference instead of the agent's `preferredModel` and `preferredProvider`. A version that names the model in both places is refused when it's published or deployed, and so is a version derived to pin such a block (a suggested change's candidate). When the agent sets `preferredModel` or `preferredProvider`, an Improve run doesn't try `model` or `providerId`. - **Kept by the same rule.** A new model or temperature is kept only when it does better, as any change is ([How it decides](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#how-it-decides)). There's no cheaper path: a model that does as well for less isn't kept. To move to one, change the block yourself and [test it](https://docs.kindgi.com/v0.5.0/guides/improve/suggest-a-change.md). ## Check it's ready Before you start, `--check` asks whether an Improve run is ready, and what it's estimated to cost. It starts nothing: ```sh pnpm exec kindgi improve runs start --agent=acme.rank.scorer --project= --check --table ``` ```text ITEM STATUS FIX ─────────────── ─────── ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── wording missing Improve rewrites a prompt block's wording with a model. If the instructions are written in code, not in a prompt block, ask your coding agent to move them into one, then deploy. If no model is named to write it, give the run a provider and model, or name them in the version's model-settings block. (in the agent's code, then a deploy) settings ok test-cases (25) ok permission ok one-at-a-time ok project ok Ready: yes Starts from: 1.0.0 (pinned in the project) Estimate: $0.84, its search stops at $5.00 (200 replays, 60 of them the proof's) This month: $3.10 of the Improve schedule's $20.00 ``` - **The items,** in this order: `wording` (a prompt block to reword, and a model to draft it), `settings` (settings marked Improve may tune), `test-cases` (the project's active cases for the agent, and how many), `permission` (`publish` on the agent), `one-at-a-time` (no Improve run of the agent already running in the project) and `project` (a version of the agent runs in the project, and the project isn't suspended). A missing one says where it's fixed, in the agent's code (then a deploy) or in Kindgi, and how. - **Ready** when `wording` or `settings` is there, and every other item is. With `--tier`, each tier named needs its own: `--tier=prompt` needs `wording`. - **Starts from:** the version the project runs, pinned there or the agent's latest. - **The estimate:** the replays the Improve run would make, times the version's mean cost over its recent completed runs. The search's share is at most `--max-cost`, and the proof's comes on top, as [the budget](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#start-an-improve-run) says. None when the version has no cost history. Here, of 25 cases it sets 5 aside and searches on 20. The settings marked in [the example above](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#mark-what-improve-may-change), at `maxSentences` 3 and `tone` `friendly`, give 5 candidates to try first (`maxSentences` at 1, 2, 4 and 6, and `tone` at `formal`). The search replays the 20 cases twice for the version it starts from and once for each candidate: 20 × (2 + 5) = 140. The proof replays every case twice for the best change, and the 5 set aside twice for the version it starts from: (25 + 5) × 2 = 60, the answer's `basis.proofReplays`. That's 200 replays; at a mean of $0.0042 a run, $0.84. When it tries new wording too, each draft is a candidate. - **Treat it as an approximation:** a candidate can cost more or less per run than that mean, drafting new wording costs more, and a second round of settings or a retried draft adds replays the estimate can't know of, so an Improve run can cost more than its estimate. [The budget](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#start-an-improve-run) stops its search; the proof runs after it and isn't capped by it. - **This month:** with an [Improve schedule](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#run-improve-on-a-schedule) on the agent and project, what their Improve runs spent this month (UTC), by hand or by the schedule, against the schedule's month cap. Only the schedule stops at the cap. `--check` exits 3 when it isn't ready. Without `--table`, it prints the answer as JSON: `ready`, `startsFrom`, `items`, `estimate` and `month`. In the API, it's `POST /v1/improve-runs/check`, with the body an Improve run would start from. ## Start an Improve run ```sh pnpm exec kindgi improve runs start --agent=acme.rank.scorer --project= --max-candidates=4 --wait ``` - **The agent and the project:** `--agent`, and `--project`, the project it works in. In the API and the clients, they're `agentId` and `projectId` (`project_id` in Python). - **The version** is the one the project runs: pinned there, else the agent's latest. The Improve run's `startsFrom` says which. - **The corrections:** when it starts, it takes the project's open corrections [sorted to Improve](https://docs.kindgi.com/v0.5.0/guides/improve/fix-what-experts-corrected.md#sort-each-correction), and `takes` lists their ids. [Whose corrections it learns from](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#whose-corrections-it-learns-from) says whose count. - **The test cases:** the project's test cases for the agent, at the version it freezes when it starts (`testSetVersion`). - **What it tries:** `--tier=prompt`, new wording for the prompt block, or `--tier=settings`, new values for the settings it may tune. Repeat `--tier` for both. Left out, it tries every tier that's ready. - **The budget:** `--max-cost` in US dollars, 5 by default and at most 100, and `--max-candidates`, 30 by default and at most 200. They cap the search, which stops at whichever it reaches first. It checks its cost before each run on the test cases, so the one already running can take it a little over `--max-cost`. Its cost counts every run on the test cases it starts, the proof of its best change too. The proof runs after the search, so the budget doesn't stop it, and it can take an Improve run past `--max-cost`. - **How candidates that pass equally are ranked:** `--objective`, `passRate` by default or `weightedPrecisionAtK`, as in a [run on the test cases](https://docs.kindgi.com/v0.5.0/guides/improve/compare-an-agent-version.md#read-the-result). The candidate that passes the most cases is the best; between ones that pass as many, the better score wins, then the one that changes fewer settings. A start that isn't ready is refused with `409 improve-not-ready`, listing every missing item with its fix, as `--check` does. While the agent has an Improve run going in the project, another is refused with `409 improve-run-running`, naming it: one runs at a time. ### How it decides It sets about a fifth of the test cases aside (`setAside` says how many, of how many), and searches on the rest: each candidate runs on those cases, and the best one is the change it may suggest. Then it proves that change on every case of the test set, the ones set aside too, as many times each as the version it started from ran. With fewer than five cases, it sets none aside, and searches and proves on all of them. The change is kept only when its proof passes more cases than the version it started from, and fails none that passed. That's the only rule: a change that does as well for less, or does better on some cases and worse on others, isn't kept. ### What it found It answers at once with the Improve run. `--wait` waits for its outcome, at most 30 minutes, and prints it. An Improve run is `running`, then `completed`, `failed` or `cancelled`, and its `outcome` is one of: - **`suggested`**, with the suggested change it wrote (`suggestedChangeId`), which a person decides: the Improve run asks for its review itself, in the name of whoever started it, so the suggested change waits in reviewers' approvals. An Improve run's suggested change always waits for a reviewer, because Improve, not a person, drafted it. Who may approve it depends on the project's [release gate](https://docs.kindgi.com/v0.5.0/guides/improve/release-an-agent-version.md#gate-a-pin): - **No `approvals`:** someone other than you, when there's another active reviewer who can see the approval. If there isn't, you can approve it yourself, as a registered reviewer. - **`approvals`:** someone other than you, with its role or a higher one. With `"separateApprover": false`, that holds only while there's such a reviewer; if there isn't, you can approve it yourself. - **`nothing-better`**, with why (`reason`). No change it tried was kept. When one got as far as its proof, `holdOut` has the proof's numbers: the version it started from (`baseline`), the change (`candidate`), and the difference (`delta`). - **`cost-cap`**, with what it spent (`spentUsd`). It reached `--max-cost` before it kept a change. - **`failed`**, with why (`message`). Here one tried four settings. Setting `fit` to 0 scored best on the cases it searched, and its proof passed more cases than version 1.0.0 and failed none that passed, so it wrote a suggested change. The cost is 0 on `dev-echo`, which costs nothing: ```json excerpt { "id": "46209449-8d69-42aa-8cdd-78a514d9c848", "agentId": "acme.rank.scorer", "projectId": "8f1e8a52-3c5d-4a5e-9b8e-0d6c2a7f4e11", "startsFrom": { "version": "1.0.0", "follows": "pin" }, "testSetVersion": "1.0.0", "setAside": { "cases": 5, "of": 25 }, "tiers": ["settings"], … "status": "completed", "tried": 4, "costUsd": "0", "outcome": { "kind": "suggested", "suggestedChangeId": "19b0f480-146a-4545-bd6b-37742b592dfe" }, "comparisons": [ { "role": "reference", "part": "search", "testRunId": "6c4d6ce7-6838-4eb1-b2ec-d7992ed86278", "score": 0.55 }, { "role": "candidate", "part": "search", "blockId": "acme.rank.weights", "changed": { "fit": 0 }, "testRunId": "7a0ba425-57dc-43aa-83ce-0e37b9d5b9f4", "score": 0.9 }, … ], … } ``` `comparisons` are its runs on the test cases, each a test run you can open: the version it started from on each part (`reference`), each candidate on the cases it searched (`candidate`), and the proof on all of them (`proof`, part `all`). `kindgi improve runs get ` reads the same Improve run back later. ### Whose corrections it learns from By default, once the project, its workspace, or the agent in the project has an [expert group](https://docs.kindgi.com/v0.5.0/guides/improve/correct-a-runs-output.md#restrict-who-may-correct-as-a-group), an Improve run counts only the corrections recorded under one (`groupWeights: "experts-only"`), so who may steer Improve is a choice you make. Until then it counts every correction, each by its group's weight (`groupWeights: "as-recorded"`), since counting only expert groups would count nothing. A group restricted for a different agent, or in a different project, doesn't count. Set it with the `--group-weights` flag, or `groupWeights` in the API or the clients, to choose either, whatever groups exist. The suggested change an Improve run writes names the corrections on the cases it learned from, never on the rest: the ones its runs on the test cases counted and, for an Improve run on the instructions, the ones its drafter was shown, including the ones with calls. Its `fixes` lists up to 100 of them: first the ones the drafter was shown, case by case, each case's corrections with calls right after its items marked wrong; then the ones behind items marked wrong, then the others. Its `correctionCount` says how many in all. An older runtime leaves it out, and there the length of `fixes` is the count. Its runs on the test cases also grade a correction's calls on each replay and show the result. An Improve run on test cases built before correction ids (`correctionIds`) names none. ## Draft the agent's instructions instead An Improve run can also draft the agent's instructions. The agent version must take its instructions from a [prompt block](https://docs.kindgi.com/v0.5.0/guides/improve/change-a-prompt-or-setting.md) it pins, and a model must draft them: the run's `--provider` and `--model`, or the provider and model the version's [model-settings block](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#let-it-try-another-model-or-temperature) names (`providerId` and `model`). If either is missing, the readiness check lists `wording` as missing. Start an Improve run on the instructions with `--tier=prompt`: ```sh pnpm exec kindgi improve runs start --agent=acme-ops.reply-drafter --project= \ --tier=prompt --provider=anthropic --model=claude-sonnet-5-5 --drafts=3 ``` - `--tier=prompt` drafts the agent's instructions. - `--provider` and `--model` are the provider and model of yours that draft them, given together. Left out, the provider and model the version's model-settings block names draft them; with neither, the prompt tier isn't ready. - `--drafts` is how many drafts of the instructions it makes: 1 to 5, 3 by default. In the API and the clients, they're `tiers: ["prompt"]`, `model` and `candidates`: ```sh curl -X POST "$KINDGI_API_URL/v1/improve-runs" \ -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") -H 'content-type: application/json' \ -d '{"agentId":"acme-ops.reply-drafter","projectId":"","tiers":["prompt"],"model":{"providerId":"anthropic","model":"claude-sonnet-5-5"},"candidates":3}' ``` - `$KINDGI_API_URL` and `$KINDGI_API_TOKEN` are your runtime's URL and API token. - `` is the project it works in. It answers at once (`202`) with the Improve run, as the CLI does without `--wait`. The drafter sees the cases marked wrong, their reasons, what an answer marked wrong should have said or why it was wrong (`shouldSay`, `why`), and what [the corrections on them](https://docs.kindgi.com/v0.5.0/guides/improve/fix-what-experts-corrected.md) say the agent should have done: the calls it should have made and the tools it shouldn't have called, as an example for that case, not a rule. It's shown only the corrections the Improve run [counts](https://docs.kindgi.com/v0.5.0/guides/improve/improve-an-agent.md#whose-corrections-it-learns-from). A draft it writes is tried only if it stays within what the agent already has: - it parses as Liquid; - it reads only the instructions' declared parameters, the variables the current instructions read, `today`, `now`, `agent`, `conversation`, and `settings[""]` of a settings block the version pins; - it names no tool, agent or block id the agent doesn't already use; - it isn't empty or the current instructions, and it's at most twice as long as the current ones: at least 2,000 characters, never over 20,000; - it quotes no test case: a run of 8 or more words in a row from what the drafter was shown is refused, and cut from the why it keeps. A refused draft shows in the Improve run's `comparisons` with its issues, as `refused`, and each drafted one with its `why`. Here a local model drafted instructions reading a variable the agent doesn't have. The drafts are that model's words, and yours will differ: ```sh pnpm exec kindgi improve runs get ``` ```json excerpt { "id": "7d56260c-0b5a-43ef-9c56-164c59bd76f7", … "model": { "model": "qwen2.5:7b", "providerId": "ollama-qwen" }, … "comparisons": [ … { "role": "candidate", "part": "search", "blockId": "acme.rank.prompt", "changed": { "template": "Rank the candidates from most to least preferred. Consider the following weights for each pair: {{score}}. Report the ranking." }, "why": "This template emphasizes ranking from most to least preferred and uses {{score}} to include specific weights, aiming to provide more context for accurate ranking.", "refused": [ { "path": "/template", "message": "it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)" } ], "failed": "The drafted instructions were refused: it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)." } ], … } ``` An Improve run's drafts and reasons live on in it, the suggested change and the published block version, which erasing a person's words doesn't reach. That's why a draft may not quote a test case, and why a person reviews it before it is pinned. ## Read Improve runs back, or stop one ```sh pnpm exec kindgi improve runs list --project= --table pnpm exec kindgi improve runs get pnpm exec kindgi improve runs cancel ``` - `list --table` prints each Improve run's agent and the version it started from, its project, status, the candidates it tried, its cost, what it found and when it started. `--agent`, `--project` and `--status` (`running`, `completed`, `failed` or `cancelled`) narrow it. - `get` adds its runs on the test cases, each a test run you can open. `` is the Improve run's `id`. - `cancel` stops a running Improve run: it ends `cancelled` and writes no suggested change. An Improve run that already ended answers `409 improve-run-finished`. From your app: `improveRuns.start`, `check`, `list`, `get` and `cancel` in TypeScript; `improve_runs` in Python. An endpoint subscribed to `improve-run.finished` gets each Improve run when it ends ([Webhook events and deliveries](https://docs.kindgi.com/v0.5.0/guides/webhooks/webhook-events.md#improve-runs)). ## Run Improve on a schedule A schedule can start an Improve run on an agent in a project when there are enough new corrections since the schedule's last Improve run, and stop for the month at its cap, as [Improve on a schedule](https://docs.kindgi.com/v0.5.0/guides/runs/run-on-a-schedule.md#improve-on-a-schedule) shows. Under `kindgi dev`, a schedule's fire starts no Improve run: `kindgi dev` runs while you work, not around the clock, so a schedule is for a deployed runtime. Start Improve by hand there. ## In the console The same loop runs in the console, on the agent's pages: **Overview**, **Runs**, **Corrections**, **Test set**, **Suggested changes** and **Pinned versions**. The breadcrumbs at the top end on the page you're on, such as **Agents › Returns helper › Overview ▾**, and its ▾ lists the others: - **Overview** has the card **How it gets better**: "People correct its runs. Each correction becomes a case in its test set. Improve tries changes on the test set and suggests the best, and a pin puts one in use." Each of its four lines opens its page, such as "Corrections: 1 open" or "Test set: 1 case". - **Runs** has the agent's runs in the project, as a project's [Runs](https://docs.kindgi.com/v0.5.0/guides/runs/list-runs.md#in-the-console) page lists them, with the same filters. Each run opens under the agent. - **Corrections** has what people said about the agent's runs, and who fixes each one, the ones not fixed yet to start with. The runs waiting for a person to review them are in the menu's **Corrections**, under [To review](https://docs.kindgi.com/v0.5.0/guides/improve/expert-queue-reference.md#to-review). - **Test set** has its cases, with **Add a case**, **Retire** and **Run Improve**, which opens **Suggested changes** at its Improve runs, and its latest test runs, each opening its page. It says what's in it: "What a new version is tested on: this agent's corrections with something to check, and the cases added by hand. Left out: Won't fix and Removed corrections, corrections made on a test run's replays, and retired cases." - **Suggested changes** has its suggested changes, and its Improve runs, each with how it ended, such as **Found nothing better**, and its reason. **Run Improve** starts an Improve run, and the Improve run's page follows it. The suggested change it writes shows its verdict and the figures from its run on the test cases, under **Evidence** the corrections it learned from, each with its verdict and reason and a link to its run, and an **Ask to pin it** button to send it for a decision. **Schedule…** makes a schedule that starts Improve runs for the agent in a project, as [Schedule Improve in the console](https://docs.kindgi.com/v0.5.0/guides/runs/run-on-a-schedule.md#schedule-improve-in-the-console) shows. - **Pinned versions** has where each version is pinned, as [In the console](https://docs.kindgi.com/v0.5.0/guides/improve/pinned-versions-reference.md#in-the-console) shows. A flow's pages are **Overview**, **Runs** and **Corrections**, as [Write a flow](https://docs.kindgi.com/v0.5.0/guides/flows/write-a-flow.md#in-the-console) shows. ## Next - [Suggest a change, and test it](https://docs.kindgi.com/v0.5.0/guides/improve/suggest-a-change.md) - [Correct a run's output](https://docs.kindgi.com/v0.5.0/guides/improve/correct-a-runs-output.md) - [Run on a schedule](https://docs.kindgi.com/v0.5.0/guides/runs/run-on-a-schedule.md)