# Let a pass look for something better

> An improvement pass tries an agent version's marked settings, or drafts prompt templates, on a test set within a budget, and writes its best candidate as a proposal a person reviews.

This guide has a pass search for better settings or a better prompt for an agent version, within a budget, and write its best candidate as a proposal a person reviews.

An **improvement pass** does the search a [proposal](https://docs.kindgi.com/v0.1.8/guides/evals/propose-a-change.md) starts from. It tries candidates on part of a test set, within a budget, proves the best one on the rest, and writes it as a proposal only when it's better. A person still decides: the proposal waits for a reviewer.

## Before you start

A pass needs what a proposal does, as [Propose a change, and test it](https://docs.kindgi.com/v0.1.8/guides/evals/propose-a-change.md#before-you-start) lists:

- **an agent whose prompt or settings are data blocks;**
- **a judged test set;**
- **a live version of the agent for the whole tenant.** Without one, the pass is refused with `proposal-needs-pin`;
- **a deployed runtime,** with Kindgi 0.1's CLI. Under `kindgi dev`, it's refused with `registry-read-only`;
- **`publish` on the agent,** to start one.

## Mark what a pass may tune

A pass changes only what you let it. A settings pass changes only the keys a settings block's schema marks with `"x-kindgi-tunable": true`: a number or an integer with a `minimum` below its `maximum`, or an `enum`. Any other mark is refused when the block is published.

Ask your coding agent: "Let an improvement pass tune my reply length and tone, but not the refund limit." For reference, it marks them in the settings block's schema like this:

```json
{
  "type": "object",
  "properties": {
    "maxSentences": { "type": "integer", "minimum": 1, "maximum": 6, "x-kindgi-tunable": true },
    "tone": { "enum": ["friendly", "formal"], "x-kindgi-tunable": true },
    "refundLimit": { "type": "number", "minimum": 0 }
  }
}
```

Here a pass may try other sentence limits and tones, and never touches the refund limit. Publish the schema with the block's next version, with `kindgi blocks publish … --schema=@reply-style.schema.json`, as [Publish the blocks](https://docs.kindgi.com/v0.1.8/guides/evals/change-a-prompt-or-setting.md#publish-the-blocks) shows. A version that pins no block with a marked key has nothing to tune, and the pass is refused, saying so.

## Start a pass

A pass searches for a better version of the agent's marked settings:

```sh
pnpm exec kindgi proposals improve --agent=acme.rank.scorer --tenant --test-set=acme.rank-prompt-52080 --max-candidates=4 --wait
```

- **The agent and scope:** `--agent` and `--tenant`, or another scope flag.
- **The version** is the one serving the scope, unless `--from-version` names another.
- **The test set:** `--test-set`, a judged one.
- **The budget:** `--max-cost` in US dollars, 5 by default and at most 100, and `--max-candidates`, 30 by default and at most 200. A pass stops at whichever it reaches first. It checks the cost between comparisons, so the one already running can take it a little over `--max-cost`.
- **What better means:** `--objective`, `weightedYesShare` by default or `weightedPrecisionAtK`, as in a [comparison](https://docs.kindgi.com/v0.1.8/guides/evals/compare-an-agent-version.md#read-the-result).
- **How it decides:** it searches on 70% of the test set's cases and proves its best candidate on the other 30%, a split fixed for the pass. Only a candidate that beats the recorded outputs there, by more than the noise, becomes a proposal.

It answers at once with the pass. `--wait` waits for its outcome, at most 30 minutes, and prints it:

- **`proposed`**, with the proposal it wrote, which a reviewer decides: run `kindgi proposals request <proposal-id>`, then decide the approval.
- **`nothing-found`**, with why. No candidate beat the current version by more than the noise, and the reason gives the best one's numbers on the hold-out part. Or the best one is a change an existing proposal already holds. Or nothing could be measured, because no candidate's output had an item the test set's judgments cover. A changed free-text answer is a new item no one has judged: then [judge the new answers, then rescore](https://docs.kindgi.com/v0.1.8/guides/evals/propose-a-change.md#judge-the-new-answers-then-rescore). The reason also says if the pass ran out of budget before comparing the rest.
- **`failed`**, with the reason.

A pass is `running`, then `completed`, `failed` or `cancelled`. Here a pass tried four candidates. On the search part, setting `fit` to 0 scored best, but an earlier proposal already held that change, so the pass wrote none. The cost is 0 on `dev-echo`, which costs nothing:

```json excerpt
{
  "id": "46209449-8d69-42aa-8cdd-78a514d9c848",
  …
  "budget": {
    "maxCostUsd": 5,
    "maxCandidates": 4
  },
  "requestedBy": "user:62e1ffe2-30f4-4dff-948e-ae774107a7db",
  "status": "completed",
  "candidatesEvaluated": 4,
  "costUsd": "0",
  "outcome": {
    "kind": "nothing-found",
    "reason": "The best candidate is the change of proposal 19b0f480-146a-4545-bd6b-37742b592dfe, which is evaluated."
  },
  "comparisons": [
    {
      "role": "reference",
      "part": "search",
      "evalRunId": "6c4d6ce7-6838-4eb1-b2ec-d7992ed86278",
      "score": 0.5714285714285714
    },
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.weights",
      "changed": {
        "fit": 0
      },
      "evalRunId": "7a0ba425-57dc-43aa-83ce-0e37b9d5b9f4",
      "score": 1
    },
    …
  ],
  …
}
```

`kindgi proposals passes get <pass-id>` reads the same pass back later.

### It learns from trusted judgments

A pass counts only judgments recorded under a [restricted judge class](https://docs.kindgi.com/v0.1.8/guides/evals/judge-a-runs-output.md#restrict-who-may-judge-as-a-class) by default, `classWeights: "restricted-only"`. So who may steer it is a choice you make. `"as-recorded"` counts every judgment, by its class's weight: set it in the API or the clients.

## Draft a prompt instead

A pass can also draft prompt templates. The agent version must take its instructions from a [prompt block](https://docs.kindgi.com/v0.1.8/guides/evals/change-a-prompt-or-setting.md) it pins. The CLI starts settings passes, so start a prompt pass from the API or the clients:

```sh
curl -X POST "$KINDGI_API_URL/v1/proposals/improve" \
  -H "Authorization: Bearer $KINDGI_API_TOKEN" -H 'content-type: application/json' \
  -d '{"agentId":"acme-ops.reply-drafter","scope":{"kind":"tenant"},"suiteId":"acme.reply-cases","tiers":["prompt"],"model":{"providerId":"anthropic","model":"claude-sonnet-5-5"},"candidates":3}'
```

- `$KINDGI_API_URL` and `$KINDGI_API_TOKEN` are your runtime's URL and API token.
- `tiers: ["prompt"]` drafts prompt templates.
- `model` is the provider and model of yours that drafts them.
- `candidates` is how many templates it drafts: 1 to 5, 3 by default.

It answers at once (`202`) with the pass, as the CLI does without `--wait`.

The drafter sees the cases judged wrong and their reasons. A template it writes is compared only if it stays within what the agent already has:

- it parses as Liquid;
- it reads only the prompt's declared parameters, the variables the current template reads, `today`, `now`, `agent`, `conversation`, and `settings["<id>"]` of a settings block the version pins;
- it names no tool, agent or block id the agent doesn't already use;
- it isn't empty or the current template, and it's at most twice as long as the current one: at least 2,000 characters, never over 20,000;
- it quotes no test case: a run of 8 or more words in a row from what the drafter was shown is refused, and cut from the hypothesis it keeps.

A refused template shows in the pass's `comparisons` with its issues, as `refused`, and each drafted one with its `hypothesis`. Here a local model drafted a template reading a variable the agent doesn't have. The drafts are that model's words, and yours will differ:

```sh
pnpm exec kindgi proposals passes get <pass-id>
```

```json excerpt
{
  "id": "7d56260c-0b5a-43ef-9c56-164c59bd76f7",
  …
  "model": {
    "model": "qwen2.5:7b",
    "providerId": "ollama-qwen"
  },
  …
  "comparisons": [
    …
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.prompt",
      "changed": {
        "template": "Rank the candidates from most to least preferred. Consider the following weights for each pair: {{score}}. Report the ranking."
      },
      "hypothesis": "This template emphasizes ranking from most to least preferred and uses {{score}} to include specific weights, aiming to provide more context for accurate ranking.",
      "refused": [
        {
          "path": "/template",
          "message": "it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)"
        }
      ],
      "failed": "The drafted template was refused: it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)."
    }
  ],
  …
}
```

A pass's drafts and hypotheses live on in the pass, the proposal and the published block version, which erasing a person's words doesn't reach. That's why a template may not quote a test case, and why a person reviews it before it goes live.

## Read passes back, or stop one

```sh
pnpm exec kindgi proposals passes list --table
pnpm exec kindgi proposals passes get <pass-id>
pnpm exec kindgi proposals passes cancel <pass-id>
```

- `list --table` prints each pass's agent version, scope, status, the candidates compared, the cost and the outcome.
- `get` adds its comparisons, each an eval run you can open. `<pass-id>` is the pass's `id`.
- `cancel` stops a running pass: it ends `cancelled` and writes no proposal. A pass that already ended answers `409 improvement-pass-finished`.

From your app: `proposals.improve` and `improvementPasses.list`, `get` and `cancel` in TypeScript; `proposals.improve` and `improvement_passes` in Python. An endpoint subscribed to `improvement-pass.finished` gets each pass when it ends ([Webhook events and deliveries](https://docs.kindgi.com/v0.1.8/guides/webhooks/webhook-events.md#improvement-passes)).

## Run passes on a schedule

A schedule can start a pass when enough new trusted judgments have come in, as [Improve on a schedule](https://docs.kindgi.com/v0.1.8/guides/runs/run-on-a-schedule.md#improve-on-a-schedule) shows.

## Next

- [Propose a change, and test it](https://docs.kindgi.com/v0.1.8/guides/evals/propose-a-change.md)
- [Judge a run's output](https://docs.kindgi.com/v0.1.8/guides/evals/judge-a-runs-output.md)
- [Run on a schedule](https://docs.kindgi.com/v0.1.8/guides/runs/run-on-a-schedule.md)