Let a pass look for something better

This guide has a pass search for better settings or a better prompt for an agent version, within a budget, and write its best candidate as a proposal a person reviews.

An improvement pass does the search a proposal starts from. It tries candidates on part of a test set, within a budget, proves the best one on the rest, and writes it as a proposal only when it's better. A person still decides: the proposal waits for a reviewer.

Before you start

A pass needs what a proposal does, as Propose a change, and test it lists:

• an agent whose prompt or settings are data blocks;
• a judged test set;
• a live version of the agent for the whole tenant. Without one, the pass is refused with proposal-needs-pin;
• a deployed runtime, with Kindgi 0.1's CLI. Under kindgi dev, it's refused with registry-read-only;
• publish on the agent, to start one.

Mark what a pass may tune

A pass changes only what you let it. A settings pass changes only the keys a settings block's schema marks with "x-kindgi-tunable": true: a number or an integer with a minimum below its maximum, or an enum. Any other mark is refused when the block is published.

Ask your coding agent: "Let an improvement pass tune my reply length and tone, but not the refund limit." For reference, it marks them in the settings block's schema like this:

{
  "type": "object",
  "properties": {
    "maxSentences": { "type": "integer", "minimum": 1, "maximum": 6, "x-kindgi-tunable": true },
    "tone": { "enum": ["friendly", "formal"], "x-kindgi-tunable": true },
    "refundLimit": { "type": "number", "minimum": 0 }
  }
}

Here a pass may try other sentence limits and tones, and never touches the refund limit. Publish the schema with the block's next version, with kindgi blocks publish … --schema=@reply-style.schema.json, as Publish the blocks shows. A version that pins no block with a marked key has nothing to tune, and the pass is refused, saying so.

Start a pass

A pass searches for a better version of the agent's marked settings:

pnpm exec kindgi proposals improve --agent=acme.rank.scorer --tenant --test-set=acme.rank-prompt-52080 --max-candidates=4 --wait

• The agent and scope: --agent and --tenant, or another scope flag.
• The version is the one serving the scope, unless --from-version names another.
• The test set: --test-set, a judged one.
• The budget: --max-cost in US dollars, 5 by default and at most 100, and --max-candidates, 30 by default and at most 200. A pass stops at whichever it reaches first. It checks the cost between comparisons, so the one already running can take it a little over --max-cost.
• What better means: --objective, weightedYesShare by default or weightedPrecisionAtK, as in a comparison.
• How it decides: it searches on 70% of the test set's cases and proves its best candidate on the other 30%, a split fixed for the pass. Only a candidate that beats the recorded outputs there, by more than the noise, becomes a proposal.

It answers at once with the pass. --wait waits for its outcome, at most 30 minutes, and prints it:

• proposed, with the proposal it wrote, which a reviewer decides: run kindgi proposals request <proposal-id>, then decide the approval.
• nothing-found, with why. No candidate beat the current version by more than the noise, and the reason gives the best one's numbers on the hold-out part. Or the best one is a change an existing proposal already holds. Or nothing could be measured, because no candidate's output had an item the test set's judgments cover. A changed free-text answer is a new item no one has judged: then judge the new answers, then rescore. The reason also says if the pass ran out of budget before comparing the rest.
• failed, with the reason.

A pass is running, then completed, failed or cancelled. Here a pass tried four candidates. On the search part, setting fit to 0 scored best, but an earlier proposal already held that change, so the pass wrote none. The cost is 0 on dev-echo, which costs nothing:

{
  "id": "46209449-8d69-42aa-8cdd-78a514d9c848",
  …
  "budget": {
    "maxCostUsd": 5,
    "maxCandidates": 4
  },
  "requestedBy": "user:62e1ffe2-30f4-4dff-948e-ae774107a7db",
  "status": "completed",
  "candidatesEvaluated": 4,
  "costUsd": "0",
  "outcome": {
    "kind": "nothing-found",
    "reason": "The best candidate is the change of proposal 19b0f480-146a-4545-bd6b-37742b592dfe, which is evaluated."
  },
  "comparisons": [
    {
      "role": "reference",
      "part": "search",
      "evalRunId": "6c4d6ce7-6838-4eb1-b2ec-d7992ed86278",
      "score": 0.5714285714285714
    },
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.weights",
      "changed": {
        "fit": 0
      },
      "evalRunId": "7a0ba425-57dc-43aa-83ce-0e37b9d5b9f4",
      "score": 1
    },
    …
  ],
  …
}

Excerpt

kindgi proposals passes get <pass-id> reads the same pass back later.

It learns from trusted judgments

A pass counts only judgments recorded under a restricted judge class by default, classWeights: "restricted-only". So who may steer it is a choice you make. "as-recorded" counts every judgment, by its class's weight: set it in the API or the clients.

Draft a prompt instead

A pass can also draft prompt templates. The agent version must take its instructions from a prompt block it pins. The CLI starts settings passes, so start a prompt pass from the API or the clients:

curl -X POST "$KINDGI_API_URL/v1/proposals/improve" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") -H 'content-type: application/json' \
  -d '{"agentId":"acme-ops.reply-drafter","scope":{"kind":"tenant"},"suiteId":"acme.reply-cases","tiers":["prompt"],"model":{"providerId":"anthropic","model":"claude-sonnet-5-5"},"candidates":3}'

• $KINDGI_API_URL and $KINDGI_API_TOKEN are your runtime's URL and API token.
• tiers: ["prompt"] drafts prompt templates.
• model is the provider and model of yours that drafts them.
• candidates is how many templates it drafts: 1 to 5, 3 by default.

It answers at once (202) with the pass, as the CLI does without --wait.

The drafter sees the cases judged wrong and their reasons. A template it writes is compared only if it stays within what the agent already has:

• it parses as Liquid;
• it reads only the prompt's declared parameters, the variables the current template reads, today, now, agent, conversation, and settings["<id>"] of a settings block the version pins;
• it names no tool, agent or block id the agent doesn't already use;
• it isn't empty or the current template, and it's at most twice as long as the current one: at least 2,000 characters, never over 20,000;
• it quotes no test case: a run of 8 or more words in a row from what the drafter was shown is refused, and cut from the hypothesis it keeps.

A refused template shows in the pass's comparisons with its issues, as refused, and each drafted one with its hypothesis. Here a local model drafted a template reading a variable the agent doesn't have. The drafts are that model's words, and yours will differ:

pnpm exec kindgi proposals passes get <pass-id>

{
  "id": "7d56260c-0b5a-43ef-9c56-164c59bd76f7",
  …
  "model": {
    "model": "qwen2.5:7b",
    "providerId": "ollama-qwen"
  },
  …
  "comparisons": [
    …
    {
      "role": "candidate",
      "part": "search",
      "blockId": "acme.rank.prompt",
      "changed": {
        "template": "Rank the candidates from most to least preferred. Consider the following weights for each pair: {{score}}. Report the ranking."
      },
      "hypothesis": "This template emphasizes ranking from most to least preferred and uses {{score}} to include specific weights, aiming to provide more context for accurate ranking.",
      "refused": [
        {
          "path": "/template",
          "message": "it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)"
        }
      ],
      "failed": "The drafted template was refused: it reads \"score\", which the agent doesn't have (a declared parameter, a variable the current template reads, or a settings block the version pins)."
    }
  ],
  …
}

Excerpt

A pass's drafts and hypotheses live on in the pass, the proposal and the published block version, which erasing a person's words doesn't reach. That's why a template may not quote a test case, and why a person reviews it before it goes live.

Read passes back, or stop one

pnpm exec kindgi proposals passes list --table
pnpm exec kindgi proposals passes get <pass-id>
pnpm exec kindgi proposals passes cancel <pass-id>

• list --table prints each pass's agent version, scope, status, the candidates compared, the cost and the outcome.
• get adds its comparisons, each an eval run you can open. <pass-id> is the pass's id.
• cancel stops a running pass: it ends cancelled and writes no proposal. A pass that already ended answers 409 improvement-pass-finished.

From your app: proposals.improve and improvementPasses.list, get and cancel in TypeScript; proposals.improve and improvement_passes in Python. An endpoint subscribed to improvement-pass.finished gets each pass when it ends (Webhook events and deliveries).

Run passes on a schedule

A schedule can start a pass when enough new trusted judgments have come in, as Improve on a schedule shows.

Next

• Propose a change, and test it
• Judge a run's output
• Run on a schedule