# Propose a change, and test it > Draft new settings or a new prompt for an agent version, compare it on a test set, and send it through the scope's gate, with a way back. This guide drafts new settings or a new prompt for an agent version, tests it on a test set, and sends it through the scope's gate, with a way back. A **proposal** is new content for one data block an agent version pins, for one live scope: new settings values, or a new prompt template. You draft it. Kindgi publishes the block version, makes the agent version that runs it, compares it on a test set, and promotes it through the scope's gate when you ask. Nothing serves any scope until then, and a promoted proposal can be rolled back. ## Before you start - **An agent that keeps its prompt or settings as data blocks,** as [Change a prompt or a setting without a deploy](https://docs.kindgi.com/v0.1.9/guides/evals/change-a-prompt-or-setting.md) shows. - **A judged test set** for it, as [Build a test set from judgments](https://docs.kindgi.com/v0.1.9/guides/evals/build-a-test-set.md) shows. - **A live version of the agent for the whole tenant,** as [Make a version live](https://docs.kindgi.com/v0.1.9/guides/evals/release-an-agent-version.md#make-a-version-live) shows: `kindgi agents promote --tenant`. Without one, the version a proposal makes would serve every scope nothing is pinned for. So evaluating is refused with `409 proposal-needs-pin`. - **A deployed runtime,** with Kindgi 0.1's CLI. Under `kindgi dev`, the pack is the source of agents, so evaluating is refused with `registry-read-only`. - **`publish` on the agent** to draft and evaluate, and **`promote`** to request and roll back. ## Draft one A draft says what to change, for which scope, and why: ```sh pnpm exec kindgi proposals draft --agent=acme.rank.scorer --from-version=1.0.0 --tenant \ --block=acme.rank.weights --values='{"recency":0.2,"fit":0.8}' \ --hypothesis="Fit matters more than recency for acme rankings." ``` - **What it changes:** `--agent` and `--from-version` are the agent version. `--block` names a block the version pins. `--values` gives a settings block's new values, and `--template` a prompt block's new template. Either takes a file as `@`. - **For which scope:** `--tenant`, `--org=`, or `--project=` with `--segment=`s for a segment. - **Why:** `--hypothesis` says what the change should improve. Each `--judgment=` keeps a judgment behind the change as evidence. Repeat it for several. It prints the proposal, with its `id` and the status `draft`: ```json { "id": "3bca94f1-61a4-48b5-a3d7-f3664e466613", "agentId": "acme.rank.scorer", "fromVersion": "1.0.0", "scope": { "kind": "tenant" }, "tier": "settings-block", "change": { "blockId": "acme.rank.weights", "content": { "values": { "fit": 0.8, "recency": 0.2 } }, "fromVersion": "1.0.0" }, "hypothesis": "Fit matters more than recency for acme rankings.", "drafter": { "by": "user:62e1ffe2-30f4-4dff-948e-ae774107a7db", "kind": "person" }, "status": "draft", "createdAt": "2026-10-09T13:24:23.787Z", "updatedAt": "2026-10-09T13:24:23.787Z" } ``` The content is checked as publishing that block version would be, and content equal to what the version pins is refused. The same change from the same version for the same scope returns the proposal drafted before. ## Evaluate it An evaluation tests the change on your test set before anything serves it: ```sh pnpm exec kindgi proposals evaluate --test-set=acme.rank-prompt-52080 --wait ``` - `` is the proposal's `id`. - `--test-set` is the judged test set to compare on. - `--wait` waits for the result, at most 30 minutes, and prints the proposal. The first evaluation publishes the block version and derives the agent version that pins it. Its `derivedFrom` names the proposal. Neither serves any scope. The comparison replays the candidate on each case of the test set, as [Compare a version on a test set](https://docs.kindgi.com/v0.1.9/guides/evals/compare-an-agent-version.md) does, and takes the same options: `--objective`, `--reads`, `--repetitions`, `--k` and `--class-weights`. The proposal is `evaluating`, then: - `evaluated`: the candidate beat the recorded outputs on the objective, by more than the noise; - `not-better`: it didn't; - `evaluation-failed`: the comparison failed or was cancelled. Here the candidate scored the same as the recorded outputs on the test set's 20 cases, with a `delta` of 0, so it's `not-better`: ```json excerpt { "id": "3bca94f1-61a4-48b5-a3d7-f3664e466613", … "status": "not-better", "candidate": { "pinsDigest": "sha256:3425842179e45dc5a491306e59ba29f7c634a9f8cdf43e6b9af6809f153b3fa7", "agentVersion": "1.0.2", "blockVersion": "1.0.2" }, "evaluation": { "suiteId": "acme.rank-prompt-52080", "evalRunId": "aefcafd1-53df-480e-b01d-05ae55724507", "objective": "weightedYesShare", "startedAt": "2026-10-09T13:24:24.401Z", "runStatus": "completed", "baseline": 0.3333333333333333, "candidate": 0.3333333333333333, "delta": 0, "cases": 20, "better": false }, … } ``` ### Judge the new answers, then rescore A comparison counts what people have judged. When a change rewrites an agent's answers, each new answer is an item no one has judged. So nothing on the candidate's side is counted, and the proposal can't be better. Here a prompt change from "Answer briefly." to "Answer warmly." rewrote the answer in each of a test set's three cases. The comparison lists them case by case: ```sh pnpm exec kindgi eval-runs show ``` `` is the proposal's `evaluation.evalRunId`. It prints the comparison: ```json excerpt { "runId": "941582fc-9e32-41bb-b148-767712664c6b", … "suiteId": "acme.desk-checks", "suiteVersion": "1.0.0", "kind": "judged", … "status": "completed", … "result": { "perCase": [ { … "caseId": "46b82ad4-1c2a-45f8-a593-a9c912964fd3", "runIds": ["c839338c-cfc1-4bf3-b4e3-c7f24e38a473"], "changes": { "new": [{ "key": "answer", "pointer": "/appended/1/content" }], "kept": [], "dropped": [{ "key": "answer" }] }, … }, … ], "summary": { … "metrics": { "judgedCoverage": { "n": 3, "delta": -1, "weight": 0, "baseline": 1, "baselineN": 3, "candidate": 0, "direction": "higher", "baselineWeight": 3 }, "weightedYesShare": { "n": 0, "delta": null, "weight": 0, "baseline": 1, "baselineN": 3, "candidate": null, "direction": "higher", "baselineWeight": 3 }, … }, … } }, … } ``` `runIds` are the comparison's replay runs. Judge each new answer on its replay run, at the `pointer` the comparison names. A replay's answer is at `/appended/1/content`, not at a turn's `/response/content`. A pointer that isn't in the run's output is refused with `400 item-not-found`: ```sh pnpm exec kindgi judgments add --run=c839338c-cfc1-4bf3-b4e3-c7f24e38a473 --item=answer --pointer=/appended/1/content --yes ``` It prints the judgment as JSON. Then rescore the proposal: ```sh pnpm exec kindgi proposals evaluate --rescore --wait ``` A rescore is a new comparison, with the same test set version and settings, that replays nothing: it scores the same replay runs again, with the judgments made since, and becomes the proposal's evaluation. Here the three new answers were judged good, as the old ones were, so the candidate scores the same as the recorded outputs, and the proposal stays `not-better`. Equal isn't better: a rescore that finds the candidate better makes it `evaluated`. ```json excerpt { "id": "9c8efe90-f4ac-4ce8-bb6f-5cd5ae7a043c", "agentId": "acme.desk", "fromVersion": "1.0.0", … "status": "not-better", "candidate": { "pinsDigest": "sha256:9ba29a87d32957a58fd798077b733c2a61e455a220e1c28ba4b1a97f10a49926", "agentVersion": "1.0.1", "blockVersion": "1.0.1" }, "evaluation": { "suiteId": "acme.desk-checks", "evalRunId": "e6d9ec9a-0341-4d18-a3f3-d8e1289f5dc0", "objective": "weightedYesShare", "startedAt": "2026-10-11T08:09:47.089Z", "runStatus": "completed", "baseline": 1, "candidate": 1, "delta": 0, "cases": 3, "better": false }, … } ``` The new comparison says what it rescored, and how much of what it counts was judged since. `comparison.rescoreOf` names the comparison rescored. Each metric's `freshWeight` is the weight of those judgments, and per case, `candidate[].fresh` and `changes.new[].judged`: ```json excerpt "comparison": { "k": 10, "reads": "recorded", "baseline": "recorded", "rescoreOf": "941582fc-9e32-41bb-b148-767712664c6b", "repetitions": 1 }, … "weightedYesShare": { "n": 3, "delta": 0, "weight": 3, "baseline": 1, "baselineN": 3, "candidate": 1, "direction": "higher", "freshWeight": 3, "baselineWeight": 3 } ``` - **The comparison rescored stays as it was.** A case whose replay run is gone since, such as one deleted with its retention, keeps the scores it had, with `rescored: false`. - **Rescoring needs `publish` on the agent,** from `evaluated`, `not-better`, `refused`, `superseded` or `expired`. - **A comparison you started yourself,** not through a proposal, rescores with `kindgi eval-runs rescore --wait`. - **In the clients:** `proposals.rescore(proposalId)` and `evalRuns.rescore(runId)`, or `eval_runs.rescore(run_id)` in Python. #### In the console The comparison's page says what it couldn't count: "3 new answers this comparison couldn't count", and "Judge each one below, then rescore: a new comparison scores the same replays again, counting your judgments. Nothing runs again, and this comparison stays as it is." Each new answer has its own card, "New, so no judgment in the test set covers it: judge it here, then rescore.", with **Yes** and **No**. **Rescore** beside the note counts what you judged ("3 judged so far."). ![The comparison of acme.desk 1.0.1 on the recorded runs: "3 new answers this comparison couldn't count", Rescore with "3 judged so far.", and the first case, "When does A-1044 ship?", its new answer judged Yes 1 · No 0.](https://docs.kindgi.com/v0.1.9/_astro/rescore-comparison-judged.B1XwH-t0_Z10LtFw.webp) The proposal's page, while nothing could be scored, says "None of the candidate's answers could be scored: they changed, and nobody has judged the new ones yet. Judge them on the comparison, then rescore.", with a link to the comparison. Its **Rescore** asks first: "After people judge the comparison's new answers, a rescore counts them: it scores the same answers on acme.desk-checks again. Nothing runs again, and the earlier comparison stays as it is. The proposal's status follows the rescore." After it, the evaluation reads "Judged yes share: 100% with the change, 100% for the recorded answers (±0 points) on 3 cases.", and the proposal stays **Not better on the test set**. ![The proposal "Change acme.desk-prompt" after the rescore: Not better on the test set; the change from "Answer briefly." to "Answer warmly."; and its evaluation, "Judged yes share: 100% with the change, 100% for the recorded answers (±0 points) on 3 cases."](https://docs.kindgi.com/v0.1.9/_astro/rescore-proposal-after.DO8mALRb_2nX4cY.webp) ## Request it A request promotes the candidate for the proposal's scope, through the scope's gate, with the evaluation as its comparison: ```sh pnpm exec kindgi proposals request --reason="Live check: promote by hand." ``` `--reason` is kept with the promotion. [Gate a release](https://docs.kindgi.com/v0.1.9/guides/evals/release-an-agent-version.md#gate-a-release) describes the gate. The proposal is then: - `promoted`: the candidate is live for the scope; - `in-review`: the gate passed and a reviewer decides, with `kindgi approvals complete`. Then it's `promoted` or `rejected`, `expired` if nobody decides in time, or `superseded` if the scope's live version or its policy changed meanwhile; - `refused`: the gate refused it, listing every check. A request is allowed once the proposal was evaluated, `not-better` included: the gate decides. This guide's proposal stayed `not-better` after its [rescore](https://docs.kindgi.com/v0.1.9/guides/evals/propose-a-change.md#judge-the-new-answers-then-rescore), and the tenant's gate let it through: ```json excerpt { "id": "3bca94f1-61a4-48b5-a3d7-f3664e466613", … "status": "promoted", … "promotion": { "id": "47319459-4131-4ef6-85c6-64afe0514900", "status": "promoted", "liveNow": true }, … } ``` A proposal a [pass](https://docs.kindgi.com/v0.1.9/guides/evals/improve-an-agent.md) wrote, rather than a person, always waits for a reviewer, even where the scope's policy asks for no approval. Requesting needs `promote` on the agent. ## Roll it back, or withdraw it A rollback puts the scope back on the version that served it before: ```sh pnpm exec kindgi proposals rollback --reason="Live check: put it back." ``` It prints the proposal, now `rolled-back`: ```json excerpt { "id": "3bca94f1-61a4-48b5-a3d7-f3664e466613", … "status": "rolled-back", … "rolledBack": { "at": "2026-10-09T13:24:29.234Z", "by": "user:62e1ffe2-30f4-4dff-948e-ae774107a7db", "reason": "Live check: put it back.", "promotionId": "d19efc3c-d08a-44f8-8daa-8e2bdaedc1d8" }, … } ``` The proposal and its versions stay, as history. A rollback needs `promote` on the agent. A withdrawal closes a proposal before review, or after its promotion was refused, superseded or expired: ```sh pnpm exec kindgi proposals withdraw --reason="Superseded by a better draft" ``` It prints the proposal, now `withdrawn`. `--reason` is required, and withdrawing needs `publish` on the agent. ## Find them ```sh pnpm exec kindgi proposals list --agent=acme.rank.scorer --table pnpm exec kindgi proposals get ``` - `--agent` lists one agent's proposals, and `--table` prints a table: each proposal's agent version, scope, block, status, and the evaluation's delta. - `--status`, `--tier=settings-block|prompt-block` and a scope flag narrow the list. A scope flag keeps exactly that scope. - `get` shows the change, the candidate versions, the comparison and the promotion. Proposals are authorized on their agent: `read` to see one. | Status | What it means | | ------------------- | ---------------------------------------------------------- | | `draft` | Not evaluated yet. | | `evaluating` | Its comparison is queued or running. | | `evaluated` | The candidate beat the recorded outputs. | | `not-better` | It didn't. | | `evaluation-failed` | The comparison failed or was cancelled. | | `in-review` | Requested; the gate passed and an approval is open. | | `promoted` | Live for its scope. | | `refused` | Requested; the gate refused it. | | `rejected` | The reviewer rejected it. | | `expired` | The approval expired undecided. | | `superseded` | Approved after the scope's live version or policy changed. | | `rolled-back` | Rolled back through the proposal. | | `withdrawn` | Withdrawn. | From your app, the clients have the same calls: `proposals.create`, `list`, `get`, `evaluate`, `rescore`, `request`, `rollback` and `withdraw`. ## Next - [Let a pass look for something better](https://docs.kindgi.com/v0.1.9/guides/evals/improve-an-agent.md) - [Release an agent version](https://docs.kindgi.com/v0.1.9/guides/evals/release-an-agent-version.md) - [Decide an approval](https://docs.kindgi.com/v0.1.9/guides/approvals/decide-an-approval.md)