Compare a proposal on a test set
const url = 'https://example.com/v1/proposals/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/evaluate';const options = { method: 'POST', headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'}, body: '{"suiteId":"example","objective":"weightedYesShare","reads":"recorded","repetitions":1,"k":1,"classWeights":"as-recorded","sample":{"part":"search","seed":"example","holdOutShare":1}}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://example.com/v1/proposals/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/evaluate \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "suiteId": "example", "objective": "weightedYesShare", "reads": "recorded", "repetitions": 1, "k": 1, "classWeights": "as-recorded", "sample": { "part": "search", "seed": "example", "holdOutShare": 1 } }'The first evaluation publishes the block version and derives the agent version (derivedFrom.proposalId); they serve no scope until a promotion makes them live. Then a comparison eval run replays that version on the test set, against the recorded outputs (baseline: 'recorded'). Needs publish on the agent, and a live version of it for the whole tenant: an agent with none serves its latest version wherever nothing is pinned, so a new version would go live there at once (409 proposal-needs-pin). Allowed from draft, evaluated, not-better, evaluation-failed, refused, superseded and expired.
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”The proposal id (a UUID).
Header Parameters
Section titled “Header Parameters”Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5).
Request Bodyrequired
Section titled “Request Bodyrequired”Compare the proposal’s candidate on a test set. The first evaluation publishes the block version and derives the agent version (both serve nowhere until promoted).
object
The test set: a judged eval suite.
The metric that says whether the candidate is better.
Only part of the test set’s cases: split once into a hold-out part (about holdOutShare of them) and a search part (the rest), stratified (the cases that carry a correction, then of the others the cases with a “no” and the rest, each split on its own, a stratum of two or more giving each part at least one, so both parts check a correction when two cases carry one), in the order of a hash of each case id and seed. The same seed always splits the same test set the same way. A promotion gate refuses a comparison on the search part (comparison.sample).
object
Responses
Section titled “Responses”The proposal, evaluating.
An improvement proposal: a change to one data block an agent version pins, for one live scope, taken through the same comparison, gate and promotion as any other version.
object
The agent version the change applies to.
The agent’s default for the whole tenant.
object
An org’s projects.
object
One project.
object
A segment path within a project: it covers every run whose path starts with it.
object
object
Lowercase, like an identifier: company, contact-role.
What a proposal changes: a settings block (new values) or a prompt block (a new template) the agent version pins.
object
The data block the change is to.
The block version the agent version pins: what’s being changed.
The new content: { values } for a settings block (its schema carries over), { template } for a prompt block (its parameters carry over).
object
object
What the change should improve, and why.
What a proposal rests on.
object
The judgments that motivated it. An improvement pass’s proposal names the judgments it learned from on the cases it searched, never the hold-out: those its comparisons counted and, for a prompt pass, those its drafter was shown. At most 100: first those its drafter was shown, then those behind items judged wrong, then the rest.
How many judgments it rests on, when judgmentIds names only the first of them. Absent when judgmentIds names them all.
Who wrote the proposal: a person, or one of the runtime’s drafters.
object
For a person: user:<id> (or service:<id>).
For a drafter: the drafter’s version.
For a drafter that used a model: which.
object
For a drafter: the improvement pass that drafted it (GET /v1/improvement-passes/{passId}).
Where a proposal stands, from its comparison and its promotion (never stored). draft: not evaluated yet. evaluating: its comparison is queued or running. evaluated: the candidate beat the recorded outputs on the objective metric by more than the noise (the spread, with more than one repetition). Under evaluation.rule: 'cheaper' it means instead: no worse than the current version on the same cases and costing less, so evaluated doesn’t always mean better. With evaluation.decisions (a model-settings change), it also means no correction the current version passes failed. not-better: it didn’t. evaluation-failed: the comparison failed or was cancelled. in-review: requested; the gate passed and an approval is open. promoted: live for the scope (promotion.liveNow says whether it still serves it). refused: the gate refused it. rejected: the reviewer rejected it. expired: the approval expired undecided. superseded: approved after the scope’s live version or policy changed. rolled-back: rolled back through the proposal. withdrawn: withdrawn.
The versions evaluating the proposal published. They serve no scope until a promotion makes the agent version live.
object
The derived agent version (its derivedFrom.proposalId names the proposal).
The block version published from the proposal’s content.
The comparison the proposal was evaluated with, and what it found on the objective metric (the full summary is the eval run’s).
object
The test set (a judged eval suite).
The eval run’s status. Absent when the run can’t be read.
The recorded outputs’ score; null without judged evidence.
The candidate’s score.
With more than one repetition: the candidate’s max − min, the noise a delta must beat.
Set once the comparison finished.
How the candidate is kept. better: it beats the recorded outputs by more than the noise. cheaper: it’s no worse than the current version replayed on the same cases (beyond that run’s spread) and costs less (beyond the cost’s own spread), with no errored case; better may then be false while the status is evaluated. Absent: better.
What the replays cost in model calls on the same cases, in US dollars, averaged over the repetitions: the current version’s and the candidate’s. Set for a change held to the current version once both comparisons finished, when both reported a cost.
object
The corrections the current version was checked on, replayed on the same cases, against the candidate’s. A model-settings change is kept only with at least one checked and none broken. Set once both comparisons finished.
object
The current version passed them, and so did the candidate.
The current version passed them; the candidate didn’t.
Neither passed them. The rest of checked the candidate fixed.
The promotion the proposal’s request made.
object
The approval a request in review waits on.
For a promoted proposal: whether its version still serves the scope.
object
The rollback’s own promotion row.
object
Example
{ "scope": { "kind": "tenant" }, "tier": "settings-block", "drafter": { "kind": "person" }, "status": "draft", "evaluation": { "objective": "weightedYesShare", "runStatus": "pending", "rule": "better" }, "promotion": { "status": "promoted" }}Malformed request body. Or reference-out-of-scope: the test set isn’t registered in the agent’s project, its org or the tenant (details.issues).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Missing / malformed / expired / revoked bearer token.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}No such proposal, or no such test set.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}proposal-needs-pin: the agent has no live version for the whole tenant. proposal-invalid-state-transition: the proposal’s status doesn’t allow it. registry-read-only: the agent registry takes no writes, not even the version a proposal makes; or, under kindgi dev, the proposal’s scope is narrower than the whole tenant, since the pack’s files serve every scope.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The request body is over the API’s limit (request-too-large; details.maxBytes says how much).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}rate-limit-exceeded with details.limit: "schema-compile-time", on a server with a compile budget (Kindgi Cloud): this tenant’s schemas spent details.budgetMs compiling in the last 60 seconds. Nothing was stored; Retry-After says when to send it again.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Server error (unmapped domain code or framework crash).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}See also
GET /v1/proposals/{proposalId}: Fetch an improvement proposalPOST /v1/proposals/{proposalId}/rescore: Rescore a proposal's latest evaluationPOST /v1/proposals/{proposalId}/request: Request a proposal's promotion for its scopePOST /v1/proposals/{proposalId}/rollback: Roll back a promoted proposal- Error code
proposal-invalid-state-transition - Error code
proposal-needs-pin - Error code
rate-limit-exceeded - Error code
reference-out-of-scope - Error code
registry-read-only - Error code
request-too-large - Every error code