Compare a proposal on a test set

POST

/v1/proposals/{proposalId}/evaluate

const url = 'https://example.com/v1/proposals/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/evaluate';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"suiteId":"example","objective":"weightedYesShare","reads":"recorded","repetitions":1,"k":1,"classWeights":"as-recorded","sample":{"part":"search","seed":"example","holdOutShare":1}}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

curl --request POST \
  --url https://example.com/v1/proposals/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/evaluate \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "suiteId": "example", "objective": "weightedYesShare", "reads": "recorded", "repetitions": 1, "k": 1, "classWeights": "as-recorded", "sample": { "part": "search", "seed": "example", "holdOutShare": 1 } }'

The first evaluation publishes the block version and derives the agent version (derivedFrom.proposalId); they serve no scope until a promotion makes them live. Then a comparison eval run replays that version on the test set, against the recorded outputs (baseline: 'recorded'). Needs publish on the agent, and a live version of it for the whole tenant: an agent with none serves its latest version wherever nothing is pinned, so a new version would go live there at once (409 proposal-needs-pin). Allowed from draft, evaluated, not-better, evaluation-failed, refused, superseded and expired.

Authorizations

bearerAuth

Parameters

Path Parameters

proposalId

required

string format: uuid

The proposal id (a UUID).

Header Parameters

Idempotency-Key

string

>= 1 characters

Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5).

Request Body^required

application/json

Compare the proposal’s candidate on a test set. The first evaluation publishes the block version and derives the agent version (both serve nowhere until promoted).

object

suiteId

required

The test set: a judged eval suite.

string

objective

The metric that says whether the candidate is better.

string

default: weightedYesShare

Allowed values: weightedYesShare weightedPrecisionAtK

reads

string

Allowed values: recorded live

repetitions

integer

>= 1 <= 10

integer

>= 1 <= 100

classWeights

string

Allowed values: as-recorded restricted-only

sample

Only part of the test set’s cases: split once into a hold-out part (about holdOutShare of them) and a search part (the rest), stratified by judgment (the cases with a “no” and the others are split on their own, a stratum of two or more giving each part at least one), in the order of a hash of each case id and seed. The same seed always splits the same test set the same way. A promotion gate refuses a comparison on the search part (comparison.sample).

object

part

required

string

Allowed values: search hold-out

seed

required

string

>= 1 characters <= 200 characters

holdOutShare

required

number

>= 0.1 <= 0.9

Responses

202

The proposal, evaluating.

application/json

An improvement proposal: a change to one data block an agent version pins, for one live scope, taken through the same comparison, gate and promotion as any other version.

object

required

string format: uuid

agentId

required

string

fromVersion

required

The agent version the change applies to.

string

scope

required

One of: discriminator: kind

The agent’s default for the whole tenant.

object

kind

required

string

Allowed values: tenant

A segment path within a project: it covers every run whose path starts with it.

object

kind

required

string

Allowed values: segment

projectId

required

string format: uuid

path

required

Array<object>

>= 1 items <= 8 items

object

key

required

Lowercase, like an identifier: company, contact-role.

string

/^[a-z][a-z0-9_-]{0,63}$/

value

required

string

>= 1 characters <= 256 characters

tier

required

What a proposal changes: a settings block (new values) or a prompt block (a new template) the agent version pins.

string

Allowed values: settings-block prompt-block

change

required

object

blockId

required

The data block the change is to.

string

fromVersion

required

The block version the agent version pins: what’s being changed.

string

content

required

The new content: { values } for a settings block (its schema carries over), { template } for a prompt block (its parameters carry over).

object

values

object

key

additional properties

any

template

string

hypothesis

required

What the change should improve, and why.

string

evidence

object

judgmentIds

Array<string>

drafter

required

Who wrote the proposal: a person, or one of the runtime’s drafters.

object

kind

required

string

Allowed values: person settings-optimizer prompt-drafter

For a person: user:<id> (or service:<id>).

string

version

For a drafter: the drafter’s version.

string

model

For a drafter that used a model: which.

object

providerId

required

string

model

required

string

passId

For a drafter: the improvement pass that drafted it (GET /v1/improvement-passes/{passId}).

string

status

required

Where a proposal stands, from its comparison and its promotion (never stored). draft: not evaluated yet. evaluating: its comparison is queued or running. evaluated: the candidate beat the recorded outputs on the objective metric by more than the noise (the spread, with more than one repetition). not-better: it didn’t. evaluation-failed: the comparison failed or was cancelled. in-review: requested; the gate passed and an approval is open. promoted: live for the scope (promotion.liveNow says whether it still serves it). refused: the gate refused it. rejected: the reviewer rejected it. expired: the approval expired undecided. superseded: approved after the scope’s live version or policy changed. rolled-back: rolled back through the proposal. withdrawn: withdrawn.

string

Allowed values: draft evaluating evaluated not-better evaluation-failed in-review promoted refused rejected expired superseded rolled-back withdrawn

candidate

The versions evaluating the proposal published. They serve no scope until a promotion makes the agent version live.

object

agentVersion

required

The derived agent version (its derivedFrom.proposalId names the proposal).

string

blockVersion

required

The block version published from the proposal’s content.

string

pinsDigest

required

string

evaluation

The comparison the proposal was evaluated with, and what it found on the objective metric (the full summary is the eval run’s).

object

evalRunId

required

string format: uuid

suiteId

required

The test set (a judged eval suite).

string

objective

required

string

Allowed values: weightedYesShare weightedPrecisionAtK

startedAt

required

string format: date-time

runStatus

The eval run’s status. Absent when the run can’t be read.

string

Allowed values: pending running completed failed cancelled

baseline

The recorded outputs’ score; null without judged evidence.

number | null

candidate

The candidate’s score.

number | null

delta

number | null

spread

With more than one repetition: the candidate’s max − min, the noise a delta must beat.

number

cases

integer

better

Set once the comparison finished.

boolean

promotion

The promotion the proposal’s request made.

object

required

string

status

required

string

Allowed values: promoted pending-approval refused superseded rejected expired

approvalId

The approval a request in review waits on.

string

liveNow

For a promoted proposal: whether its version still serves the scope.

boolean

rolledBack

object

required

string format: date-time

promotionId

required

The rollback’s own promotion row.

string

required

string

reason

string

withdrawn

object

required

string format: date-time

required

string

reason

required

string

createdAt

required

string format: date-time

updatedAt

required

string format: date-time

Example

{
  "scope": {
    "kind": "tenant"
  },
  "tier": "settings-block",
  "drafter": {
    "kind": "person"
  },
  "status": "draft",
  "evaluation": {
    "objective": "weightedYesShare",
    "runStatus": "pending"
  },
  "promotion": {
    "status": "promoted"
  }
}

400

Malformed request body.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

401

Missing / malformed / expired / revoked bearer token.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

404

No such proposal, or no such test set.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

409

proposal-needs-pin: the agent has no live version for the whole tenant. proposal-invalid-state-transition: the proposal’s status doesn’t allow it. registry-read-only: the agent registry takes no writes (under kindgi dev).

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

500

Server error (unmapped domain code or framework crash).

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

Compare a proposal on a test set

Authorizations

Parameters

Path Parameters

Header Parameters

Request Bodyrequired

Responses

202

Example

400

Examplegenerated

401

Examplegenerated

404

Examplegenerated

409

Examplegenerated

500

Examplegenerated

Request Body^required

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated