Start an eval run against a suite

POST

/v1/eval-suites/{suiteId}/runs

const url = 'https://example.com/v1/eval-suites/example/runs';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"projectId":"2489E9AD-2EE2-8E00-8EC9-32D5F69181C0","agentRef":{"agentId":"example","version":"example"},"flowRef":{"flowId":"example","version":"example"},"dryRun":true,"correlationId":"example","baseline":"recorded","reads":"recorded","repetitions":1,"k":1,"versions":{"tools":{"additionalProperty":"example"},"agents":{"additionalProperty":"example"}},"classWeights":"as-recorded","overrides":{"settings":{"additionalProperty":{}},"prompts":{"additionalProperty":{"template":"example"}}},"sample":{"part":"search","seed":"example","holdOutShare":1}}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

curl --request POST \
  --url https://example.com/v1/eval-suites/example/runs \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "projectId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "agentRef": { "agentId": "example", "version": "example" }, "flowRef": { "flowId": "example", "version": "example" }, "dryRun": true, "correlationId": "example", "baseline": "recorded", "reads": "recorded", "repetitions": 1, "k": 1, "versions": { "tools": { "additionalProperty": "example" }, "agents": { "additionalProperty": "example" } }, "classWeights": "as-recorded", "overrides": { "settings": { "additionalProperty": {} }, "prompts": { "additionalProperty": { "template": "example" } } }, "sample": { "part": "search", "seed": "example", "holdOutShare": 1 } }'

Dispatches an eval run against the latest version of the suite. Exactly one of agentRef / flowRef is required — the subject the suite evaluates. Dispatch is per-kind: accuracy runs the subject once per case and grades via a configured judge; other kinds return 422 dispatcher-not-registered unless the deployment registers a dispatcher for them. Nothing starts in a suspended project (409 project-suspended), and a run going when its project or tenant stops taking new runs ends cancelled at the next case, with the reason in cancelReason. Idempotency-Key applies.

Authorizations

bearerAuth

Parameters

Path Parameters

suiteId

required

string

>= 1 characters

EvalSuiteId — stable string chosen by the caller (e.g. acme.drafting-accuracy).

Header Parameters

Idempotency-Key

string

>= 1 characters

Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5).

Request Body^required

application/json

Exactly one of agentRef or flowRef MUST be supplied. dryRun: true returns a plan preview without invoking the subject. For a judged suite (a test set), the run is a comparison: agentRef or flowRef with its version is the candidate, replayed on each case without doing anything the past run didn’t (a flow stops at a write the replay refuses); baseline (default 'recorded'), reads (default recorded), repetitions (default 1) and k (default 10) set how. With flowRef, versions runs the flow with some of its agents or tools at other versions; an id the flow doesn’t use, or a version that isn’t published, is refused (400 validation-failed, each under details.issues).

object

projectId

required

Project this belongs to (its content scope). Required: missing, or not a project in the caller’s tenant → 400 bad-input.

string format: uuid

agentRef

object

agentId

required

string

version

string

/^\d+\.\d+\.\d+$/

flowRef

object

flowId

required

string

version

string

/^\d+\.\d+\.\d+$/

dryRun

boolean

correlationId

string

baseline

One of:

string

Allowed values: recorded

object

live

required

object

projectId

string

segments

A segment path in projectId, coarse to fine, as live versions resolve it. Needs projectId.

Array<object>

<= 8 items

object

key

required

Lowercase, like an identifier: company, contact-role.

string

/^[a-z][a-z0-9_-]{0,63}$/

value

required

string

>= 1 characters <= 256 characters

reads

string

Allowed values: recorded live

repetitions

integer

>= 1 <= 10

integer

>= 1 <= 100

versions

Agents and tools a flow runs at other exact versions than the flow version’s pins (“this flow, with acme.scorer at 0.4.0”), without publishing a new flow version: a comparison’s flow candidate (versions on the start body and the run’s comparison), and the flow runs that replay it (a run’s versions). Each id must be an agent or tool the flow uses.

object

tools

Tool id → exact version.

object

key

additional properties

string

agents

Agent id → exact version, for agent nodes with or without a version of their own.

object

key

additional properties

string

classWeights

Which judgments count: each at its class’s weight (as-recorded, the default), or only those recorded while their class was restricted (Judgment.restricted), the others weighing 0 (restricted-only).

string

Allowed values: as-recorded restricted-only

overrides

For an agent candidate: block content its replays run instead of the version’s pinned content (an improvement pass’s search). settings: values by settings block id, each a block the version pins, satisfying its schema. prompts: a template for the prompt block the version pins, which reads and names only what the agent has (its parameters, the variables the current template reads, the settings blocks it pins, its tools’ and blocks’ ids) and is at most twice as long. Anything else is 400 validation-failed. A comparison with overrides can’t gate a promotion.

object

settings

object

<= 20 properties

key

additional properties

object

key

additional properties

any

prompts

object

<= 1 properties

key

additional properties

object

template

required

string

sample

Only part of the test set’s cases: split once into a hold-out part (about holdOutShare of them) and a search part (the rest), stratified by judgment (the cases with a “no” and the others are split on their own, a stratum of two or more giving each part at least one), in the order of a hash of each case id and seed. The same seed always splits the same test set the same way. A promotion gate refuses a comparison on the search part (comparison.sample).

object

part

required

string

Allowed values: search hold-out

seed

required

string

>= 1 characters <= 200 characters

holdOutShare

required

number

>= 0.1 <= 0.9

Responses

201

Eval run started.

application/json

object

runId

required

string format: uuid

dryRunPreview

object

key

additional properties

any

Example^generated

{
  "runId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "dryRunPreview": {}
}

400

Malformed body (both / neither of agentRef / flowRef, dispatcher-input-invalid); or projectId isn’t a project id (a UUID). Or reference-out-of-scope: the suite, the agent or flow, or an agent or tool a flow candidate’s versions names, isn’t registered in projectId’s project, its org or the tenant; or a version named (by agentRef, flowRef or versions.agents) isn’t one projectId’s project runs (details.issues, each at the path it was named, whose message names the version). Nothing runs.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

401

Missing / malformed / expired / revoked bearer token.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

404

No eval suite with that id under this tenant; or projectId names no project of this tenant (project-not-found).

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

409

project-suspended: the project is suspended, so no eval run starts in it. Or Idempotency-Key was reused with a different body.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

413

The request body is over the API’s limit (request-too-large; details.maxBytes says how much).

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

422

No dispatcher registered for the suite kind.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

500

Server error (unmapped domain code or framework crash).

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

Start an eval run against a suite

Authorizations

Parameters

Path Parameters

Header Parameters

Request Bodyrequired

Responses

201

Examplegenerated

400

Examplegenerated

401

Examplegenerated

404

Examplegenerated

409

Examplegenerated

413

Examplegenerated

422

Examplegenerated

500

Examplegenerated

See also

Request Body^required

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated