Start an eval run against a suite

POST

/v1/eval-suites/{suiteId}/runs

curl --request POST \
  --url https://example.com/v1/eval-suites/example/runs \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "projectId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "agentRef": { "agentId": "example", "version": "example" }, "flowRef": { "flowId": "example", "version": "example" }, "dryRun": true, "correlationId": "example" }'

• Same-origin — path relative to the deployment root.

Dispatches an eval run against the latest version of the suite. Exactly one of agentRef / flowRef is required — the subject the suite evaluates. Dispatch is per-kind: accuracy runs the subject once per case and grades via a configured judge; other kinds return 422 dispatcher-not-registered unless the deployment registers a dispatcher for them. Idempotency-Key applies.

Authorizations

• bearerAuth

Parameters

Path Parameters

suiteId

required

string

>= 1 characters

EvalSuiteId — stable string chosen by the caller (e.g. acme.drafting-accuracy).

Header Parameters

Idempotency-Key

string

>= 1 characters

Caller-supplied idempotency key. Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1).

Request Body^required

Media typeapplication/json

Exactly one of agentRef or flowRef MUST be supplied. dryRun: true returns a plan preview without invoking the subject.

object

projectId

required

Project this belongs to (its content scope). Required: missing, or not a project in the caller’s tenant → 400 bad-input.

string format: uuid

agentRef

object

agentId

required

string

version

string

/^\d+\.\d+\.\d+$/

flowRef

object

flowId

required

string

version

string

/^\d+\.\d+\.\d+$/

dryRun

boolean

correlationId

string

Example^generated

{
  "projectId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "agentRef": {
    "agentId": "example",
    "version": "example"
  },
  "flowRef": {
    "flowId": "example",
    "version": "example"
  },
  "dryRun": true,
  "correlationId": "example"
}

Responses

201

Eval run started.

Media typeapplication/json

object

runId

required

string format: uuid

dryRunPreview

object

key

additional properties

any

Example^generated

{
  "runId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "dryRunPreview": {}
}

400

Malformed body (both / neither of agentRef / flowRef, dispatcher-input-invalid).

Media typeapplication/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

401

Missing / malformed / expired / revoked bearer token.

Media typeapplication/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

404

No eval suite with that id under this tenant.

Media typeapplication/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

409

Idempotency-Key was reused with a different body, or resource-state conflict.

Media typeapplication/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

422

No dispatcher registered for the suite kind.

Media typeapplication/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

500

Server error (unmapped domain code or framework crash).

Media typeapplication/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}