# Start an eval run against a suite

POST

/v1/eval-suites/{suiteId}/runs

```sh
curl --request POST \
  --url https://example.com/v1/eval-suites/example/runs \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "projectId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "agentRef": { "agentId": "example", "version": "example" }, "flowRef": { "flowId": "example", "version": "example" }, "dryRun": true, "correlationId": "example", "baseline": "recorded", "reads": "recorded", "repetitions": 1, "k": 1, "versions": { "tools": { "additionalProperty": "example" }, "agents": { "additionalProperty": "example" } }, "classWeights": "as-recorded", "overrides": { "settings": { "additionalProperty": {} }, "prompts": { "additionalProperty": { "template": "example" } } }, "sample": { "part": "search", "seed": "example", "holdOutShare": 1 } }'
```

Dispatches an eval run against the latest version of the suite. Exactly one of `agentRef` / `flowRef` is required — the subject the suite evaluates. Dispatch is per-kind: `accuracy` runs the subject once per case and grades via a configured judge; other kinds return `422 dispatcher-not-registered` unless the deployment registers a dispatcher for them. Nothing starts in a suspended project (`409 project-suspended`), and a run going when its project or tenant stops taking new runs ends `cancelled` at the next case, with the reason in `cancelReason`. Idempotency-Key applies.

## Authorizations

- **[bearerAuth](https://docs.kindgi.com/v0.1.10/reference/api.md#bearerauth)**

## Parameters

### Path Parameters

**suiteId**

required

string

\>= 1 characters

EvalSuiteId — stable string chosen by the caller (e.g. `acme.drafting-accuracy`).

### Header Parameters

**Idempotency-Key**

string

\>= 1 characters

Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per `docs/API-ROUTE-CONVENTIONS.md` §3.1). A retry sent while the first request still runs is answered `409 idempotency-key-in-flight` with `Retry-After`, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets `409 idempotency-key-replay-withheld`, with the first request’s status and when it succeeded (from 0.1.5).

## Request Bodyrequired

application/json

Exactly one of `agentRef` or `flowRef` MUST be supplied. `dryRun: true` returns a plan preview without invoking the subject. For a `judged` suite (a test set), the run is a comparison: `agentRef` or `flowRef` with its `version` is the candidate, replayed on each case without doing anything the past run didn’t (a flow stops at a write the replay refuses); `baseline` (default `'recorded'`), `reads` (default `recorded`), `repetitions` (default 1) and `k` (default 10) set how. With `flowRef`, `versions` runs the flow with some of its agents or tools at other versions; an id the flow doesn’t use, or a version that isn’t published, is refused (`400 validation-failed`, each under `details.issues`).

object

**projectId**

required

Project this belongs to (its content scope). Required: missing, or not a project in the caller’s tenant → `400 bad-input`.

string format: uuid

**agentRef**

object

**agentId**

required

string

**version**

string

/^\d+\\.\d+\\.\d+$/

**flowRef**

object

**flowId**

required

string

**version**

string

/^\d+\\.\d+\\.\d+$/

**dryRun**

boolean

**correlationId**

string

**baseline**

One of:

- **string**

  string

  Allowed values: recorded

- **object**

  object

  **agentId**

  required

  string

  **version**

  required

  string

- **object**

  object

  **live**

  required

  object

  **projectId**

  string

  **segments**

  A segment path in `projectId`, coarse to fine, as live versions resolve it. Needs `projectId`.

  Array\<object>

  <= 8 items

  object

  **key**

  required

  Lowercase, like an identifier: `company`, `contact-role`.

  string

  /^\[a-z]\[a-z0-9\_-]{0,63}$/

  **value**

  required

  string

  \>= 1 characters <= 256 characters

**reads**

string

Allowed values: recorded live

**repetitions**

integer

\>= 1 <= 10

**k**

integer

\>= 1 <= 100

**versions**

Agents and tools a flow runs at other exact versions than the flow version’s pins (“this flow, with `acme.scorer` at 0.4.0”), without publishing a new flow version: a comparison’s flow candidate (`versions` on the start body and the run’s `comparison`), and the flow runs that replay it (a run’s `versions`). Each id must be an agent or tool the flow uses.

object

**tools**

Tool id → exact version.

object

**_key_**

additional properties

string

**agents**

Agent id → exact version, for agent nodes with or without a version of their own.

object

**_key_**

additional properties

string

**classWeights**

Which judgments count: each at its class’s weight (`as-recorded`, the default), or only those recorded while their class was restricted (`Judgment.restricted`), the others weighing 0 (`restricted-only`).

string

Allowed values: as-recorded restricted-only

**overrides**

For an agent candidate: block content its replays run instead of the version’s pinned content (an improvement pass’s search). `settings`: values by settings block id, each a block the version pins, satisfying its schema. `prompts`: a template for the prompt block the version pins, which reads and names only what the agent has (its parameters, the variables the current template reads, the settings blocks it pins, its tools’ and blocks’ ids) and is at most twice as long. Anything else is `400 validation-failed`. A comparison with overrides can’t gate a promotion.

object

**settings**

object

<= 20 properties

**_key_**

additional properties

object

**_key_**

additional properties

any

**prompts**

object

<= 1 properties

**_key_**

additional properties

object

**template**

required

string

**sample**

Only part of the test set’s cases: split once into a hold-out part (about `holdOutShare` of them) and a search part (the rest), stratified by judgment (the cases with a “no” and the others are split on their own, a stratum of two or more giving each part at least one), in the order of a hash of each case id and `seed`. The same seed always splits the same test set the same way. A promotion gate refuses a comparison on the search part (`comparison.sample`).

object

**part**

required

string

Allowed values: search hold-out

**seed**

required

string

\>= 1 characters <= 200 characters

**holdOutShare**

required

number

\>= 0.1 <= 0.9

## Responses

### 201

Eval run started.

application/json

object

**runId**

required

string format: uuid

**dryRunPreview**

object

**_key_**

additional properties

any

##### Examplegenerated

```json
{
  "runId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "dryRunPreview": {}
}
```

### 400

Malformed body (both / neither of `agentRef` / `flowRef`, dispatcher-input-invalid); or `projectId` isn’t a project id (a UUID). Or `reference-out-of-scope`: the suite, the agent or flow, or an agent or tool a flow candidate’s `versions` names, isn’t registered in `projectId`’s project, its org or the tenant; or a version named (by `agentRef`, `flowRef` or `versions.agents`) isn’t one `projectId`’s project runs (`details.issues`, each at the path it was named, whose message names the version). Nothing runs.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 401

Missing / malformed / expired / revoked bearer token.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 404

No eval suite with that id under this tenant; or `projectId` names no project of this tenant (`project-not-found`).

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 409

`project-suspended`: the project is suspended, so no eval run starts in it. Or Idempotency-Key was reused with a different body.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 413

The request body is over the API’s limit (`request-too-large`; `details.maxBytes` says how much).

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 422

No dispatcher registered for the suite kind.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 500

Server error (unmapped domain code or framework crash).

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

## See also

- [`GET /v1/eval-suites/{suiteId}`: Fetch an eval suite (latest version)](https://docs.kindgi.com/v0.1.10/reference/api/operations/evalsuitesget.md)
- [`GET /v1/eval-suites/{suiteId}/versions`: List versions of an eval suite](https://docs.kindgi.com/v0.1.10/reference/api/operations/evalsuitesversionslist.md)
- [`GET /v1/eval-suites/{suiteId}/versions/{version}`: Fetch a specific eval suite version](https://docs.kindgi.com/v0.1.10/reference/api/operations/evalsuitesversionsget.md)
- [`POST /v1/eval-suites/{suiteId}/versions/from-judgments`: Build a test set from judgments](https://docs.kindgi.com/v0.1.10/reference/api/operations/evalsuitesbuildfromjudgments.md)
- [Error code `project-not-found`](https://docs.kindgi.com/v0.1.10/reference/errors.md#project-not-found)
- [Error code `project-suspended`](https://docs.kindgi.com/v0.1.10/reference/errors.md#project-suspended)
- [Error code `reference-out-of-scope`](https://docs.kindgi.com/v0.1.10/reference/errors.md#reference-out-of-scope)
- [Error code `request-too-large`](https://docs.kindgi.com/v0.1.10/reference/errors.md#request-too-large)
- [Every error code](https://docs.kindgi.com/v0.1.10/reference/errors.md)