# Start a test run against a test set

POST

/v1/test-sets/{testSetId}/runs

```sh
curl --request POST \
  --url https://example.com/v1/test-sets/example/runs \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "projectId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "agentRef": { "agentId": "example", "version": "example" }, "flowRef": { "flowId": "example", "version": "example" }, "dryRun": true, "correlationId": "example", "baseline": "recorded", "reads": "recorded", "repetitions": 1, "k": 1, "versions": { "tools": { "additionalProperty": "example" }, "agents": { "additionalProperty": "example" } }, "groupWeights": "as-recorded", "overrides": { "settings": { "additionalProperty": {} }, "prompts": { "additionalProperty": { "template": "example" } } }, "sample": { "part": "search", "seed": "example", "holdOutShare": 1 } }'
```

Dispatches a test run against the latest version of the test set. An agent’s own test set (`agent.<agentId>.<projectId>`, read at `GET /v1/agents/{agentId}/test-set`) is frozen first, as a suggested change’s check freezes it, and the run records the frozen version as `testSetVersion`; a dry run freezes it too; it runs only its agent, in its project. Exactly one of `agentRef` / `flowRef` is required — the subject the test set evaluates. Dispatch is per-kind: `accuracy` runs the subject once per case and grades via a configured judge; other kinds return `422 dispatcher-not-registered` unless the deployment registers a dispatcher for them. Nothing starts in a suspended project (`409 project-suspended`), and a run going when its project or tenant stops taking new runs ends `cancelled` at the next case, with the reason in `cancelReason`. Idempotency-Key applies.

## Authorizations

- **[bearerAuth](https://docs.kindgi.com/v0.4.0/reference/api.md#bearerauth)**

## Parameters

### Path Parameters

**testSetId**

required

string

\>= 1 characters

TestSetId — stable string chosen by the caller (e.g. `acme.drafting-accuracy`).

### Header Parameters

**Idempotency-Key**

string

\>= 1 characters

Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per `docs/API-ROUTE-CONVENTIONS.md` §3.1). A retry sent while the first request still runs is answered `409 idempotency-key-in-flight` with `Retry-After`, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets `409 idempotency-key-replay-withheld`, with the first request’s status and when it succeeded (from 0.1.5). A request whose key store can’t be read before it runs is answered `503 idempotency-store-unavailable` with `Retry-After`: nothing ran, so retry with the same key (from 0.3.0).

## Request Bodyrequired

application/json

Exactly one of `agentRef` or `flowRef` MUST be supplied. `dryRun: true` returns a plan preview without invoking the subject. For a `corrections` test set (a test set), the run is a comparison: `agentRef` or `flowRef` with its `version` is the candidate, replayed on each case without doing anything the past run didn’t (a flow stops at a write the replay refuses); `baseline` (default `'recorded'`), `reads` (default `recorded`), `repetitions` (default 1) and `k` (default 10) set how. With `flowRef`, `versions` runs the flow with some of its agents or tools at other versions; an id the flow doesn’t use, or a version that isn’t published, is refused (`400 validation-failed`, each under `details.issues`).

object

**projectId**

required

Project this belongs to (its content scope). Required: missing, or not a project in the caller’s tenant → `400 bad-input`.

string format: uuid

**agentRef**

object

**agentId**

required

string

**version**

string

/^\d+\\.\d+\\.\d+$/

**flowRef**

object

**flowId**

required

string

**version**

string

/^\d+\\.\d+\\.\d+$/

**dryRun**

boolean

**correlationId**

An id of your own, kept on the run. It may not start with `pin:` or `latest:` (`bad-input`): those are the test runs Kindgi starts itself, for a pin change or Latest moving.

string

**baseline**

One of:

- **string**

  string

  Allowed values: recorded

- **object**

  object

  **agentId**

  required

  string

  **version**

  required

  string

- **object**

  object

  **pinned**

  required

  object

  **projectId**

  string

  **segments**

  A segment path in `projectId`, coarse to fine, as pinned versions resolve it. Needs `projectId`.

  Array\<object>

  <= 8 items

  object

  **key**

  required

  Lowercase, like an identifier: `company`, `contact-role`.

  string

  /^\[a-z]\[a-z0-9\_-]{0,63}$/

  **value**

  required

  string

  \>= 1 characters <= 256 characters

**reads**

string

Allowed values: recorded live

**repetitions**

integer

\>= 1 <= 10

**k**

integer

\>= 1 <= 100

**versions**

Agents and tools a flow runs at other exact versions than the flow version’s pins (“this flow, with `acme.scorer` at 0.4.0”), without publishing a new flow version: a comparison’s flow candidate (`versions` on the start body and the run’s `comparison`), and the flow runs that replay it (a run’s `versions`). Each id must be an agent or tool the flow uses.

object

**tools**

Tool id → exact version.

object

**_key_**

additional properties

string

**agents**

Agent id → exact version, for agent nodes with or without a version of their own.

object

**_key_**

additional properties

string

**groupWeights**

Which corrections count: each at its group’s weight (`as-recorded`, the default), or only those recorded while their group was restricted (`Correction.expert`), the others weighing 0 (`experts-only`).

string

Allowed values: as-recorded experts-only

**overrides**

For an agent candidate: block content its replays run instead of the version’s pinned content (an Improve run’s search). `settings`: values by settings block id, each a block the version pins, satisfying its schema. `prompts`: a template for the prompt block the version pins, which reads and names only what the agent has (its parameters, the variables the current template reads, the settings blocks it pins, its tools’ and blocks’ ids) and is at most twice as long. Anything else is `400 validation-failed`. A comparison with overrides can’t gate a pin.

object

**settings**

object

<= 20 properties

**_key_**

additional properties

object

**_key_**

additional properties

any

**prompts**

object

<= 1 properties

**_key_**

additional properties

object

**template**

required

string

**sample**

Only part of the test set’s cases: split once into a hold-out part (about `holdOutShare` of them) and a search part (the rest), stratified (the cases that carry a correction, then of the others the cases with a “no” and the rest, each split on its own, a stratum of two or more giving each part at least one, so both parts check a correction when two cases carry one), in the order of a hash of each case id and `seed`. The same seed always splits the same test set the same way. A release gate refuses a comparison on the search part (`comparison.sample`).

object

**part**

required

string

Allowed values: search hold-out

**seed**

required

string

\>= 1 characters <= 200 characters

**holdOutShare**

required

number

\>= 0.1 <= 0.9

## Responses

### 201

Test run started.

application/json

object

**testRunId**

required

string format: uuid

**dryRunPreview**

object

**_key_**

additional properties

any

##### Examplegenerated

```json
{
  "testRunId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "dryRunPreview": {}
}
```

### 400

Malformed body (both / neither of `agentRef` / `flowRef`, dispatcher-input-invalid); or `projectId` isn’t a project id (a UUID). Or `reference-out-of-scope`: the test set, the agent or flow, or an agent or tool a flow candidate’s `versions` names, isn’t registered in `projectId`’s project, its org or the tenant; or a version named (by `agentRef`, `flowRef` or `versions.agents`) isn’t one `projectId`’s project runs (`details.issues`, each at the path it was named, whose message names the version). Or `bad-input` for an agent’s own test set run with another agent, in another project, or with `flowRef`: the message names the test set this run’s agent and project have. Nothing runs.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 401

Missing / malformed / expired / revoked bearer token.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 404

No test set with that id under this tenant; or `projectId` names no project of this tenant (`project-not-found`).

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 409

`project-suspended`: the project is suspended, so no test run starts in it. Or `suggested-change-nothing-to-check`: an agent’s own test set has no cases in the project (`details.projectId`), as a suggested change’s check answers. Or Idempotency-Key was reused with a different body.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 413

The request body is over the API’s limit (`request-too-large`; `details.maxBytes` says how much).

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 422

No dispatcher registered for the test set kind.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 500

Server error (unmapped domain code or framework crash).

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 501

`suggested-change-check-unsupported`: an agent’s own test set, on a runtime that doesn’t freeze a project’s test set.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

### 503

The `Idempotency-Key` store couldn’t be read before the request ran (`idempotency-store-unavailable`), so nothing was changed; `Retry-After` (whole seconds) says when to try again with the same key.

application/json

object

**error**

required

object

**code**

required

Stable machine-readable discriminant. Values match domain error codes (see `docs/API-ROUTE-CONVENTIONS.md` §4.3).

string

**message**

required

string

**details**

Optional, kind-specific.

object

**_key_**

additional properties

any

**requestId**

required

Server-assigned request id; also echoed via `X-Request-Id` header.

string

##### Examplegenerated

```json
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}
```

## See also

- [`GET /v1/test-sets/{testSetId}`: Fetch a test set (latest version)](https://docs.kindgi.com/v0.4.0/reference/api/operations/testsetsget.md)
- [`GET /v1/test-sets/{testSetId}/versions`: List versions of a test set](https://docs.kindgi.com/v0.4.0/reference/api/operations/testsetsversionslist.md)
- [`GET /v1/test-sets/{testSetId}/versions/{version}`: Fetch a specific test set version](https://docs.kindgi.com/v0.4.0/reference/api/operations/testsetsversionsget.md)
- [`GET /v1/test-sets/{testSetId}/versions/{version}/cases`: List the cases of a corrections test set version](https://docs.kindgi.com/v0.4.0/reference/api/operations/testsetslistcases.md)
- [Kindgi 0.4.0](https://docs.kindgi.com/v0.4.0/releases/0.4.0.md)
- [Error code `bad-input`](https://docs.kindgi.com/v0.4.0/reference/errors.md#bad-input)
- [Error code `idempotency-store-unavailable`](https://docs.kindgi.com/v0.4.0/reference/errors.md#idempotency-store-unavailable)
- [Error code `project-not-found`](https://docs.kindgi.com/v0.4.0/reference/errors.md#project-not-found)
- [Error code `project-suspended`](https://docs.kindgi.com/v0.4.0/reference/errors.md#project-suspended)
- [Error code `reference-out-of-scope`](https://docs.kindgi.com/v0.4.0/reference/errors.md#reference-out-of-scope)
- [Error code `request-too-large`](https://docs.kindgi.com/v0.4.0/reference/errors.md#request-too-large)
- [Error code `suggested-change-check-unsupported`](https://docs.kindgi.com/v0.4.0/reference/errors.md#suggested-change-check-unsupported)
- [Error code `suggested-change-nothing-to-check`](https://docs.kindgi.com/v0.4.0/reference/errors.md#suggested-change-nothing-to-check)
- [Every error code](https://docs.kindgi.com/v0.4.0/reference/errors.md)