Start a test run against a test set
const url = 'https://example.com/v1/test-sets/example/runs';const options = { method: 'POST', headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'}, body: '{"projectId":"2489E9AD-2EE2-8E00-8EC9-32D5F69181C0","agentRef":{"agentId":"example","version":"example"},"flowRef":{"flowId":"example","version":"example"},"dryRun":true,"correlationId":"example","baseline":"recorded","reads":"recorded","repetitions":1,"k":1,"versions":{"tools":{"additionalProperty":"example"},"agents":{"additionalProperty":"example"}},"groupWeights":"as-recorded","overrides":{"settings":{"additionalProperty":{}},"prompts":{"additionalProperty":{"template":"example"}}},"sample":{"part":"search","seed":"example","holdOutShare":1}}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://example.com/v1/test-sets/example/runs \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "projectId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "agentRef": { "agentId": "example", "version": "example" }, "flowRef": { "flowId": "example", "version": "example" }, "dryRun": true, "correlationId": "example", "baseline": "recorded", "reads": "recorded", "repetitions": 1, "k": 1, "versions": { "tools": { "additionalProperty": "example" }, "agents": { "additionalProperty": "example" } }, "groupWeights": "as-recorded", "overrides": { "settings": { "additionalProperty": {} }, "prompts": { "additionalProperty": { "template": "example" } } }, "sample": { "part": "search", "seed": "example", "holdOutShare": 1 } }'Dispatches a test run against the latest version of the test set. An agent’s own test set (agent.<agentId>.<projectId>, read at GET /v1/agents/{agentId}/test-set) is frozen first, as a suggested change’s check freezes it, and the run records the frozen version as testSetVersion; a dry run freezes it too; it runs only its agent, in its project. Exactly one of agentRef / flowRef is required — the subject the test set evaluates. Dispatch is per-kind: accuracy runs the subject once per case and grades via a configured judge; other kinds return 422 dispatcher-not-registered unless the deployment registers a dispatcher for them. Nothing starts in a suspended project (409 project-suspended), and a run going when its project or workspace stops taking new runs ends cancelled at the next case, with the reason in cancelReason. Idempotency-Key applies.
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”TestSetId — stable string chosen by the caller (e.g. acme.drafting-accuracy).
Header Parameters
Section titled “Header Parameters”Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the workspace is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5). A request whose key store can’t be read before it runs is answered 503 idempotency-store-unavailable with Retry-After: nothing ran, so retry with the same key (from 0.3.0).
Request Bodyrequired
Section titled “Request Bodyrequired”Exactly one of agentRef or flowRef MUST be supplied. dryRun: true returns a plan preview without invoking the subject. For a corrections test set (a test set), the run is a comparison: agentRef or flowRef with its version is the candidate, replayed on each case without doing anything the past run didn’t (a flow stops at a write the replay refuses); baseline (default 'recorded'), reads (default recorded), repetitions (default 1) and k (default 10) set how. With flowRef, versions runs the flow with some of its agents or tools at other versions; an id the flow doesn’t use, or a version that isn’t published, is refused (400 validation-failed, each under details.issues).
object
Project this belongs to (its content scope). Required: missing, or not a project in the caller’s workspace → 400 bad-input.
object
object
An id of your own, kept on the run. It may not start with pin: or latest: (bad-input): those are the test runs Kindgi starts itself, for a pin change or Latest moving.
object
object
object
A segment path in projectId, coarse to fine, as pinned versions resolve it. Needs projectId.
object
Lowercase, like an identifier: company, contact-role.
Agents and tools a flow runs at other exact versions than the flow version’s pins (“this flow, with acme.scorer at 0.4.0”), without publishing a new flow version: a comparison’s flow candidate (versions on the start body and the run’s comparison), and the flow runs that replay it (a run’s versions). Each id must be an agent or tool the flow uses.
object
Tool id → exact version.
object
Agent id → exact version, for agent nodes with or without a version of their own.
object
Which corrections count: each at its group’s weight (as-recorded, the default), or only those recorded while their group was restricted (Correction.expert), the others weighing 0 (experts-only).
For an agent candidate: block content its replays run instead of the version’s pinned content (an Improve run’s search). settings: values by settings block id, each a block the version pins, satisfying its schema. prompts: a template for the prompt block the version pins, which reads and names only what the agent has (its parameters, the variables the current template reads, the settings blocks it pins, its tools’ and blocks’ ids) and is at most twice as long. Anything else is 400 validation-failed. A comparison with overrides can’t gate a pin.
object
object
object
object
object
Only part of the test set’s cases: split once into a hold-out part (about holdOutShare of them) and a search part (the rest), stratified (the cases that carry a correction, then of the others the cases with a “no” and the rest, each split on its own, a stratum of two or more giving each part at least one, so both parts check a correction when two cases carry one), in the order of a hash of each case id and seed. The same seed always splits the same test set the same way. A release gate refuses a comparison on the search part (comparison.sample).
object
Responses
Section titled “Responses”Test run started.
object
object
Examplegenerated
{ "testRunId": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "dryRunPreview": {}}Malformed body (both / neither of agentRef / flowRef, dispatcher-input-invalid); or projectId isn’t a project id (a UUID). Or reference-out-of-scope: the test set, the agent or flow, or an agent or tool a flow candidate’s versions names, isn’t registered in projectId’s project, its org or the workspace; or a version named (by agentRef, flowRef or versions.agents) isn’t one projectId’s project runs (details.issues, each at the path it was named, whose message names the version). Or bad-input for an agent’s own test set run with another agent, in another project, or with flowRef: the message names the test set this run’s agent and project have. Nothing runs.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Missing / malformed / expired / revoked bearer token.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}No test set with that id under this workspace; or projectId names no project of this workspace (project-not-found).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}project-suspended: the project is suspended, so no test run starts in it. Or suggested-change-nothing-to-check: an agent’s own test set has no cases in the project (details.projectId), as a suggested change’s check answers. Or Idempotency-Key was reused with a different body.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The request body is over the API’s limit (request-too-large; details.maxBytes says how much).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}No dispatcher registered for the test set kind.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Server error (unmapped domain code or framework crash).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}suggested-change-check-unsupported: an agent’s own test set, on a runtime that doesn’t freeze a project’s test set.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The Idempotency-Key store couldn’t be read before the request ran (idempotency-store-unavailable), so nothing was changed; Retry-After (whole seconds) says when to try again with the same key.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}See also
GET /v1/test-sets/{testSetId}: Fetch a test set (latest version)GET /v1/test-sets/{testSetId}/versions: List versions of a test setGET /v1/test-sets/{testSetId}/versions/{version}: Fetch a specific test set versionGET /v1/test-sets/{testSetId}/versions/{version}/cases: List the cases of a corrections test set version- Kindgi 0.4.0
- Error code
bad-input - Error code
idempotency-store-unavailable - Error code
project-not-found - Error code
project-suspended - Error code
reference-out-of-scope - Error code
request-too-large - Error code
suggested-change-check-unsupported - Error code
suggested-change-nothing-to-check - Every error code