Build a test set from judgments
const url = 'https://example.com/v1/eval-suites/example/versions/from-judgments';const options = { method: 'POST', headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'}, body: '{"version":"example","projectId":"example","agentId":"example","agentVersion":"example","flowId":"example","since":"2026-04-15T12:00:00Z","until":"2026-04-15T12:00:00Z","judgeClassIds":["example"],"minJudgments":1,"segments":[{"key":"example","value":"example"}],"description":"example"}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://example.com/v1/eval-suites/example/versions/from-judgments \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "version": "example", "projectId": "example", "agentId": "example", "agentVersion": "example", "flowId": "example", "since": "2026-04-15T12:00:00Z", "until": "2026-04-15T12:00:00Z", "judgeClassIds": [ "example" ], "minJudgments": 1, "segments": [ { "key": "example", "value": "example" } ], "description": "example" }'Publishes a judged eval suite version whose cases are copies of judged runs of one agent (optionally one version) or flow, newest first, at most 1000. Each case holds the run’s input, what the turn read (context), the judged output, and each item’s judgments summed up: yes and no counts, the weight behind yes and behind all judgments (an unclassified judgment counts 1), and the reasons. judgeClassIds counts only judgments of those classes; minJudgments leaves out runs with fewer; segments keeps only runs started in that segment path or below it. Needs admin on the project.
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”EvalSuiteId — stable string chosen by the caller (e.g. acme.drafting-accuracy).
Header Parameters
Section titled “Header Parameters”Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5).
Request Bodyrequired
Section titled “Request Bodyrequired”Name the agent (agentId) or the flow (flowId) whose judged runs to use.
object
Semver version to publish, e.g. 1.0.0.
Needs agentId.
Runs first judged at or after this time.
Runs first judged before this time.
Count only judgments recorded under these judge classes.
Leave out runs with fewer counted judgments. Default 1.
Only runs started in this segment path or below it, coarse to fine (e.g. company=acme). A run judged before its segments were recorded with its judgments is in no segment, so it’s left out.
object
Lowercase, like an identifier: company, contact-role.
Examplegenerated
{ "version": "example", "projectId": "example", "agentId": "example", "agentVersion": "example", "flowId": "example", "since": "2026-04-15T12:00:00Z", "until": "2026-04-15T12:00:00Z", "judgeClassIds": [ "example" ], "minJudgments": 1, "segments": [ { "key": "example", "value": "example" } ], "description": "example"}Responses
Section titled “Responses”Version published.
object
Whether more judged runs matched than the 1000 cases a set holds.
Example
{ "kind": "judged"}Malformed body; or projectId isn’t a project id (a UUID).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Missing / malformed / expired / revoked bearer token.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}permission-denied.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The body’s projectId names no project of this tenant (project-not-found).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}eval-suite-already-registered: that (id, version) is taken. Or eval-suite-project-mismatch: the suite belongs to another project than the body’s projectId (suites never move).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Server error (unmapped domain code or framework crash).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}test-sets-not-supported: this deployment cannot build test sets.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}