List the cases of a corrections test set version
const url = 'https://example.com/v1/test-sets/example/versions/example/cases?limit=25';const options = {method: 'GET', headers: {Authorization: 'Bearer <token>'}};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request GET \ --url 'https://example.com/v1/test-sets/example/versions/example/cases?limit=25' \ --header 'Authorization: Bearer <token>'Cursor-paginated, in the order the cases were stored (newest corrected run first).
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”TestSetId — stable string chosen by the caller (e.g. acme.drafting-accuracy).
Semver version string of the test set (e.g. 1.2.0).
Query Parameters
Section titled “Query Parameters”1..100. Default 25.
Where the previous page ended: its nextCursor, as it came. A runtime that seals cursors takes one only for the same list, filters and caller, within a day; otherwise 400 bad-input, and the list starts again without it. Absent → first page.
Responses
Section titled “Responses”Page of cases.
object
One case of a corrections test set: a copy of a corrected run with its items’ corrections summed up.
object
The case’s id. A frozen case of an agent’s test set has its correction’s id, or a case- id when it was added by hand. A corrections test set built before 0.4.0 used the corrected run’s id.
The run the case was copied from. Absent from older servers: there a case of a corrections test set built before 0.4.0 has the run’s id as its caseId.
What a corrected run ran: an agent at a version, or a flow at a version.
object
What a corrected run needs besides its input to be replayed, captured when it was first corrected. For an agent turn: the conversation before it, what its retrievals returned, and the decision at its session approval gate. For a flow run: its tool calls with their results. For both: the env values its tools were sent.
object
The conversation’s messages before the turn, oldest first (at most the last 200).
Whether older messages were left out of history.
What the turn’s retrievals returned.
Messages of earlier conversations the turn recalled (intents over conversations), as quoted to the model.
The reviewer’s decision at the turn’s session approval gate, when the turn waited on one. A replay of the turn follows it.
object
The reviewer’s reason for a rejection.
For a flow run: what it did, kept at its first correction so it can be replayed. Every tool call it made with its result (at its tool nodes, in its agent steps’ turns and in its sub-flows), at most 500, and its agent steps.
object
object
The run that made it: the flow run, a sub-flow’s, or an agent step’s turn.
The tool node that made it, or the agent step whose turn did.
The loop iteration, in a loop body.
An agent step’s turn kept the result as outside data: another system’s answer to a paused call, or a result its tool says is outside data. A replay hands it to the model as data, as the turn did. Absent for a tool node’s call, and for a run corrected before the mark was kept.
object
What the step’s turn retrieved.
Messages of earlier conversations the step’s turn recalled.
More calls were made than were kept.
The env values each tool’s calls were sent (needsSpec.env), by tool id: its first call’s, as the run recorded them. A replay sends them to a read-only tool it runs live, so the tool reads the config the run saw, not today’s. Absent for a run from before env was recorded.
object
object
When the corrected run is a comparison’s replay: the run it re-ran, stamped at its first correction. A test set built from corrections leaves replays out.
The segment path its run was started with, coarse to fine (empty: none). Set on a frozen case of an agent’s test set when its run’s copy has it.
object
Lowercase, like an identifier: company, contact-role.
The corrections of one item of a case’s output, summed up.
object
How many corrections said yes.
How many corrections said no.
The weight behind “yes” (an unclassified correction counts 1).
The weight behind all corrections of the item.
The same weights, counting only corrections recorded while their group was restricted (Correction.expert), for a comparison weighted experts-only. Absent from a test set built before restrictions.
object
The corrections behind these two. Absent from a test set built before correction ids.
The corrections summed into the item, by id: what a comparison on this case counts. Absent from a test set built before correction ids.
The reasons given, newest first.
object
The correction’s group, when it had one.
Set when the correction was recorded while its group was restricted (Correction.expert).
Experts’ expected decisions carried onto this case from its corrections, each graded on the replay. Empty or absent when none. experts-only keeps only the entries whose correction was restricted.
object
The correction they were recorded on.
The correction’s group, when it had one.
Set when the correction was recorded while its group was restricted (Correction.expert).
The calls the agent should have made.
object
The partial set of top-level argument fields to compare, by value. Fields left out are not compared; an empty object asks only that the tool was called. At most 4000 characters in canonical JSON.
object
Tools that should not have been called at all.
What the case expects of a run, as the agent’s test set lists it. Set on a frozen case of an agent’s test set, from its own correction or hand case, which also fill items and corrections. A grader reads it when it’s present, and items and corrections otherwise.
object
right: it calls the same tools the corrected run called, by name, in any order (args aren’t compared). wrong: it calls every shouldCall tool (with its args where given) and none of mustNotCall, checked exactly, without a model.
object
object
The partial set of top-level argument fields to compare, by value. Fields left out are not compared; an empty object asks only that the tool was called. At most 4000 characters in canonical JSON.
object
A judging model checks it. right: it says what the corrected run’s answer said. wrong: it does what shouldSay or why asks.
object
A ranked answer’s items, scored as weightedPrecisionAtK.
object
The caller’s stable id for the item, such as a matched case’s id, never the item’s own text.
Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2.
The item’s position in the ranked list (0 = first).
Present (true) on a frozen case whose test case is an expert’s: a test run weighted experts-only grades its expect, and drops the expect of every other case. Absent from older servers.
An erasure cleared this case (a person’s words were erased): input and output are null, items empty, corrections, expect and segments gone, and test runs leave it out (counted as erased).
Opaque cursor. Treat as opaque on the client.
Example
{ "data": [ { "subject": { "kind": "agent" }, "context": { "flow": { "calls": [ { "fromEvent": true } ] } }, "items": [ { "reasons": [ { "verdict": "yes", "expert": true } ] } ], "corrections": [ { "expert": true } ], "expect": { "decisions": { "verdict": "right" }, "answer": { "verdict": "right" }, "items": [ { "verdict": "right" } ] }, "expert": true, "erased": true } ]}Missing / malformed / expired / revoked bearer token.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}No test set with that id.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The database was too busy to answer in time (database-busy); Retry-After (whole seconds) says when to try again.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}See also
GET /v1/test-sets/{testSetId}: Fetch a test set (latest version)GET /v1/test-sets/{testSetId}/versions: List versions of a test setGET /v1/test-sets/{testSetId}/versions/{version}: Fetch a specific test set versionPOST /v1/test-sets/{testSetId}/versions/{version}/unregister: Unregister a test set version- An agent's test set
- Error code
database-busy - Every error code