Add a case to an agent's test set
const url = 'https://example.com/v1/agents/example/test-set/cases';const options = { method: 'POST', headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'}, body: '{"runId":"example","decisions":"right","shouldCall":[{"tool":"example","args":{}}],"mustNotCall":["example"],"answer":"right","shouldSay":"example","why":"example","items":[{"key":"example","pointer":"example","rank":1,"verdict":"right"}]}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://example.com/v1/agents/example/test-set/cases \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "runId": "example", "decisions": "right", "shouldCall": [ { "tool": "example", "args": {} } ], "mustNotCall": [ "example" ], "answer": "right", "shouldSay": "example", "why": "example", "items": [ { "key": "example", "pointer": "example", "rank": 1, "verdict": "right" } ] }'A case by hand, from a correction’s body on one of the agent’s finished runs. It has no status, isn’t sorted, and nothing that counts corrections counts it (Improve’s trigger, the expert queue’s progress, a rule’s yes-rate). The run’s first correction or case stores a copy of its input and output. A comparison’s replay can’t be a case. Needs publish on the agent and read on the case’s project; for an agent more than one project serves, write on the case’s project.
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”AgentId — dotted namespace (acme.drafting).
Header Parameters
Section titled “Header Parameters”Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5). A request whose key store can’t be read before it runs is answered 503 idempotency-store-unavailable with Retry-After: nothing ran, so retry with the same key (from 0.3.0).
Request Bodyrequired
Section titled “Request Bodyrequired”A correction’s body on one of the agent’s runs. At least one of decisions, answer and items. decisions: "wrong" needs an entry in shouldCall or mustNotCall; answer: "wrong" needs shouldSay or why.
object
Only with decisions: "wrong": the calls the agent should have made.
object
Only when the inputs matter: the top-level argument fields to compare, by value. Left out, only the call is checked. At most 4000 characters in canonical JSON.
object
Only with decisions: "wrong": tools that shouldn’t have been called. A tool can’t be in both.
Only with answer: "wrong": what it should have said.
For an answer that is a ranked list: each item, and whether it was right. Each key once.
object
The caller’s stable id for the item, such as a matched case’s id, never the item’s own text.
Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2.
The item’s position in the ranked list (0 = first).
Responses
Section titled “Responses”Case added.
object
A correction’s case has the correction’s id; a hand case’s starts case-.
correction: a correction of the run. hand: added by hand.
The run’s project.
The agent version the run ran.
One part for each part its correction gave a verdict on; at least one.
object
right: it calls the same tools the corrected run called, by name, in any order (args aren’t compared). wrong: it calls every shouldCall tool (with its args where given) and none of mustNotCall, checked exactly, without a model.
object
object
The partial set of top-level argument fields to compare, by value. Fields left out are not compared; an empty object asks only that the tool was called. At most 4000 characters in canonical JSON.
object
A judging model checks it. right: it says what the corrected run’s answer said. wrong: it does what shouldSay or why asks.
object
A ranked answer’s items, scored as weightedPrecisionAtK.
object
The caller’s stable id for the item, such as a matched case’s id, never the item’s own text.
Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2.
The item’s position in the ranked list (0 = first).
Who made the correction, or added the case.
object
A user id, or for a service token its token or session id.
When the case last changed: its addedAt until it’s retired.
Present on a retired case: it stays listed, and no test run checks it.
object
Who asserted a correction: the authenticated caller, never a typed name.
object
A user id, or for a service token its token or session id.
Present (true) on an expert’s case: a correction’s case when the correction was recorded while its group was restricted, and every hand case, since adding one needs publish on the agent, or write on its project when more than one project serves the agent. A test run weighted experts-only grades only these. Absent from older servers.
Example
{ "kind": "correction", "expect": { "decisions": { "verdict": "right" }, "answer": { "verdict": "right" }, "items": [ { "verdict": "right" } ] }, "addedBy": { "kind": "user" }, "retired": { "by": { "kind": "user" } }, "expert": true}Malformed body; the run isn’t one of the agent’s, or is a comparison’s replay; or item-not-found (an item’s pointer resolves to nothing in the output).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Missing / malformed / expired / revoked bearer token.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}run-not-found.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}run-not-finished: the run has no output yet.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}run-erased: an erasure cleared the run’s content.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The request body is over the API’s limit (request-too-large; details.maxBytes says how much).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Server error (unmapped domain code or framework crash).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The Idempotency-Key store couldn’t be read before the request ran (idempotency-store-unavailable), so nothing was changed; Retry-After (whole seconds) says when to try again with the same key.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}See also
GET /v1/agents/{agentId}/test-set/cases: List a project's cases of an agent's test setGET /v1/agents/{agentId}: Fetch the highest version of an agentGET /v1/agents/{agentId}/projects: List the projects an agent runs in, and what runs in eachGET /v1/agents/{agentId}/versions: List versions of an agent- Error code
idempotency-store-unavailable - Error code
item-not-found - Error code
request-too-large - Error code
run-erased - Error code
run-not-finished - Error code
run-not-found - Every error code