Judge an item of a run's output
const url = 'https://example.com/v1/judgments';const options = { method: 'POST', headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'}, body: '{"runId":"example","item":{"key":"example","pointer":"example","rank":1},"verdict":"yes","reason":"example","judgeClassId":"example","participantId":"example","correction":{"calls":[{"tool":"example","args":{}}],"notCalled":["example"]}}'};
try { const response = await fetch(url, options); const data = await response.json(); console.log(data);} catch (error) { console.error(error);}curl --request POST \ --url https://example.com/v1/judgments \ --header 'Authorization: Bearer <token>' \ --header 'Content-Type: application/json' \ --data '{ "runId": "example", "item": { "key": "example", "pointer": "example", "rank": 1 }, "verdict": "yes", "reason": "example", "judgeClassId": "example", "participantId": "example", "correction": { "calls": [ { "tool": "example", "args": {} } ], "notCalled": [ "example" ] } }'Records yes or no, with an optional reason, about one item of a finished run’s output, optionally under a judge class that applies to the run’s project or agent (unclassified judgments count with weight 1). item.pointer (a JSON Pointer) must resolve in the run’s output; its value is kept as itemValue. The first judgment of a run also stores a copy of the run’s input and output. assertedBy is the authenticated caller, never the body. Judging again as the same caller for the same run, item key and participantId supersedes the earlier judgment. Needs judge on the run.
Authorizations
Section titled “Authorizations”Parameters
Section titled “Parameters”Header Parameters
Section titled “Header Parameters”Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5). A request whose key store can’t be read before it runs is answered 503 idempotency-store-unavailable with Retry-After: nothing ran, so retry with the same key (from 0.3.0).
Request Bodyrequired
Section titled “Request Bodyrequired”object
The judged item of a run’s output.
object
The caller’s stable id for the item. kindgi:correction is reserved: a judgment on it is accepted only when it carries a correction at pointer "".
Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2. "" is the whole output.
The item’s position in a ranked list (0 = first).
Optional. When given it must exist and apply to the run.
An app’s opaque id for its end user, when judging on their behalf.
Optional. The tool calls the agent should have made, graded on a replay. A correction judges the run’s whole output, so it is recorded under one reserved item (item.key "kindgi:correction", item.pointer ""): a correction on any other item is a 400, and a second correction of the run by the same expert replaces the first, so it counts once.
object
The calls the agent should have made.
object
The partial set of top-level argument fields to compare, by value. Fields left out are not compared; an empty object asks only that the tool was called. At most 4000 characters in canonical JSON.
object
Tools that should not have been called at all.
Responses
Section titled “Responses”Judgment recorded.
object
What a judged run ran: an agent at a version, or a flow at a version.
object
The judged item of a run’s output.
object
The caller’s stable id for the item. kindgi:correction is reserved: a judgment on it is accepted only when it carries a correction at pointer "".
Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2. "" is the whole output.
The item’s position in a ranked list (0 = first).
The expert’s correction of the run’s decisions, when they gave one.
object
The calls the agent should have made.
object
The partial set of top-level argument fields to compare, by value. Fields left out are not compared; an empty object asks only that the tool was called. At most 4000 characters in canonical JSON.
object
Tools that should not have been called at all.
The judge class the judgment is recorded under. Absent when unclassified (counts with weight 1).
Present (true) when the judge class was restricted (assertableBy) when the judgment was recorded, so the judge was checked against it. A restriction added or lifted later doesn’t change it.
Who asserted a judgment: the authenticated caller, never a typed name.
object
A user id, or for a service token its token or session id.
The app’s opaque id for its end user who judged, when an app judged on their behalf.
Set when the judgment was removed or superseded.
The judgment that replaced this one.
Example
{ "subject": { "kind": "agent" }, "verdict": "yes", "restricted": true, "assertedBy": { "kind": "user" }}Malformed body, or item-not-found (the pointer resolves to nothing in the output), or judge-class-not-applicable.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Missing / malformed / expired / revoked bearer token.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}permission-denied: not allowed to judge this run. judge-class-not-allowed: the judge class is restricted (assertableBy) and the caller may not assert it.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}run-not-found.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}run-not-finished: the run has no output to judge yet.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}run-erased: an erasure cleared the run’s content (a person’s words were removed); there’s nothing to judge.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The request body is over the API’s limit (request-too-large; details.maxBytes says how much).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}Server error (unmapped domain code or framework crash).
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}The Idempotency-Key store couldn’t be read before the request ran (idempotency-store-unavailable), so nothing was changed; Retry-After (whole seconds) says when to try again with the same key.
object
object
Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).
Optional, kind-specific.
object
Server-assigned request id; also echoed via X-Request-Id header.
Examplegenerated
{ "error": { "code": "example", "message": "example", "details": {}, "requestId": "example" }}See also
GET /v1/judgments: List judgments- Error code
idempotency-store-unavailable - Error code
item-not-found - Error code
judge-class-not-allowed - Error code
judge-class-not-applicable - Error code
permission-denied - Error code
request-too-large - Error code
run-erased - Error code
run-not-finished - Error code
run-not-found - Every error code