Skip to content
Kindgi

Judge an item of a run's output

POST
/v1/judgments
curl --request POST \
  --url https://example.com/v1/judgments \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "runId": "example", "item": { "key": "example", "pointer": "example", "rank": 1 }, "verdict": "yes", "reason": "example", "judgeClassId": "example", "participantId": "example" }'

Records yes or no, with an optional reason, about one item of a finished run’s output, optionally under a judge class that applies to the run’s project or agent (unclassified judgments count with weight 1). item.pointer (a JSON Pointer) must resolve in the run’s output; its value is kept as itemValue. The first judgment of a run also stores a copy of the run’s input and output. assertedBy is the authenticated caller, never the body. Judging again as the same caller for the same run, item key and participantId supersedes the earlier judgment. Needs judge on the run.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Header Parameters

Section titled “Header Parameters”
Idempotency-Key
string
>= 1 characters

Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5).

Request Bodyrequired

Section titled “Request Bodyrequired”
Media typeapplication/json
object
runId
required
string
>= 1 characters
item
required

The judged item of a run’s output.

object
key
required

The caller’s stable id for the item.

string
>= 1 characters
pointer

Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2. "" is the whole output.

string
rank

The item’s position in a ranked list (0 = first).

integer
verdict
required
string
Allowed values: yes no
reason
string
<= 4000 characters
judgeClassId

Optional. When given it must exist and apply to the run.

string
>= 1 characters
participantId

An app’s opaque id for its end user, when judging on their behalf.

string
>= 1 characters

Responses

Section titled “Responses”

201

Section titled “201”

Judgment recorded.

Media typeapplication/json
object
id
required
string
tenantId
required
string
projectId
required
string
runId
required
string
subject
required

What a judged run ran: an agent at a version, or a flow at a version.

object
kind
required
string
Allowed values: agent flow
id
required
string
version
required
string
item
required

The judged item of a run’s output.

object
key
required

The caller’s stable id for the item.

string
>= 1 characters
pointer

Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2. "" is the whole output.

string
rank

The item’s position in a ranked list (0 = first).

integer
verdict
required
string
Allowed values: yes no
reason
string
judgeClassId

The judge class the judgment is recorded under. Absent when unclassified (counts with weight 1).

string
restricted

Present (true) when the judge class was restricted (assertableBy) when the judgment was recorded, so the judge was checked against it. A restriction added or lifted later doesn’t change it.

boolean
assertedBy
required

Who asserted a judgment: the authenticated caller, never a typed name.

object
kind
required
string
Allowed values: user service
id
required

A user id, or for a service token its token or session id.

string
participantId

The app’s opaque id for its end user who judged, when an app judged on their behalf.

string
createdAt
required
string format: date-time
unregisteredAt

Set when the judgment was removed or superseded.

string format: date-time
supersededBy

The judgment that replaced this one.

string
Example
{
  "subject": {
    "kind": "agent"
  },
  "verdict": "yes",
  "restricted": true,
  "assertedBy": {
    "kind": "user"
  }
}

400

Section titled “400”

Malformed body, or item-not-found (the pointer resolves to nothing in the output), or judge-class-not-applicable.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

401

Section titled “401”

Missing / malformed / expired / revoked bearer token.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

403

Section titled “403”

permission-denied: not allowed to judge this run. judge-class-not-allowed: the judge class is restricted (assertableBy) and the caller may not assert it.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

404

Section titled “404”

run-not-found.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

409

Section titled “409”

run-not-finished: the run has no output to judge yet.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

410

Section titled “410”

run-erased: an erasure cleared the run’s content (a person’s words were removed); there’s nothing to judge.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

500

Section titled “500”

Server error (unmapped domain code or framework crash).

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}