Skip to content
Kindgi

List the cases of a judged eval suite version

GET
/v1/eval-suites/{suiteId}/versions/{version}/cases
curl --request GET \
  --url 'https://example.com/v1/eval-suites/example/versions/example/cases?limit=25' \
  --header 'Authorization: Bearer <token>'

Cursor-paginated, in the order the cases were stored (newest judged run first).

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
suiteId
required
string
>= 1 characters

EvalSuiteId — stable string chosen by the caller (e.g. acme.drafting-accuracy).

version
required
string
/^\d+\.\d+\.\d+$/

Semver version string of the eval suite (e.g. 1.2.0).

Query Parameters

Section titled “Query Parameters”
limit
integer
default: 25 >= 1 <= 100

1..100. Default 25.

cursor
string

Where the previous page ended: its nextCursor, as it came. A runtime that seals cursors takes one only for the same list, filters and caller, within a day; otherwise 400 bad-input, and the list starts again without it. Absent → first page.

Responses

Section titled “Responses”

200

Section titled “200”

Page of cases.

Media typeapplication/json
object
data
required
Array<object>

One case of a judged eval suite: a copy of a judged run with its items’ judgments summed up.

object
caseId
required

The judged run’s id.

string
subject
required

What a judged run ran: an agent at a version, or a flow at a version.

object
kind
required
string
Allowed values: agent flow
id
required
string
version
required
string
input
required
context

What a judged run needs besides its input to be replayed, captured when it was first judged. For an agent turn: the conversation before it, what its retrievals returned, and the decision at its session approval gate. For a flow run: its tool calls with their results. For both: the env values its tools were sent.

object
history

The conversation’s messages before the turn, oldest first (at most the last 200).

Array
historyTruncated

Whether older messages were left out of history.

boolean
retrieved

What the turn’s retrievals returned.

recalled

Messages of earlier conversations the turn recalled (intents over conversations), as quoted to the model.

sessionApproval

The reviewer’s decision at the turn’s session approval gate, when the turn waited on one. A replay of the turn follows it.

object
approved
required
boolean
rationale

The reviewer’s reason for a rejection.

string
flow

For a flow run: what it did, kept at its first judgment so it can be replayed. Every tool call it made with its result (at its tool nodes, in its agent steps’ turns and in its sub-flows), at most 500, and its agent steps.

object
calls
required
Array<object>
object
runId
required

The run that made it: the flow run, a sub-flow’s, or an agent step’s turn.

string
nodeId

The tool node that made it, or the agent step whose turn did.

string
scope

The loop iteration, in a loop body.

string
toolId
required
string
arguments
result
steps
required
Array<object>
object
runId
required
string
nodeId
string
scope
string
agentId
required
string
agentVersion
required
string
retrieved

What the step’s turn retrieved.

recalled

Messages of earlier conversations the step’s turn recalled.

truncated

More calls were made than were kept.

boolean
toolEnv

The env values each tool’s calls were sent (needsSpec.env), by tool id: its first call’s, as the run recorded them. A replay sends them to a read-only tool it runs live, so the tool reads the config the run saw, not today’s. Absent for a run from before env was recorded.

object
key
additional properties
object
key
additional properties
string
replayOf

When the judged run is a comparison’s replay: the run it re-ran, stamped at its first judgment. A test set built from judgments leaves replays out.

string
output
required
items
required
Array<object>

The judgments of one item of a case’s output, summed up.

object
key
required
string
pointer
string
rank
integer
yes
required

How many judgments said yes.

integer
no
required

How many judgments said no.

integer
yesWeight
required

The weight behind “yes” (an unclassified judgment counts 1).

number
totalWeight
required

The weight behind all judgments of the item.

number
restricted

The same weights, counting only judgments recorded while their class was restricted (Judgment.restricted), for a comparison weighted restricted-only. Absent from a test set built before restrictions.

object
yesWeight
required
number
totalWeight
required
number
reasons
required

The reasons given, newest first.

Array<object>
object
verdict
required
string
Allowed values: yes no
reason
required
string
judgeClassId

The judgment’s class, when it had one.

string
restricted

Set when the judgment was recorded while its class was restricted (Judgment.restricted).

boolean
erased

An erasure cleared this case (a person’s words were erased): input and output are null, items empty, and eval runs leave it out (counted as erased).

boolean
nextCursor

Opaque cursor. Treat as opaque on the client.

string
hasMore
required
boolean
Example
{
  "data": [
    {
      "subject": {
        "kind": "agent"
      },
      "items": [
        {
          "reasons": [
            {
              "verdict": "yes",
              "restricted": true
            }
          ]
        }
      ],
      "erased": true
    }
  ]
}

401

Section titled “401”

Missing / malformed / expired / revoked bearer token.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

404

Section titled “404”

No eval suite with that id.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}