Skip to content
Kindgi

List eval runs

View as Markdown
GET
/v1/eval-runs
curl --request GET \
  --url 'https://example.com/v1/eval-runs?limit=25&status=pending&scopeKind=tenant&inherit=true' \
  --header 'Authorization: Bearer <token>'

Cursor-paginated. Filters: ?suiteId= / ?status= / ?agentId= / ?flowId= / ?from= / ?to=. Sort order: startedAt descending, runId descending as tie-breaker.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Query Parameters

Section titled “Query Parameters”
limit
integer
default: 25 >= 1 <= 100

1..100. Default 25.

cursor
string

Where the previous page ended: its nextCursor, as it came. A runtime that seals cursors takes one only for the same list, filters and caller, within a day; otherwise 400 bad-input, and the list starts again without it. Absent → first page.

suiteId
string

Filter to runs against a specific suite id.

status
string
Allowed values: pending running completed failed cancelled

Filter to a single EvalRunStatus.

agentId
string

Filter to runs targeting a specific agent id.

flowId
string

Filter to runs targeting a specific flow id.

from
string format: date-time

ISO-8601 lower bound on startedAt (inclusive).

to
string format: date-time

ISO-8601 upper bound on startedAt (inclusive).

scopeKind

Discriminator for the ?scopeKind + ?scopeId + ?inherit triplet. Tenant carries no id (implicit from session); org/project require scopeId.

string
Allowed values: tenant org project

Optional scope discriminator. If absent, no scope filter is applied.

scopeId
string
>= 1 characters

Required IF scopeKind is org or project. MUST be absent if scopeKind=tenant (tenant is implicit from the session). Malformed combinations return 400 scope-invalid.

inherit
boolean
default: true

Default true. false = literal-at-this-scope only (admin/audit view). Load-bearing for policy/config-scoped resources (mcp-endpoints); documented no-op for content-scoped resources (agents/flows/tools/…).

Responses

Section titled “Responses”

200

Section titled “200”

Page of eval runs.

Media typeapplication/json
object
data
required
Array<object>
object
runId
required
string format: uuid
tenantId
required
string format: uuid
projectId

The project the eval run is in: the one it was started in. Absent on a runtime before Kindgi 0.1.6.

string format: uuid
suiteId
required
string
suiteVersion
required
string
/^\d+\.\d+\.\d+$/
kind
required
string
Allowed values: accuracy pairwise regression human-review benchmark custom judged
agentRef
object
agentId
required
string
version
string
/^\d+\.\d+\.\d+$/
flowRef
object
flowId
required
string
version
string
/^\d+\.\d+\.\d+$/
status
required
string
Allowed values: pending running completed failed cancelled
dryRun
required
boolean
startedAt
required
string format: date-time
completedAt
string format: date-time
result

Kind-specific opaque JSON. For accuracy, contains { passCount, totalCount, meanScore, perCase[] }. For judged (a comparison), { summary, perCase[] }: the summary has the baseline (the versions behind the recorded runs) and the candidate ({ kind: "agent", agentId, version } or { kind: "flow", flowId, version, versions? }), the case counts (cases, diverged, refusedWrites, errors, and stopped: flow cases that stopped at a write the replay refused, left out of the metrics), the models that answered, and metrics (weightedYesShare, judgedCoverage, weightedPrecisionAtK, each { baseline, candidate, delta, n, weight, baselineN, baselineWeight, direction, k?, spread? }); each case has its replay runs, the scores, the items kept, dropped and new, the tool calls with what happened to each, and stopped (what it would have done) when it stopped. Other kinds define their own shapes as their dispatchers ship.

object
key
additional properties
any
error
string
cancelReason

Why a cancelled run stopped when nobody cancelled it: its project or tenant took no new runs when a case started (project-suspended, tenant-suspended, tenant-closing). The cases before it are in result. Absent when it was cancelled, and from a runtime that doesn’t say. A plain string: a later reason doesn’t break a client.

string
correlationId
string
comparison

A comparison eval run’s settings (a judged suite).

object
baseline
required
One of:
string
Allowed values: recorded
reads
required

Whether replayed reads use the past run’s results when it has them (recorded), or run live.

string
Allowed values: recorded live
repetitions
required
integer
>= 1 <= 10
k
required
integer
>= 1 <= 100
versions

Agents and tools a flow runs at other exact versions than the flow version’s pins (“this flow, with acme.scorer at 0.4.0”), without publishing a new flow version: a comparison’s flow candidate (versions on the start body and the run’s comparison), and the flow runs that replay it (a run’s versions). Each id must be an agent or tool the flow uses.

object
tools

Tool id → exact version.

object
key
additional properties
string
agents

Agent id → exact version, for agent nodes with or without a version of their own.

object
key
additional properties
string
classWeights

Which judgments count: each at its class’s weight (as-recorded, the default), or only those recorded while their class was restricted (Judgment.restricted), the others weighing 0 (restricted-only).

string
Allowed values: as-recorded restricted-only
overrides

For an agent candidate: block content its replays run instead of the version’s pinned content (an improvement pass’s search). settings: values by settings block id, each a block the version pins, satisfying its schema. prompts: a template for the prompt block the version pins, which reads and names only what the agent has (its parameters, the variables the current template reads, the settings blocks it pins, its tools’ and blocks’ ids) and is at most twice as long. Anything else is 400 validation-failed. A comparison with overrides can’t gate a promotion.

object
settings
object
<= 20 properties
key
additional properties
object
key
additional properties
any
prompts
object
<= 1 properties
key
additional properties
object
template
required
string
sample

Only part of the test set’s cases: split once into a hold-out part (about holdOutShare of them) and a search part (the rest), stratified by judgment (the cases with a “no” and the others are split on their own, a stratum of two or more giving each part at least one), in the order of a hash of each case id and seed. The same seed always splits the same test set the same way. A promotion gate refuses a comparison on the search part (comparison.sample).

object
part
required
string
Allowed values: search hold-out
seed
required
string
>= 1 characters <= 200 characters
holdOutShare
required
number
>= 0.1 <= 0.9
rescoreOf

Rescore that comparison eval run instead of replaying: its replays’ outputs are scored again, with the judgments recorded on them since (a changed answer judged on the replay itself). Nothing runs, and the run rescored stays as it was. Set by POST /v1/eval-runs/{runId}/rescore.

string
nextCursor
string
hasMore
required
boolean
Example
{
  "data": [
    {
      "kind": "accuracy",
      "status": "pending",
      "comparison": {
        "baseline": "recorded",
        "reads": "recorded",
        "classWeights": "as-recorded",
        "sample": {
          "part": "search"
        }
      }
    }
  ]
}

400

Section titled “400”

Malformed query parameter (unknown status).

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

401

Section titled “401”

Missing / malformed / expired / revoked bearer token.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

See also