client.eval_runs

client.eval_runs — the evalRuns operations.

On AsyncKindgi every method is the same, awaited.

client.eval_runs.start()

start(
    suite_id: str | UUID,
    body: StartEvalRunBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> StartEvalRunResult

Signature

Start an eval run against a suite. POST /v1/eval-suites/{suiteId}/runs

Dispatches an eval run against the latest version of the suite. Exactly one of agentRef / flowRef is required — the subject the suite evaluates. Dispatch is per-kind: accuracy runs the subject once per case and grades via a configured judge; other kinds return 422 dispatcher-not-registered unless the deployment registers a dispatcher for them. Idempotency-Key applies.

client.eval_runs.list()

list(
    *,
    limit: int | None = None,
    cursor: str | None = None,
    suite_id: str | UUID | None = None,
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled'] | None = None,
    agent_id: str | UUID | None = None,
    flow_id: str | UUID | None = None,
    from_: str | None = None,
    to: str | None = None,
    scope_kind: Literal['tenant', 'org', 'project'] | None = None,
    scope_id: str | UUID | None = None,
    inherit: bool | None = None,
    timeout: float | None = None,
) -> EvalRunCollectionPage

Signature

List eval runs. GET /v1/eval-runs

Cursor-paginated. Filters: ?suiteId= / ?status= / ?agentId= / ?flowId= / ?from= / ?to=. Sort order: startedAt descending, runId descending as tie-breaker.

client.eval_runs.get()

get(run_id: str | UUID, /, *, timeout: float | None = None) -> EvalRun

Signature

Fetch an eval run. GET /v1/eval-runs/{runId}

client.eval_runs.cancel()

cancel(run_id: str | UUID, /, *, timeout: float | None = None) -> EvalRun

Signature

Cancel an eval run. POST /v1/eval-runs/{runId}/cancel

Best-effort cancel. Returns 409 when the run is already terminal.

client.eval_runs.rescore()

rescore(
    run_id: str | UUID,
    body: RescoreEvalRunBody | Mapping[str, Any] | None = None,
    /,
    *,
    timeout: float | None = None,
    **fields: Any,
) -> StartEvalRunResult

Signature

Rescore a comparison eval run. POST /v1/eval-runs/{runId}/rescore

Starts a new comparison eval run of the same test set version, candidate and settings that replays nothing: it scores the run's replays again, with the judgments recorded on them since (a changed answer judged on the replay itself; changes.new lists the items to judge). The run rescored stays as it was; the new one names it (comparison.rescoreOf, summary.rescoreOf). A case whose replays can't be read again keeps its scores (rescored: false). Needs admin on the run's suite and write on its project.

client.eval_runs.events()

events(
    run_id: str | UUID,
    /,
    *,
    last_event_id: str | UUID | None = None,
    timeout: float | None = None,
) -> None

Signature

Stream eval-run events (SSE). GET /v1/eval-runs/{runId}/events

Server-Sent Events. Per-case progress events (eval-run.case-completed / eval-run.case-failed) followed by a terminal event (eval-run.completed / eval-run.failed / eval-run.cancelled) carrying the aggregate result. Mirrors /v1/runs/:runId/stream structurally.