client.test_runs

client.test_runs — the testRuns operations.

On AsyncKindgi every method is the same, awaited.

client.test_runs.start()

start(
    test_set_id: str | UUID,
    body: StartTestRunBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> StartTestRunResult

Signature

Start a test run against a test set. POST /v1/test-sets/{testSetId}/runs

Dispatches a test run against the latest version of the test set. An agent's own test set (agent.<agentId>.<projectId>, read at GET /v1/agents/{agentId}/test-set) is frozen first, as a suggested change's check freezes it, and the run records the frozen version as testSetVersion; a dry run freezes it too; it runs only its agent, in its project. Exactly one of agentRef / flowRef is required — the subject the test set evaluates. Dispatch is per-kind: accuracy runs the subject once per case and grades via a configured judge; other kinds return 422 dispatcher-not-registered unless the deployment registers a dispatcher for them. Nothing starts in a suspended project (409 project-suspended), and a run going when its project or tenant stops taking new runs ends cancelled at the next case, with the reason in cancelReason. Idempotency-Key applies.

client.test_runs.list()

list(
    *,
    limit: int | None = None,
    cursor: str | None = None,
    test_set_id: str | UUID | None = None,
    status: Literal['pending', 'running', 'completed', 'failed', 'cancelled'] | None = None,
    agent_id: str | UUID | None = None,
    flow_id: str | UUID | None = None,
    from_: str | None = None,
    to: str | None = None,
    scope_kind: Literal['tenant', 'org', 'project'] | None = None,
    scope_id: str | UUID | None = None,
    inherit: bool | None = None,
    timeout: float | None = None,
) -> TestRunCollectionPage

Signature

List test runs. GET /v1/test-runs

Cursor-paginated. Filters: ?testSetId= / ?status= / ?agentId= / ?flowId= / ?from= / ?to=. Sort order: startedAt descending, runId descending as tie-breaker.

client.test_runs.get()

get(test_run_id: str | UUID, /, *, timeout: float | None = None) -> TestRun

Signature

Fetch a test run. GET /v1/test-runs/{testRunId}

client.test_runs.cancel()

cancel(test_run_id: str | UUID, /, *, timeout: float | None = None) -> TestRun

Signature

Cancel a test run. POST /v1/test-runs/{testRunId}/cancel

Best-effort cancel. Returns 409 when the run is already terminal.

client.test_runs.rescore()

rescore(
    test_run_id: str | UUID,
    body: RescoreTestRunBody | Mapping[str, Any] | None = None,
    /,
    *,
    timeout: float | None = None,
    **fields: Any,
) -> StartTestRunResult

Signature

Rescore a comparison test run. POST /v1/test-runs/{testRunId}/rescore

Starts a new comparison test run of the same test set version, candidate and settings that replays nothing: it scores the run's replays again, with the corrections recorded on them since (a changed answer corrected on the replay itself; changes.new lists the items to correct). The run rescored stays as it was; the new one names it (comparison.rescoreOf, summary.rescoreOf). A case whose replays can't be read again keeps its scores (rescored: false). Needs admin on the run's test set and write on its project.

client.test_runs.events()

events(
    test_run_id: str | UUID,
    /,
    *,
    last_event_id: str | UUID | None = None,
    timeout: float | None = None,
) -> None

Signature

Stream test-run events (SSE). GET /v1/test-runs/{testRunId}/events

Server-Sent Events. Per-case progress events (test-run.case-completed / test-run.case-failed) followed by a terminal event (test-run.completed / test-run.failed / test-run.cancelled) carrying the aggregate result. Mirrors /v1/runs/:runId/stream structurally.