client.eval_suites

client.eval_suites — the evalSuites operations.

On AsyncKindgi every method is the same, awaited.

client.eval_suites.list()

list(
    *,
    limit: int | None = None,
    cursor: str | None = None,
    kind: Literal['accuracy', 'pairwise', 'regression', 'human-review', 'benchmark', 'custom', 'judged'] | None = None,
    name: str | None = None,
    scope_kind: Literal['tenant', 'org', 'project'] | None = None,
    scope_id: str | UUID | None = None,
    inherit: bool | None = None,
    include_retired: bool | None = None,
    timeout: float | None = None,
) -> EvalSuiteCollectionPage

Signature

List tenant eval suites. GET /v1/eval-suites

Cursor-paginated. Optional ?kind= narrows to a single eval kind (exact match); optional ?name= is a prefix match on EvalSuite.id. Sort order: suite id ascending. Latest version per id. A retired suite (every version unregistered) is listed only with ?includeRetired=true, as its highest version with unregisteredAt, so it can be found and a version reinstated.

client.eval_suites.publish()

publish(
    body: PublishEvalSuiteBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> PublishEvalSuiteResult

Signature

Publish an eval suite. POST /v1/eval-suites

Body is a full EvalSuite — the server validates top-level shape (id, tenantId, semver version, kind ∈ closed enum, spec is an object). Deeper spec validation is the runtime consumer's responsibility per kind (eval-judge adapter for accuracy/pairwise/regression, HITL bridge for human-review, sandbox handler for custom). Re-publishing an existing (suiteId, version) returns 409 eval-suite-already-registered. Idempotency-Key applies (retries with the same key replay the original 201).

client.eval_suites.get()

get(suite_id: str | UUID, /, *, timeout: float | None = None) -> EvalSuite

Signature

Fetch an eval suite (latest version). GET /v1/eval-suites/{suiteId}

client.eval_suites.build_from_judgments()

build_from_judgments(
    suite_id: str | UUID,
    body: BuildJudgedSuiteBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> BuildJudgedSuiteResult

Signature

Build a test set from judgments. POST /v1/eval-suites/{suiteId}/versions/from-judgments

Publishes a judged eval suite version whose cases are copies of judged runs of one agent (optionally one version) or flow, newest first, at most 1000. Each case holds the run's input, what the turn read (context), the judged output, and each item's judgments summed up: yes and no counts, the weight behind yes and behind all judgments (an unclassified judgment counts 1), and the reasons. judgeClassIds counts only judgments of those classes; minJudgments leaves out runs with fewer; segments keeps only runs started in that segment path or below it. Needs admin on the project.

client.eval_suites.list_cases()

list_cases(
    suite_id: str | UUID,
    version: str,
    /,
    *,
    limit: int | None = None,
    cursor: str | None = None,
    timeout: float | None = None,
) -> JudgedEvalCaseCollectionPage

Signature

List the cases of a judged eval suite version. GET /v1/eval-suites/{suiteId}/versions/{version}/cases

Cursor-paginated, in the order the cases were stored (newest judged run first).

client.eval_suites.reinstate_version()

reinstate_version(
    suite_id: str | UUID,
    version: str,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
) -> ReinstateEvalSuiteVersionResult

Signature

Reinstate a specific tombstoned eval suite version. POST /v1/eval-suites/{suiteId}/versions/{version}/reinstate

Un-tombstones a previously-unregistered version: clears the tombstone on that version and recomputes the latest active version (refreshes suiteKind to match). Restores manifest bytes verbatim (semver hygiene). Idempotent. Publish is always allowed regardless of retirement state.

client.eval_suites.unregister()

unregister(
    suite_id: str | UUID,
    version: str,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
) -> UnregisterEvalSuiteResult

Signature

Unregister an eval suite version. POST /v1/eval-suites/{suiteId}/versions/{version}/unregister