client.eval_suites

client.eval_suites — the evalSuites operations.

On AsyncKindgi every method is the same, awaited.

client.eval_suites.list()

list(
    *,
    limit: int | None = None,
    cursor: str | None = None,
    kind: Literal['accuracy', 'pairwise', 'regression', 'human-review', 'benchmark', 'custom'] | None = None,
    name: str | None = None,
    scope_kind: Literal['tenant', 'org', 'project'] | None = None,
    scope_id: str | None = None,
    inherit: bool | None = None,
    timeout: float | None = None,
) -> EvalSuiteCollectionPage

List tenant eval suites. GET /v1/eval-suites

Cursor-paginated. Optional ?kind= narrows to a single eval kind (exact match); optional ?name= is a prefix match on EvalSuite.id. Sort order: suite id ascending. Latest version per id.

client.eval_suites.publish()

publish(
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> PublishEvalSuiteResult

Publish an eval suite. POST /v1/eval-suites

Body is a full EvalSuite — the server validates top-level shape (id, tenantId, semver version, kind ∈ closed enum, spec is an object). Deeper spec validation is the runtime consumer's responsibility per kind (eval-judge adapter for accuracy/pairwise/regression, HITL bridge for human-review, sandbox handler for custom). Re-publishing an existing (suiteId, version) returns 409 eval-suite-already-registered. Idempotency-Key applies (retries with the same key replay the original 201).

client.eval_suites.get()

get(*, timeout: float | None = None) -> EvalSuite

Fetch an eval suite (latest version). GET /v1/eval-suites/{suiteId}

client.eval_suites.reinstate_version()

reinstate_version(
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
) -> ReinstateEvalSuiteVersionResult

Reinstate a specific tombstoned eval suite version. POST /v1/eval-suites/{suiteId}/versions/{version}/reinstate

Un-tombstones a previously-unregistered version: clears the tombstone on that version and recomputes the latest active version (refreshes suiteKind to match). Restores manifest bytes verbatim (semver hygiene). Idempotent. Publish is always allowed regardless of retirement state.