# client.eval_suites

> client.eval_suites — the evalSuites operations.

`client.eval_suites` — the `evalSuites` operations.

On `AsyncKindgi` every method is the same, awaited.

## `client.eval_suites.list()`

```python signature
list(
    *,
    limit: int | None = None,
    cursor: str | None = None,
    kind: Literal['accuracy', 'pairwise', 'regression', 'human-review', 'benchmark', 'custom', 'judged'] | None = None,
    name: str | None = None,
    scope_kind: Literal['tenant', 'org', 'project'] | None = None,
    scope_id: str | UUID | None = None,
    inherit: bool | None = None,
    include_retired: bool | None = None,
    timeout: float | None = None,
) -> EvalSuiteCollectionPage
```

List tenant eval suites. `GET /v1/eval-suites`

Cursor-paginated. Optional `?kind=` narrows to a single eval kind (exact match); optional `?name=` is a prefix match on `EvalSuite.id`. Sort order: suite id ascending. Latest version per id. A retired suite (every version unregistered) is listed only with `?includeRetired=true`, as its highest version with `unregisteredAt`, so it can be found and a version reinstated.

## `client.eval_suites.publish()`

```python signature
publish(
    body: PublishEvalSuiteBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> PublishEvalSuiteResult
```

Publish an eval suite. `POST /v1/eval-suites`

Body is a full `EvalSuite` — the server validates top-level shape (id, tenantId, semver version, kind ∈ closed enum, spec is an object). Deeper `spec` validation is the runtime consumer's responsibility per kind (eval-judge adapter for `accuracy`/`pairwise`/`regression`, HITL bridge for `human-review`, sandbox handler for `custom`). Re-publishing an existing `(suiteId, version)` returns `409 eval-suite-already-registered`. Idempotency-Key applies (retries with the same key replay the original 201).

## `client.eval_suites.get()`

```python signature
get(suite_id: str | UUID, /, *, timeout: float | None = None) -> EvalSuite
```

Fetch an eval suite (latest version). `GET /v1/eval-suites/{suiteId}`

## `client.eval_suites.build_from_judgments()`

```python signature
build_from_judgments(
    suite_id: str | UUID,
    body: BuildJudgedSuiteBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> BuildJudgedSuiteResult
```

Build a test set from judgments. `POST /v1/eval-suites/{suiteId}/versions/from-judgments`

Publishes a `judged` eval suite version whose cases are copies of judged runs of one agent (optionally one version) or flow, newest first, at most 1000. Each case holds the run's input, what the turn read (`context`), the judged output, and each item's judgments summed up: yes and no counts, the weight behind yes and behind all judgments (an unclassified judgment counts 1), and the reasons. `judgeClassIds` counts only judgments of those classes; `minJudgments` leaves out runs with fewer; `segments` keeps only runs started in that segment path or below it. Needs `admin` on the project.

## `client.eval_suites.list_cases()`

```python signature
list_cases(
    suite_id: str | UUID,
    version: str,
    /,
    *,
    limit: int | None = None,
    cursor: str | None = None,
    timeout: float | None = None,
) -> JudgedEvalCaseCollectionPage
```

List the cases of a judged eval suite version. `GET /v1/eval-suites/{suiteId}/versions/{version}/cases`

Cursor-paginated, in the order the cases were stored (newest judged run first).

## `client.eval_suites.reinstate_version()`

```python signature
reinstate_version(
    suite_id: str | UUID,
    version: str,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
) -> ReinstateEvalSuiteVersionResult
```

Reinstate a specific tombstoned eval suite version. `POST /v1/eval-suites/{suiteId}/versions/{version}/reinstate`

Un-tombstones a previously-unregistered version: clears the tombstone on that version and recomputes the latest active version (refreshes `suiteKind` to match). Restores manifest bytes verbatim (semver hygiene). Idempotent. Publish is always allowed regardless of retirement state.

## `client.eval_suites.unregister()`

```python signature
unregister(
    suite_id: str | UUID,
    version: str,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
) -> UnregisterEvalSuiteResult
```

Unregister an eval suite version. `POST /v1/eval-suites/{suiteId}/versions/{version}/unregister`