client.test_sets

client.test_sets — the testSets operations.

On AsyncKindgi every method is the same, awaited.

client.test_sets.list()

list(
    *,
    limit: int | None = None,
    cursor: str | None = None,
    kind: Literal['accuracy', 'pairwise', 'regression', 'human-review', 'benchmark', 'custom', 'corrections'] | None = None,
    name: str | None = None,
    scope_kind: Literal['tenant', 'org', 'project'] | None = None,
    scope_id: str | UUID | None = None,
    inherit: bool | None = None,
    include_retired: bool | None = None,
    timeout: float | None = None,
) -> TestSetCollectionPage

Signature

List tenant test sets. GET /v1/test-sets

Cursor-paginated. Optional ?kind= narrows to a single eval kind (exact match); optional ?name= is a prefix match on TestSet.id. Sort order: test set id ascending. Latest version per id. A retired test set (every version unregistered) is listed only with ?includeRetired=true, as its highest version with unregisteredAt, so it can be found and a version reinstated. A corrections test set (an agent's frozen versions, and those built from corrections before) is listed only with ?kind=corrections: an agent's corrections are its test set (/v1/agents/{agentId}/test-set).

client.test_sets.publish()

publish(
    body: PublishTestSetBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> PublishTestSetResult

Signature

Publish a test set. POST /v1/test-sets

Body is a full TestSet — the server validates top-level shape (id, tenantId, semver version, kind ∈ closed enum, spec is an object). Deeper spec validation is the runtime consumer's responsibility per kind (eval-judge adapter for accuracy/pairwise/regression, HITL bridge for human-review, sandbox handler for custom). Re-publishing an existing (testSetId, version) returns 409 test-set-already-registered. Idempotency-Key applies (retries with the same key replay the original 201). An id that starts with agent. is refused with 400 bad-input: those are agents' own test sets, whose versions only a check or a test run makes.

client.test_sets.get()

get(
    test_set_id: str | UUID,
    /,
    *,
    project_id: str | UUID | None = None,
    timeout: float | None = None,
) -> TestSet

Signature

Fetch a test set (latest version). GET /v1/test-sets/{testSetId}

An item more than one project serves (an environment order carries it to later environments) answers its newest active version that can run in a project the caller reads, and 404 when there is none. projectId narrows that to one project. An item with one project answers its latest active version.

client.test_sets.list_cases()

list_cases(
    test_set_id: str | UUID,
    version: str,
    /,
    *,
    limit: int | None = None,
    cursor: str | None = None,
    timeout: float | None = None,
) -> CorrectedTestCaseCollectionPage

Signature

List the cases of a corrections test set version. GET /v1/test-sets/{testSetId}/versions/{version}/cases

Cursor-paginated, in the order the cases were stored (newest corrected run first).

client.test_sets.reinstate_version()

reinstate_version(
    test_set_id: str | UUID,
    version: str,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
) -> ReinstateTestSetVersionResult

Signature

Reinstate a specific tombstoned test set version. POST /v1/test-sets/{testSetId}/versions/{version}/reinstate

Un-tombstones a previously-unregistered version: clears the tombstone on that version and recomputes the latest active version (refreshes testSetKind to match). Restores manifest bytes verbatim (semver hygiene). Idempotent. Publish is always allowed regardless of retirement state.

client.test_sets.unregister()

unregister(
    test_set_id: str | UUID,
    version: str,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
) -> UnregisterTestSetResult

Signature

Unregister a test set version. POST /v1/test-sets/{testSetId}/versions/{version}/unregister