client.agents.test_set

client.agents.test_set — the agents.testSet operations.

On AsyncKindgi every method is the same, awaited.

client.agents.test_set.get()

get(
    agent_id: str | UUID,
    /,
    *,
    project_id: str | UUID,
    timeout: float | None = None,
) -> AgentTestSet

Signature

A project's test set of an agent. GET /v1/agents/{agentId}/test-set

How many cases the agent's test set has in the project. Its cases are the agent's corrections there, except Won't fix and Removed ones (Fixed and With the owner included), and cases added by hand. It's computed when it's read. Needs read on the project.

client.agents.test_set.list_cases()

list_cases(
    agent_id: str | UUID,
    /,
    *,
    project_id: str | UUID,
    include: Literal['retired'] | None = None,
    kind: str | None = None,
    limit: int | None = None,
    cursor: str | None = None,
    timeout: float | None = None,
) -> TestCaseCollectionPage

Signature

List a project's cases of an agent's test set. GET /v1/agents/{agentId}/test-set/cases

Cursor-paginated, newest added first. A retired case comes back only with include=retired. Needs read on the project.

client.agents.test_set.add_case()

add_case(
    agent_id: str | UUID,
    body: AddTestCaseBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> TestCase

Signature

Add a case to an agent's test set. POST /v1/agents/{agentId}/test-set/cases

A case by hand, from a correction's body on one of the agent's finished runs. It has no status, isn't sorted, and nothing that counts corrections counts it (Improve's trigger, the expert queue's progress, a rule's yes-rate). The run's first correction or case stores a copy of its input and output. A comparison's replay can't be a case. Needs publish on the agent and read on the case's project; for an agent more than one project serves, write on the case's project.

client.agents.test_set.retire_case()

retire_case(
    agent_id: str | UUID,
    case_id: str | UUID,
    body: RetireTestCaseBody | Mapping[str, Any] | None = None,
    /,
    *,
    idempotency_key: str | None = None,
    timeout: float | None = None,
    **fields: Any,
) -> TestCase

Signature

Retire a case of an agent's test set. POST /v1/agents/{agentId}/test-set/cases/{caseId}/retire

A retired case stays listed, and no test run checks it again. There's no un-retire, and retiring a correction's case doesn't change the correction's status. Needs publish on the agent and read on the case's project; for an agent more than one project serves, write on the case's project.