# Test set

> Workspace-scoped test set definition — the *registry* shape (/v1/test-sets).

`@kindgi/specs/test-set.schema.json`, schema version 1.0.0.

Workspace-scoped test set definition — the _registry_ shape (`/v1/test-sets`). A test set defines WHAT a version is checked on; test runs + per-kind grader dispatch are separate concerns. `kind` selects the shape of `spec`; the registry treats `spec` as opaque JSON so new kinds can extend the catalog without breaking existing readers. Each kind's runtime consumer (eval-judge adapter for `accuracy` / `pairwise` / `regression`, HITL bridge for `human-review`, sandbox handler for `custom`) deserializes `spec` against its own contract. Case data lives in `spec`.

- `id` (string, required): Stable test set id chosen by the caller (e.g. `acme.drafting-accuracy`).
- `tenantId` (string, required): The workspace owning the test set. When published via HTTP, must match the caller's workspace (server-derived from the token). Cross-tenant publish is rejected.
- `version` (string, required): Semver — publishing a modified test set produces a new version. Test runs are traceable back to the version in force at the time.
- `kind` (`"accuracy"` | `"pairwise"` | `"regression"` | `"human-review"` | `"benchmark"` | `"custom"` | `"corrections"`, required): Closed enum, extended additively — new kinds require a spec + validator update in tandem so the registry never accepts a kind no runtime consumer honors. `accuracy` — metric-based grading against ground truth (input → expectedOutput pairs; grader is an eval-judge adapter). `pairwise` — A/B comparisons between two agent/flow versions. `regression` — compare against a stored baseline run. `human-review` — delegates rubric grading to HITL. `benchmark` — reference to a standardized test set (name + version). `custom` — caller-supplied evaluator handler (sandboxed handler-as-code). `corrections` — built from corrections of past runs: each case is a copy of a corrected run.
- `description` (string)
- `spec` (map of any, required): Kind-specific test set body. Registry validates only that this is an object; deeper validation is the runtime consumer's responsibility per kind. For `accuracy`, typically `{ cases: [{ input, expectedOutput }], grader?: { adapterId, config? } }`. For `pairwise`, typically `{ prompts: [...], variantA: { agentId, version }, variantB: { agentId, version } }`. For `regression`, typically `{ baseline: { runId } | { testSetId, version }, cases: [...] }`. For `human-review`, typically `{ rubric: [...], reviewerRole }`. For `benchmark`, typically `{ benchmark: { name, version } }`. For `custom`, typically `{ handler: { modulePath, entrypointPath }, cases: [...] }`.