Skip to content
Kindgi

Test set

View as Markdown

@kindgi/specs/test-set.schema.json, schema version 1.0.0.

Tenant-scoped test set definition — the registry shape (/v1/test-sets). A test set defines WHAT a version is checked on; test runs + per-kind grader dispatch are separate concerns. kind selects the shape of spec; the registry treats spec as opaque JSON so new kinds can extend the catalog without breaking existing readers. Each kind's runtime consumer (eval-judge adapter for accuracy / pairwise / regression, HITL bridge for human-review, sandbox handler for custom) deserializes spec against its own contract. Case data lives in spec.

  • id (string, required): Stable test set id chosen by the caller (e.g. acme.drafting-accuracy).
  • tenantId (string, required): Tenant owning the test set. When published via HTTP, must match the caller tenant (server-derived from the token). Cross-tenant publish is rejected.
  • version (string, required): Semver — publishing a modified test set produces a new version. Test runs are traceable back to the version in force at the time.
  • kind ("accuracy" | "pairwise" | "regression" | "human-review" | "benchmark" | "custom" | "corrections", required): Closed enum, extended additively — new kinds require a spec + validator update in tandem so the registry never accepts a kind no runtime consumer honors. accuracy — metric-based grading against ground truth (input → expectedOutput pairs; grader is an eval-judge adapter). pairwise — A/B comparisons between two agent/flow versions. regression — compare against a stored baseline run. human-review — delegates rubric grading to HITL. benchmark — reference to a standardized test set (name + version). custom — caller-supplied evaluator handler (sandboxed handler-as-code). corrections — built from corrections of past runs: each case is a copy of a corrected run.
  • description (string)
  • spec (map of any, required): Kind-specific test set body. Registry validates only that this is an object; deeper validation is the runtime consumer's responsibility per kind. For accuracy, typically { cases: [{ input, expectedOutput }], grader?: { adapterId, config? } }. For pairwise, typically { prompts: [...], variantA: { agentId, version }, variantB: { agentId, version } }. For regression, typically { baseline: { runId } | { testSetId, version }, cases: [...] }. For human-review, typically { rubric: [...], reviewerRole }. For benchmark, typically { benchmark: { name, version } }. For custom, typically { handler: { modulePath, entrypointPath }, cases: [...] }.