kindgi eval-suites
Eval suites and test sets built from judgments (list / show / from-judgments / cases).
kindgi eval-suites list
Section titled “kindgi eval-suites list”List eval suites (the latest version of each).
kindgi eval-suites list [--project=<id>] [--kind=accuracy|pairwise|regression|human-review|benchmark|custom|judged] [--include-retired] [--limit=<n>] [--cursor=<c>]Flags:
--cursor <value>: Resume after this cursor, from the previous page'snextCursor.--include-retired: Include retired suites (every version unregistered), each as its highest version withunregisteredAt.--kind <value>: Only this kind: accuracy, pairwise, regression, human-review, benchmark, custom, judged.--limit <value>: The most to return (default 25, at most 100).--project <value>: Only suites in this project.
kindgi eval-suites show
Section titled “kindgi eval-suites show”Show an eval suite: its latest version, or the one named by --suite-version.
kindgi eval-suites show <suite-id> [--suite-version=<semver>]Flags:
--suite-version <value>: The version to show. Default: the latest.
kindgi eval-suites from-judgments
Section titled “kindgi eval-suites from-judgments”Build a test set: publish a judged suite version whose cases are copies of an agent's or flow's judged runs, with each item's judgments summed up.
kindgi eval-suites from-judgments <suite-id> --suite-version=<semver> --project=<id> (--agent=<id> [--agent-version=<v>] | --flow=<id>) [--since=<iso>] [--until=<iso>] [--class=<judge-class-id> ...] [--min-judgments=<n>] [--segment=<key:value> ...] [--description=<text>]Flags:
--agent <value>: Use judged runs of this agent.--agent-version <value>: Only runs of this agent version. Needs--agent.--class <value>: Count only judgments under this judge class id. Repeat the flag for several classes. (repeatable)--description <value>: A description for the suite version.--flow <value>: Use judged runs of this flow.--min-judgments <value>: Leave out runs with fewer counted judgments (default 1).--project <value>: The project whose judged runs to use. Required.--segment <value>: Only runs started in this segment or below it, as key:value; repeat it for a path, coarse to fine (--segment=company:acme --segment=role:cfo). (repeatable)--since <value>: Only runs first judged at or after this time: an ISO 8601 time with a zone, or a date (its start, UTC).--suite-version <value>: The version to publish. Required.--until <value>: Only runs first judged before this time: an ISO 8601 time with a zone, or a date (its start, UTC).
kindgi eval-suites cases
Section titled “kindgi eval-suites cases”List the cases of a judged suite version.
kindgi eval-suites cases <suite-id> --suite-version=<semver> [--limit=<n>] [--cursor=<c>]Flags:
--cursor <value>: Resume after this cursor, from the previous page'snextCursor.--limit <value>: The most to return (default 25, at most 100).--suite-version <value>: The suite version. Required.
Every command also takes the global flags.