Skip to content
Kindgi

List a project's cases of an agent's test set

View as Markdown
GET
/v1/agents/{agentId}/test-set/cases
curl --request GET \
  --url 'https://example.com/v1/agents/example/test-set/cases?projectId=2489E9AD-2EE2-8E00-8EC9-32D5F69181C0&include=retired&limit=25' \
  --header 'Authorization: Bearer <token>'

Cursor-paginated, newest added first. A retired case comes back only with include=retired. Needs read on the project.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
agentId
required
string

AgentId — dotted namespace (acme.drafting).

Query Parameters

Section titled “Query Parameters”
projectId
required
string format: uuid

The project whose cases to read: the agent’s set, narrowed to that project’s cases.

include
string
Allowed values: retired

retired: retired cases too. Left out, active ones only.

kind
string
/^(correction|hand)(,(correction|hand))*$/

Comma-separated correction and hand. Left out, both.

limit
integer
default: 25 >= 1 <= 100

1..100. Default 25.

cursor
string

Where the previous page ended: its nextCursor, as it came. A runtime that seals cursors takes one only for the same list, filters and caller, within a day; otherwise 400 bad-input, and the list starts again without it. Absent → first page.

Responses

Section titled “Responses”

200

Section titled “200”

Page of cases.

Media typeapplication/json
object
data
required
Array<object>
object
id
required

A correction’s case has the correction’s id; a hand case’s starts case-.

string
kind
required

correction: a correction of the run. hand: added by hand.

string
Allowed values: correction hand
projectId
required

The run’s project.

string format: uuid
runId
required
string
madeOn
required

The agent version the run ran.

string
expect
required

One part for each part its correction gave a verdict on; at least one.

object
decisions

right: it calls the same tools the corrected run called, by name, in any order (args aren’t compared). wrong: it calls every shouldCall tool (with its args where given) and none of mustNotCall, checked exactly, without a model.

object
verdict
required
string
Allowed values: right wrong
shouldCall
Array<object>
object
tool
required
string
>= 1 characters <= 200 characters
args
required

The partial set of top-level argument fields to compare, by value. Fields left out are not compared; an empty object asks only that the tool was called. At most 4000 characters in canonical JSON.

object
<= 100 properties
mustNotCall
Array<string>
answer

A judging model checks it. right: it says what the corrected run’s answer said. wrong: it does what shouldSay or why asks.

object
verdict
required
string
Allowed values: right wrong
shouldSay
string
why
string
items

A ranked answer’s items, scored as weightedPrecisionAtK.

Array<object>
object
key
required

The caller’s stable id for the item, such as a matched case’s id, never the item’s own text.

string
>= 1 characters
pointer

Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2.

string
rank

The item’s position in the ranked list (0 = first).

integer
verdict
required
string
Allowed values: right wrong
addedBy
required

Who made the correction, or added the case.

object
kind
required
string
Allowed values: user service
id
required

A user id, or for a service token its token or session id.

string
addedAt
required
string format: date-time
updatedAt
required

When the case last changed: its addedAt until it’s retired.

string format: date-time
retired

Present on a retired case: it stays listed, and no test run checks it.

object
reason
required
string
by
required

Who asserted a correction: the authenticated caller, never a typed name.

object
kind
required
string
Allowed values: user service
id
required

A user id, or for a service token its token or session id.

string
at
required
string format: date-time
expert

Present (true) on an expert’s case: a correction’s case when the correction was recorded while its group was restricted, and every hand case, since adding one needs publish on the agent, or write on its project when more than one project serves the agent. A test run weighted experts-only grades only these. Absent from older servers.

boolean
nextCursor

Opaque cursor. Treat as opaque on the client.

string
hasMore
required
boolean
Example
{
  "data": [
    {
      "kind": "correction",
      "expect": {
        "decisions": {
          "verdict": "right"
        },
        "answer": {
          "verdict": "right"
        },
        "items": [
          {
            "verdict": "right"
          }
        ]
      },
      "addedBy": {
        "kind": "user"
      },
      "retired": {
        "by": {
          "kind": "user"
        }
      },
      "expert": true
    }
  ]
}

400

Section titled “400”

projectId is missing or not a project id (a UUID), or include or kind names something else.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

401

Section titled “401”

Missing / malformed / expired / revoked bearer token.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

404

Section titled “404”

agent-not-found.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

503

Section titled “503”

The database was too busy to answer in time (database-busy); Retry-After (whole seconds) says when to try again.

Media typeapplication/json
object
error
required
object
code
required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string
message
required
string
details

Optional, kind-specific.

object
key
additional properties
any
requestId
required

Server-assigned request id; also echoed via X-Request-Id header.

string
Examplegenerated
{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

See also