Retire a case of an agent's test set

POST

/v1/agents/{agentId}/test-set/cases/{caseId}/retire

const url = 'https://example.com/v1/agents/example/test-set/cases/example/retire';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"reason":"example"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

curl --request POST \
  --url https://example.com/v1/agents/example/test-set/cases/example/retire \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "reason": "example" }'

A retired case stays listed, and no test run checks it again. There’s no un-retire, and retiring a correction’s case doesn’t change the correction’s status. Needs publish on the agent and read on the case’s project; for an agent more than one project serves, write on the case’s project.

Authorizations

bearerAuth

Parameters

Path Parameters

agentId

required

string

AgentId — dotted namespace (acme.drafting).

caseId

required

string

>= 1 characters

A correction’s id, or a hand case’s case-… id.

Header Parameters

Idempotency-Key

string

>= 1 characters

Caller-supplied idempotency key, scoped to the caller: the same key from someone else in the tenant is their own request (from 0.1.5). Retries with the same key return the original response byte-identical (per docs/API-ROUTE-CONVENTIONS.md §3.1). A retry sent while the first request still runs is answered 409 idempotency-key-in-flight with Retry-After, on a runtime whose store holds keys (from 0.1.5); retry after it, and you get the first request’s answer. An answer that carries a secret (a new API key or public run token, a session token, a generated signing secret) isn’t kept: a retry gets 409 idempotency-key-replay-withheld, with the first request’s status and when it succeeded (from 0.1.5). A request whose key store can’t be read before it runs is answered 503 idempotency-store-unavailable with Retry-After: nothing ran, so retry with the same key (from 0.3.0).

Request Body^required

application/json

object

reason

required

Why the case no longer applies.

string

>= 1 characters <= 4000 characters

Example^generated

{
  "reason": "example"
}

Responses

200

Retired.

application/json

object

required

A correction’s case has the correction’s id; a hand case’s starts case-.

string

kind

required

correction: a correction of the run. hand: added by hand.

string

Allowed values: correction hand

projectId

required

The run’s project.

string format: uuid

runId

required

string

madeOn

required

The agent version the run ran.

string

expect

required

One part for each part its correction gave a verdict on; at least one.

object

decisions

right: it calls the same tools the corrected run called, by name, in any order (args aren’t compared). wrong: it calls every shouldCall tool (with its args where given) and none of mustNotCall, checked exactly, without a model.

object

verdict

required

string

Allowed values: right wrong

shouldCall

Array<object>

object

tool

required

string

>= 1 characters <= 200 characters

args

required

The partial set of top-level argument fields to compare, by value. Fields left out are not compared; an empty object asks only that the tool was called. At most 4000 characters in canonical JSON.

object

<= 100 properties

mustNotCall

Array<string>

answer

A judging model checks it. right: it says what the corrected run’s answer said. wrong: it does what shouldSay or why asks.

object

verdict

required

string

Allowed values: right wrong

shouldSay

string

why

string

items

A ranked answer’s items, scored as weightedPrecisionAtK.

Array<object>

object

key

required

The caller’s stable id for the item, such as a matched case’s id, never the item’s own text.

string

>= 1 characters

pointer

Where the item is in the run’s output, as a JSON Pointer (RFC 6901), e.g. /matches/2.

string

rank

The item’s position in the ranked list (0 = first).

integer

verdict

required

string

Allowed values: right wrong

addedBy

required

Who made the correction, or added the case.

object

kind

required

string

Allowed values: user service

required

A user id, or for a service token its token or session id.

string

addedAt

required

string format: date-time

updatedAt

required

When the case last changed: its addedAt until it’s retired.

string format: date-time

retired

Present on a retired case: it stays listed, and no test run checks it.

object

reason

required

string

required

Who asserted a correction: the authenticated caller, never a typed name.

object

kind

required

string

Allowed values: user service

required

A user id, or for a service token its token or session id.

string

required

string format: date-time

expert

Present (true) on an expert’s case: a correction’s case when the correction was recorded while its group was restricted, and every hand case, since adding one needs publish on the agent, or write on its project when more than one project serves the agent. A test run weighted experts-only grades only these. Absent from older servers.

boolean

Example

{
  "kind": "correction",
  "expect": {
    "decisions": {
      "verdict": "right"
    },
    "answer": {
      "verdict": "right"
    },
    "items": [
      {
        "verdict": "right"
      }
    ]
  },
  "addedBy": {
    "kind": "user"
  },
  "retired": {
    "by": {
      "kind": "user"
    }
  },
  "expert": true
}

400

reason is missing, empty, or over 4000 characters.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

401

Missing / malformed / expired / revoked bearer token.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

404

test-case-not-found: the agent’s test set has no case by that id.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

409

test-case-retired: the case is retired already.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

413

The request body is over the API’s limit (request-too-large; details.maxBytes says how much).

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

500

Server error (unmapped domain code or framework crash).

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

503

The Idempotency-Key store couldn’t be read before the request ran (idempotency-store-unavailable), so nothing was changed; Retry-After (whole seconds) says when to try again with the same key.

application/json

object

error

required

object

code

required

Stable machine-readable discriminant. Values match domain error codes (see docs/API-ROUTE-CONVENTIONS.md §4.3).

string

message

required

string

details

Optional, kind-specific.

object

key

additional properties

any

requestId

required

Server-assigned request id; also echoed via X-Request-Id header.

string

Example^generated

{
  "error": {
    "code": "example",
    "message": "example",
    "details": {},
    "requestId": "example"
  }
}

Retire a case of an agent's test set

Authorizations

Parameters

Path Parameters

Header Parameters

Request Bodyrequired

Examplegenerated

Responses

200

Example

400

Examplegenerated

401

Examplegenerated

404

Examplegenerated

409

Examplegenerated

413

Examplegenerated

500

Examplegenerated

503

Examplegenerated

See also

Request Body^required

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated

Example^generated