Corrections reference

This page lists everything about correcting runs: what a correction holds, where it goes until it's fixed, how corrector groups weigh it, what queue rules match, what the answers waiting for a correction hold, how a rule's results count, and what the console's Corrections section shows. To correct a run, follow Correct a run's output.

Corrections

A correction is one person's word on one run. It says at least one of these, each right or wrong:

• decisions says whether the tools the run called were right. When it's wrong, it says what the run should have done, in shouldCall and mustNotCall, with at least one entry between them.
• answer says whether the run's answer was right. When it's wrong, it needs shouldSay, what the answer should have said, or why.
• items correct an answer that is a ranked list, one entry for each element: { key, pointer?, rank?, verdict }.
  • key is your stable id for the element, such as a matched case's id, never the element's own text. It's at most 200 characters, each key appears once, and kindgi:correction is reserved.
  • pointer (optional) is where the element is in the run's output, as a JSON Pointer such as /matches/2, at most 500 characters.
  • rank (optional) is its place in the list, 0 first.
  • verdict is right or wrong. A wrong item needs nothing more.

And around them:

• why (optional) says why, for the whole correction.
• Who corrected it comes from the token, as madeBy. An app that records corrections for its own users adds participantId, its own opaque id for the user.
• Each person has one correction per run. Correcting the run again, as the same caller and participant, under any group or none, replaces the earlier correction. The earlier one stays in the history, and its group only says how much the correction counts.
• A correction keeps copies of the run's input and output, so it outlives the run's own retention.
• Correcting a run takes write access to its project, as cancelling it does.
• What it should have done (runtime 0.2.0 or later): a correction of an agent's run can say it, with decisions: "wrong". Each of shouldCall is { tool, args? }, and each of mustNotCall is a tool id. args holds only the arguments that matter; leaving it out, or {}, asks only that the tool was called. Both lists go with decisions: "wrong" only.
• How it's checked, on a replay: only the tools it names are compared. Each tool in shouldCall must be called as many times as it's listed, each call matching one listed call, in any order. A listed call's args match when each field given equals the call's top-level argument of that name, compared whole, and {} matches any arguments. A tool in mustNotCall must not be called. A call the replay refused still counts as called.
• Its limits: up to 50 shouldCall and 50 mustNotCall, tool ids up to 200 characters, each args up to 100 fields and 4,000 characters of JSON, shouldSay and why up to 4,000 characters each, and up to 100 items.

Only a finished run can be corrected, and each item's pointer must find something:

Run "fc26b436-611b-4962-9913-2f2669ade7e1" has no output to correct yet (status suspended).
Nothing at "/no/such" in run "bc6d1b53-3562-48c9-9f29-719779e3d110"'s output.

A correction that breaks a rule is refused with 400 bad-input, such as:

A correction says at least one of decisions, answer and items (right or wrong).
decisions: "wrong" needs at least one entry in shouldCall or mustNotCall.
"acme-store.refund" can't be in both shouldCall and mustNotCall.
shouldCall and mustNotCall go with decisions: "wrong".
answer: "wrong" needs shouldSay or why.
shouldSay goes with answer: "wrong".
the item key "kb-101" is in items twice.
an item's key is at most 200 characters: its id, not its text.
an item's pointer is at most 500 characters.
verdict is gone: a correction says decisions and answer.
The corrected version has no tool "acme-store.refunds".

The fields item, verdict, reason and correction are gone, and each is refused as verdict is above.

Status and sorting

Each correction moves on until it's fixed. Its status says where it is, and sortedTo says who fixes it:

• status: open, being-fixed, with-owner (the agent's owner decides), fixed, wont-fix or removed.
• sortedTo: improve (the instructions, through an Improve run), developer (a change in code), or owner. It's absent while the correction is unsorted.
• sorted says how it was sorted, with by ({ "kind": "rule" } when it's saved, { "kind": "fix-job" }, or { "kind": "user", "id": … }), an optional reason, and at.
• link, while it's being-fixed, is the work fixing it: an Improve run, a suggested change, or a pull request's URL. An Improve run or a suggested change is named by its id alone, without the ir- or sc- you send ({ "kind": "improve-run", "id": … }), as their own routes take it.
• fixedIn, while it's fixed, says how: how and at, with by and version when they apply. A correction that found nothing wrong is fixed as recorded-right.
• reopenedIn is there after a test run got a fixed correction wrong again: the version that gets it wrong, how the project came to run it (pin, went-back, unpin or latest), at, and by unless Latest moved. The correction is open again, or with-owner, and keeps the fixedIn it had. A status you set clears both; a re-sort keeps them.
• statusNote is why it's wont-fix, or why it went back to open, with text, by and at.

When a correction is saved, it's sorted by these rules, in order:

1. A user's correction (one with participantId, not made as an expert under a restricted group) is open and unsorted, until a person sorts it, unless the project sorts its end users' corrections (sortEndUserCorrections): then rules 2 to 4 sort it like the others.
2. Wrong decisions go to the owner: with-owner, sorted to owner. A change to what the agent does may loosen a limit, and only the owner decides that. This holds when the answer is wrong too.
3. A wrong answer, or a wrong item, is open, sorted to improve.
4. Nothing wrong: it's fixed, as recorded-right.

Change a correction

PATCH /v1/corrections/{id}, or corrections.change in the clients, takes sortedTo, status, or both, with a link or a reason. In the CLI, it's kindgi corrections set <id> with --sorted-to, --status, --link and --reason. It needs publish on the correction's agent or flow, or, for one that more than one project serves, write on the correction's project.

• Re-sort with sortedTo, from open, being-fixed or with-owner. Sorted to owner, it becomes with-owner; sorted away from owner, it becomes open. It's recorded as sorted by you. A being-fixed correction keeps its link, unless the re-sort moves it to or away from the owner.
• being-fixed, from open or with-owner, needs a link: a URL, an Improve run (ir-<its id>) or a suggested change (sc-<its id>).
• open, from being-fixed or wont-fix, takes an optional reason. One sorted to the owner becomes with-owner.
• wont-fix, from anything but removed, needs a reason. A fixed correction loses its fixedIn.
• fixed can't be set by hand.

Any other change is refused with 409 correction-status-change-not-allowed, with the two statuses in details as from and to:

A correction that's fixed can't become being-fixed.

A change that's missing something, or whose link is none of those, is refused with 400 bad-input:

status: "being-fixed" needs a link: a URL, ir-<its id> or sc-<its id>.
link must be a URL, an Improve run (ir-<its id>) or a suggested change (sc-<its id>).
status: "wont-fix" needs a reason.
link goes with status: "being-fixed".
A change sets sortedTo, status, or both.

Removing a correction is for its author, or an admin of its project. It becomes removed, and it can't come back.

List and count

GET /v1/corrections, or corrections.list in the clients, lists newest first. Besides the run, agent, agent version, flow, group and participant, it filters by:

• status, repeated or comma-separated. not-fixed means open, being-fixed and with-owner. Removed corrections are listed only when status asks for removed.
• sortedTo: improve, developer, owner, or none for the unsorted ones, repeated or comma-separated too.
• from: experts (made under a restricted group), end-users (with participantId, and not made as experts), or all.
• madeBy=me: your own corrections.
• since and until: made at or after since, and before until, in ISO 8601.

GET /v1/corrections/counts?scopeKind=project&scopeId=<project-id>, with an optional agentId, counts a project's corrections by status: open, beingFixed, withOwner, fixed, wontFix and removed. In the TypeScript client it's corrections.counts({ projectId, agentId }), and in Python corrections.counts(scope_kind="project", scope_id=..., agent_id=...). It counts in a project only, and takes read access to it.

Export

GET /v1/corrections/export?agentId=<agent-id>, or corrections.export in the TypeScript and Python clients, streams every correction of the agent that the list's filters match, without paging:

• format: csv, the default (RFC 4180, UTF-8, a header row, text/csv), or jsonl (one correction per line, as the list sends it, application/jsonl).
• Filters: the list's, above. agentId is required, and it takes no limit or cursor.
• Access: each correction comes only when you may read its project. A project scope you can't read is a 403, before anything is sent.

It's recorded in the audit log as two events with the same exportId: corrections-export-started, before the first row, with who, the filters and the format, and corrections-exported when the stream ends, with the rows sent and an outcome of succeeded or failed (a page read broke, or the client left). When the first can't be written, the export is a 500 and sends nothing.

Corrector groups

• A group weighs its corrections: --weight=3 counts each three times. Without a group, a correction counts once.

• A group applies to the tenant, a project, or an agent in a project.

• A restriction says who may correct as the group:

  • --min-reviewer-role: standard, senior or admin, a reviewer of that role or above;
  • --principal-kind: user for people, or service for service tokens;
  • --principal-id: named users or tokens, by id.

  The last two repeat. A caller must meet each one you set. In the API and the clients, it's assertableBy.

• Whom a group names is for the admins of its scope. Anyone else who reads the group gets how many it names, as principalCount, not their ids.

• A correction made under a restriction carries "expert": true, in kindgi corrections list too. It's decided when the correction is made: corrections recorded before the restriction don't count as expert.

• Lifting a restriction, with kindgi corrector-groups set <corrector-group-id> --unrestricted, keeps the earlier corrections marked expert. New ones aren't.

• kindgi corrector-groups set with restriction flags replaces the whole restriction.

Test cases count expert-group corrections apart, and a run on the test cases can count only them, as Try a version on the test cases shows.

Queue rules

What a rule matches, in when, each field narrowing it:

• agentIds or flowIds: runs of these agents or flows, one or the other, not both. A rule naming an agent matches the agent's own runs, not its turns as a step of a flow.
• versions: these versions exactly, or pinned: runs that ran the agent's pinned version rather than one the caller named. So the rule follows each release.
• status: how the run ended. Only completed, since only a completed run can be corrected. It's also the default.
• includeDryRuns: dry runs are left out unless this is true. The runs that a run on the test cases makes are never queued.

How much of it to queue:

• sample: the share of matching runs, 0.05 for 5%. The same run and rule always give the same answer, across the rule's versions too. So raising it later keeps the runs it took before, and adds more.
• maxOpen: the rule adds nothing while this many of its runs wait, and starts again as they're corrected.
• correctorGroupId: whose correction the rule wants, a corrector group. Without it, any correction does.

A rule applies to runs that end after it's made. To try one before you make it, queueRules.preview takes the same fields. It answers how many of the project's last 100 runs it would have queued, with its sample but not its maxOpen. last sets how many runs, up to 500. Give an existing rule's ruleId to sample as it does. Without one, the count is an estimate. A run that doesn't record how its version was chosen never matches pinned.

To catch a rule up on runs that ended before it was made, queueRules.queueRecent queues its matches among the project's recent runs: the last of them (up to 1,000, newest first), those completed at or after since, or, with both, the last of those completed at or after since. It samples as the rule samples, so it picks the runs the rule itself would have. It doesn't apply maxOpen: a catch-up can leave more than maxOpen of the rule's runs waiting, and the rule then adds nothing at run end until fewer wait. The answer counts the matches three ways, which together are all of them:

• queued: queued for the rule by this call, as a new item or added to one that's already waiting for another rule.
• alreadyQueued: the rule had queued them already, and they're still waiting.
• alreadyReviewed: their run was already corrected or dismissed, so they're left as they are.

An erased run isn't a match, so it's in none of the three.

A rule only lists runs: it never runs a model or spends anything.

Change a rule

• queueRules.update writes a new version of the rule. Runs queued from then on count toward the new version. null removes maxOpen or correctorGroupId.
• queueRules.versions lists every version, newest first, with who wrote each and when.
• queueRules.unregister stops a rule. It queues nothing more. Its versions and the runs it queued stay.

Answers waiting for a correction

expertQueue.list gives the queue of answers waiting for a correction, oldest first.

• Each item is a run: its agent or flow and version, how it ended, the rules that queued it, each at the version that did, the corrector groups they want, and the corrections on it so far, by group. It carries none of the run's content: open the run to correct it.
• forMe: true keeps the items you can close: those that want a group you may correct as, or any correction.
• An item closes as corrected once each rule that queued it has the correction it wants: one of its group, or any correction for a rule without a group.
• expertQueue.dismiss, with an optional reason, takes a run not worth correcting out of the queue. expertQueue.reopen puts it back. Each item's can says whether you may.
• Counts: limit: 0 answers only the total. With state, addedAfter and closedAfter, it counts, for example, how many are waiting and how many were corrected this week.
• When a run's content is gone, erased or purged by retention, its waiting item closes as erased and shows nothing of the run.

Reading the rules and the queue takes read access to the project. Changing a rule, queueing its recent matches, and dismissing or reopening an item, take write access, as correcting a run does.

A rule's results

queueRules.results gives a group for each version of the rule and each agent version:

• The runs it queued: added, and of them open, corrected, dismissed and erased.
• The corrections on them, with their weighted share marked right, and the corrections by group.
• The share marked right is per correction: one person's verdict on one item of a run's output.
• runsWithCorrections counts the runs those corrections come from: the queued runs with at least one live correction, whatever their item's state now. That's still open, closed as corrected, or dismissed after someone corrected it. corrections counts verdicts, so several people correcting one run make several corrections but one run here. To see how much a share rests on, count runs, not corrections.
• skippedByCap counts the runs the rule matched and sampled but didn't queue because maxOpen were waiting, even when another rule queued them. So added + skippedByCap is every run it matched and sampled. When it isn't 0, the queued runs lean toward quiet times.
• Never pooled: each version of each rule is a sample of its own when, so its results are never added to another's.

Here a rule that wants an expert's correction queued four runs. An expert, with weight 3, and a peer corrected the first. A peer marked the second wrong, and it's still open. A peer corrected the third, which was then dismissed. The fourth was never corrected. So corrections is 4, runsWithCorrections 3, and yesShare 0.83: 5 of 6 weighted verdicts.

{
  "ruleId": "fd7a1785-717b-404e-bcdc-3b819806d784",
  "groups": [
    {
      "ruleVersion": 1,
      "agentVersion": "1.0.0",
      "added": 4,
      "open": 2,
      "corrected": 1,
      "dismissed": 1,
      "erased": 0,
      "skippedByCap": 0,
      "corrections": 4,
      "runsWithCorrections": 3,
      "yesShare": 0.8333333333333334,
      "byGroup": [
        {
          "correctorGroupId": "2da17046-eb37-4157-b175-3381a926545a",
          "corrections": 3,
          "yes": 2
        },
        {
          "correctorGroupId": "787b6f6f-30cf-4bac-bf27-ac5ef32cb6d3",
          "corrections": 1,
          "yes": 1
        }
      ]
    }
  ]
}

In the console

Corrections, in the menu's Improve group, is a project's corrections and the runs waiting for one. Its header says what it holds: "Runs waiting for a person to review them, every correction not fixed yet and who fixes it, and who corrects." Its pages are To review, To fix, Who corrects, Rules and Trends. The breadcrumbs at the top end on the one you're on, such as Corrections › Rules ▾, and its ▾, Corrections pages, lists the others. Their tables sort by a column's header, and Columns chooses which columns each shows. The page, the filters and each table's view go in the page's address, such as ?tab=rules, so a link opens the same view. An agent's own corrections, test set and suggested changes are pages of its own, as Improve an agent shows.

To review

Each row is a run in the queue:

• What ran: the agent or flow and its version, and when it finished. It links the run's Corrections page, where you correct it, as Correct a run's answer shows.
• Why it's here: the rules that queued it.
• Where it stands: what it still needs, such as "Needs an expert correction (1 so far, ungrouped)", or how a closed one ended.

Filter it by state, Waiting, Corrected, Dismissed or Erased. Or by For me or Everyone's ("2 more wait for other correctors."), by rule and by agent. Dismiss takes a run out of the queue with a reason, and Reopen puts a dismissed one back.

To fix

To fix lists the project's corrections that aren't fixed yet, in parts by who fixes them, and leaves out a part with none:

• Nobody yet: "end users' corrections until someone hands each one on, and a flow's corrections sorted to Improve, which takes only an agent's."
• Developer: "fixes it in the pack's code or files."
• Owner: "decides the agent's or flow's rules and tools."
• Improve: "tries new wording or settings on the test set. Run it from the agent's Suggested changes page."

Each row names its agent or flow, which links that one's Corrections page, and Hand it to someone else… changes who fixes it. With none left: "Nothing to fix: every correction here is Fixed, Won't fix or Removed."

Who corrects

Who corrects lists, under Corrector groups, each corrector group that applies to the project's runs, in words:

• how much it counts, such as "Counts 3× one with no group";
• where it applies, such as "This project";
• who may give it, such as "Only people" or "Only services". Who may give an expert group is counted here, never listed;
• the rules that want it.

Below the list, the page says what makes an expert: "Corrections in an expert group are the experts' corrections: "Count only experts" counts only those."

Project admins add, change and remove a project's groups, and tenant admins a tenant's. Removing a group names the rules that want it.

Rules

Each rule is in words:

• Which runs: "Completed runs of Dev-Echo Agent", and how many it takes: "Takes every matching run", or "Takes 1 in 10".
• What it wants: "An expert correction", "Any correction" or "A bot correction".
• Its cap: "No cap", or no new items while that many are open.
• What it has open now: for example "3 open", linking To review, filtered to the rule.

A rule matches an agent's or a flow's own runs, not an agent's turns as a flow's step. Replays never match.

Project editors see New rule, and Change on each rule. The editor shows what the rule would have queued from the recent runs, and saving makes a new version of the rule. History lists what each version changed, who changed it and when. Remove stops a rule, and the runs it queued stay in the queue.

Trends

Trends shows, per rule and by its latest version, the share marked right among people's corrections of each agent version, over all time or the last 90 or 30 days. These are a rule's results:

• A version shows from 10 corrections. With fewer, it says so: "1.0.0: too few corrections to say (2 so far)".
• Each line names its corrections and its range: "1.1.0: about 95% marked right, from 12 corrections (12 runs corrected so far); range 64–100%".
• A comparison only when the ranges are apart: "1.1.0 is marked right more often than 1.0.0."
• The weights, in words: "Weighted by corrector group: An expert's correction counts 3× one with no group; a bot's correction counts the same as one with no group."
• What the numbers leave out: an earlier version of the rule is "shown apart, never added in". Runs the cap kept out are counted: "1 matched run wasn't queued because the rule already had its most open runs waiting, so the queued ones lean toward quieter times."

Every count opens the counts behind a rule's trend.