Skip to content
Kindgi

Trace an answer and its cost

View as Markdown

This guide reads a turn's provenance record, to see where an answer came from and what it cost.

Every agent turn leaves a provenance record: a graph of the turn, from the user's message through each model call and tool call to the answer. It names the model of each call and the version of each tool, and, beside the graph, each call's tokens and cost. Use it to answer "where did this answer come from?" and "what did it cost?". A finished flow run has a record too, of its steps and the runs they started: A flow's answers reads it.

Before you start

Section titled “Before you start”
  • A runtime, such as kindgi dev, with Kindgi 0.2's CLI.
  • The id of an agent turn's run. kindgi runs list shows it.
  • KINDGI_API_URL and KINDGI_API_TOKEN set in your shell, for the HTTP requests. With kindgi dev, they're apiUrl and token in the pack's .kindgirc.json.

Read a turn's record

Section titled “Read a turn's record”

Fetch it by the run's id:

Terminal window
pnpm exec kindgi provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c

Over HTTP it's GET /v1/provenance/<run-id>:

Terminal window
curl "$KINDGI_API_URL/v1/provenance/a64ce71c-9cfc-4263-87b1-b5562f09079c" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN")

Both answer the record:

{
  "id": "9f0f7385-5270-484d-9ee4-13716c4edb6e",
  "runId": "a64ce71c-9cfc-4263-87b1-b5562f09079c",
  …
  "dag": {
    "nodes": [
      {"id": "input:0", "kind": "input", "timestamp": "2026-10-08T15:13:29.826Z"},
      {"id": "model-call:1", "kind": "model-call", "timestamp": "2026-10-08T15:13:31.345Z", "modelVersion": "anthropic/claude-sonnet-5-5", "attributes": {"step": 1, "model": "claude-sonnet-5-5", "callId": "975d170b-d14f-4fbf-b662-85bcf26b9431", "providerId": "anthropic", "finishReason": "tool-use"}},
      {"id": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "tool-call", "timestamp": "2026-10-08T15:13:31.369Z", "attributes": {"toolId": "acme.echo", "toolVersion": "0.1.0", "invocationId": "toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "toolVersionRange": "0.1.0"}},
      {"id": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "tool-result", "timestamp": "2026-10-08T15:13:31.369Z"},
      {"id": "model-call:2", "kind": "model-call", "timestamp": "2026-10-08T15:13:32.418Z", "modelVersion": "anthropic/claude-sonnet-5-5", "attributes": {"step": 2, "model": "claude-sonnet-5-5", "callId": "99929d2e-05e2-4452-853e-b6c6f0deed6e", "providerId": "anthropic", "finishReason": "stop"}},
      {"id": "model-output:2", "kind": "model-output", "timestamp": "2026-10-08T15:13:32.442Z", "modelVersion": "anthropic/claude-sonnet-5-5"}
    ],
    "edges": [
      {"from": "model-call:1", "to": "input:0", "kind": "caused-by"},
      {"from": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "to": "model-call:1", "kind": "invoked"},
      {"from": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "to": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "produced"},
      {"from": "model-call:2", "to": "input:0", "kind": "caused-by"},
      {"from": "model-call:2", "to": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "influenced-by"},
      {"from": "model-output:2", "to": "model-call:2", "kind": "produced"}
    ]
  },
  "callUsage": {
    "975d170b-d14f-4fbf-b662-85bcf26b9431": {"usage": {"promptTokens": 645, "completionTokens": 132, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "costUsd": 0.0026100000000000003, "durationMs": 1460, "servedModel": "claude-sonnet-5-5"},
    "99929d2e-05e2-4452-853e-b6c6f0deed6e": {"usage": {"promptTokens": 740, "completionTokens": 42, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "costUsd": 0.0019, "durationMs": 1012, "servedModel": "claude-sonnet-5-5"}
  }
}
Excerpt

Read it from the answer backwards:

  • The answer, model-output:2, was produced by the second model call.
  • That call answered the user's message: it's caused-by input:0.
  • It also read the echo tool's result: it's influenced-by that tool-result.
  • The tool call that produced the result was invoked by the first model call.
Node What it is
input The message the turn answered.
model-call One call to a model: modelVersion is provider/model; attributes has its step, model, providerId, why it stopped (finishReason), its callId, and its thinking when it asked for or sent a level: asked, the level asked for, and level, the level sent (Temperature and thinking).
tool-call, tool-result A tool the model called, with the exact tool version that ran, and what it returned.
model-output The answer.
wait, resume A tool call that waited for an approval: the wait, and the decision that ended it.

Edges point from an effect to its cause: caused-by, invoked, produced, and influenced-by from a model call to each tool result it read.

callUsage holds each model call's tokens, cost in USD, duration and the model version that answered, servedModel, by the call's callId. A token count the provider reports is there, 0 included, as in cacheReadTokens: 0. One it doesn't report is absent.

A turn that waited for an approval is recorded whole, before and after the wait. The wait sits between the tool call and its result: the tool-call waited-on a wait node, which was resumed-from a resume node, and the tool-result was caused-by the resume. The resume node's actor is who decided, and its attributes have the decision, the rationale and the approval's id. A call a reviewer rejected:

{"id": "tool-hitl-gate-resume:toolu_01MAfC8Fb7e24iY12Cenoh11", "kind": "resume", "timestamp": "2026-10-05T16:26:25.504Z", "actor": "user:26fc414a-b49f-4e1f-a022-162865770178", "attributes": {"gate": "tool-call", "decision": "reject", "rationale": "Not confirmed yet", "approvalId": "12c38101-4cac-4241-8fc7-57323e4f63b9"}}

Its tool-call is there too, and its tool-result is the rejection the model read.

kindgi runs get <run-id> shows the same graph, under output.provenance, without callUsage.

From your app

Section titled “From your app”

Read the record in your app's code. Ask your coding agent: "I want to see which model calls and tool calls produced a run's answer, and what each model call cost." For reference, the code looks like this. Each script takes the run's id as its first argument:

trace.ts
import { createClient } from '@kindgi/sdk/client';
import type { RunId } from '@kindgi/sdk/types';


const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev


const record = await kindgi.provenance.get(process.argv[2] as RunId);
for (const node of record.dag.nodes) {
  const { callId } = (node.attributes ?? {}) as { callId?: string };
  const call = typeof callId === 'string' ? record.callUsage?.[callId] : undefined;
  console.log(node.kind, node.modelVersion ?? '', call ? `${call.usage.promptTokens}+${call.usage.completionTokens} tokens, ${call.costUsd} USD` : '');
}

Each prints every node, with the tokens and cost of each model call:

input
model-call anthropic/claude-sonnet-5-5 645+132 tokens, 0.0026100000000000003 USD
tool-call
tool-result
model-call anthropic/claude-sonnet-5-5 740+42 tokens, 0.0019 USD
model-output anthropic/claude-sonnet-5-5

What a turn cost

Section titled “What a turn cost”

Each model call's cost, in callUsage, is its tokens times the prices its provider was registered with, as Models describes. The whole turn's total is in the run's output, under usage. Get the run. <run-id> is its id:

Terminal window
pnpm exec kindgi runs get <run-id>
 
    "usage": {
      "steps": 2,
      "durationMs": 4971,
      "promptTokens": 1671,
      "totalCostUsd": 0.002156,
      "completionTokens": 97
    },
Excerpt

For the cost of several runs, a flow's whole run, or one customer's month, read the cost API, as Cost per run and per customer shows.

A flow's answers

Section titled “A flow's answers”

A finished flow run has a record of its own: each step that ran, why it ran, the runs it started, and which steps the run's output came from. It's written just after the run ends, and from the run's end until it's written, the read says so. <flow-run-id> is the flow run's id:

Terminal window
pnpm exec kindgi provenance get <flow-run-id>
 
Error [not-found]: No provenance record for run "27882332-2f09-48fd-a607-e44a673b6a64"
Record state: being-written

Over HTTP it's a 404 provenance-not-found whose details.recordState is being-written: ask again in a moment. If writing the record failed, recordState is failed, and details.reason says which part failed. An older runtime writes no flow records, and its 404 has no recordState.

Once it's written, the same command answers the record. This flow has three steps: a tool (parse), an agent (classify) and another tool (count):

{
  "id": "8379449c-327e-53d2-8fb9-ce358ff152ee",
  "runId": "22e085a4-103a-41a1-8667-f68796c5edb1",
  …
  "version": "1.1.0",
  "flowRef": {"id": "acme.typed-agent", "version": "0.1.0"},
  "dag": {
    "nodes": [
      {"id": "input", "kind": "input", "timestamp": "2026-10-28T11:20:21.844Z"},
      {"id": "step:parse", "kind": "step", "timestamp": "2026-10-28T11:20:21.852Z", "attributes": {"ref": {"id": "acme.parse"}, "nodeId": "parse", "status": "completed", "endedAt": "2026-10-28T11:20:22.947Z", "attempts": 1, "nodeKind": "tool", "reachedBy": ["e0"], "startedAt": "2026-10-28T11:20:21.852Z"}},
      {"id": "step:classify", "kind": "step", "timestamp": "2026-10-28T11:20:22.947Z", "attributes": {"ref": {"id": "acme.classify"}, "runs": [{"runId": "607f0079-4bba-4773-b2c1-e660ff7e7be6", "flowId": "agent.turn", "status": "completed", "provenanceId": "edbf2702-653f-4b44-b935-5134c8b092a4"}], "nodeId": "classify", "status": "completed", "endedAt": "2026-10-28T11:20:27.479Z", "attempts": 1, "nodeKind": "agent", "reachedBy": ["e1"], "startedAt": "2026-10-28T11:20:22.947Z"}},
      {"id": "step:count", "kind": "step", "timestamp": "2026-10-28T11:20:27.480Z", "attributes": {"ref": {"id": "acme.count"}, "nodeId": "count", "status": "completed", "endedAt": "2026-10-28T11:20:27.547Z", "attempts": 1, "nodeKind": "tool", "reachedBy": ["e2"], "startedAt": "2026-10-28T11:20:27.480Z"}},
      {"id": "output", "kind": "artifact", "timestamp": "2026-10-28T11:20:27.593Z"}
    ],
    "edges": [
      {"from": "step:parse", "to": "input", "kind": "caused-by"},
      {"from": "step:classify", "to": "step:parse", "kind": "caused-by"},
      {"from": "step:count", "to": "step:classify", "kind": "caused-by"},
      {"from": "output", "to": "step:count", "kind": "produced"},
      {"from": "output", "to": "step:classify", "kind": "produced"}
    ]
  },
  "stepUsage": {
    "step:classify": {"calls": 1, "costUsd": 0, "tokens": {"prompt": 20, "completion": 6, "cacheRead": 0, "cacheWrite": 0, "reasoning": 0}}
  }
}
Excerpt

Read it from the output backwards:

  • The output was produced by count and classify: the flow's output.mapping reads both. With no mapping, it's the step that fed $end.
  • count was caused-by classify, classify by parse, and parse by input, the run's input. A step's reachedBy names the edges it was reached by.
  • classify is an agent step. Its turn ran as a run of its own, listed in runs with the id of the turn's own record, provenanceId. Fetch that record by the turn's runId to see its model calls and tool calls.
Attribute What it is
nodeId, nodeKind, ref The step in the flow's definition: its id, its kind (tool, agent, loop, …) and what it runs.
status, attempts, startedAt, endedAt How it ran.
reachedBy The edges it was reached by.
runs The runs it started: an agent step's turn, or a sub-flow's run.
reusedFrom The earlier results it reused, when the runtime records them.

stepUsage holds each step's model usage from the cost ledger, keyed by the step's node id in the record (step:classify): its calls, their cost in USD and their tokens, over every run it started and the runs under them. A step that made no model calls has no entry. Here the turn ran on dev-echo, which costs nothing.

The record holds ids, kinds, statuses, times and codes, never a value. What a step read or returned stays in the run's journal: kindgi runs journal <flow-run-id>.

A step in a loop has a node for each iteration it ran. A foreach over two grievances, with one agent step in its body:

      {"id": "step:each", "kind": "step", "timestamp": "2026-10-28T11:20:32.789Z", "attributes": {"nodeId": "each", "status": "completed", "nodeKind": "loop", "iterations": {"count": 2, "stopReason": "array-exhausted"}, …}},
      {"id": "step:classify-each@each#0", "kind": "step", "timestamp": "2026-10-28T11:20:32.816Z", "attributes": {"ref": {"id": "acme.classify"}, "runs": [{…}], "nodeId": "classify-each", "status": "completed", "nodeKind": "agent", …}},
      {"id": "step:classify-each@each#1", "kind": "step", "timestamp": "2026-10-28T11:20:34.562Z", "attributes": {"ref": {"id": "acme.classify"}, "runs": [{…}], "nodeId": "classify-each", "status": "completed", "nodeKind": "agent", …}},
…
      {"from": "step:classify-each@each#0", "to": "step:each", "kind": "invoked"},
      {"from": "step:classify-each@each#1", "to": "step:each", "kind": "invoked"},
Excerpt

Each iteration's step is step:<step>@<loop>#<iteration>, invoked by its loop, with its own entry in stepUsage. In a nested loop, the inner loop's iteration comes first: step:<step>@<inner>#<j>/<outer>#<i>. A long loop keeps whole iterations from both ends, at most 25 from each. The iterations between fold into one node, step:<loop>#folded, whose folded attribute has their range (from, to), their count and byStatus, and how many runs their steps started (runs). Its stepUsage entry sums theirs.

A record holds at most 5,000 nodes and 4 MiB. A larger one is cut from the middle: a truncated node says what was left out, and a step cut with it has no stepUsage entry. An id over 256 characters takes a digest form, step:~<hex>, and keeps its start in idHead.

Find records

Section titled “Find records”

kindgi provenance list lists records, newest first, without their graphs. --created-after=<time> keeps the newer ones, --run=<run-id> one run's, --agent=<agent-id> one agent's turns at any version, --flow=<flow-id> one flow's runs at any version, and --project=<id> or --org=<id> one project's or one org's. --table prints a row per record, and a flow run's row names its flow:

Terminal window
pnpm exec kindgi provenance list --created-after=2026-10-06T17:00:00Z --limit=3 --table
 
RUN                                   CREATED                   FLOW  SIGNED
────────────────────────────────────  ────────────────────────  ────  ──────
e4cb91b4-d0b1-499a-a282-c75b83c7693c  2026-10-06T17:12:02.090Z        no

One flow's records:

Terminal window
pnpm exec kindgi provenance list --flow=acme.typed-agent --limit=3 --table
 
RUN                                   CREATED                   FLOW                    SIGNED
────────────────────────────────────  ────────────────────────  ──────────────────────  ──────
706d9ef4-6b5f-44a0-8dd5-54c6b488071c  2026-10-28T11:20:44.208Z  acme.typed-agent@0.1.0  no
22e085a4-103a-41a1-8667-f68796c5edb1  2026-10-28T11:20:27.593Z  acme.typed-agent@0.1.0  no

Over HTTP it's GET /v1/provenance, filtered by runId, agentId, flowId or createdAfter:

Terminal window
curl "$KINDGI_API_URL/v1/provenance?createdAfter=2026-10-05T16:25:00Z&limit=3" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN")
{"data":[{"id":"e87e13ae-472f-404b-8dfe-94ec738980cd","runId":"f91ff793-b029-48e8-9ce4-f9fa70bd0c40","tenantId":"9126df1e-094b-496d-9ea8-7daab93c3f4f","version":"1.0.0","createdAt":"2026-10-05T16:26:30.533Z","signed":false,"projectId":"13b7cfca-8b77-4983-858f-149c16afe56d"},…]}
Excerpt

A flow run's record has its flowRef, and a page filtered by flowId says flowId back. A turn's record names no flow, so flowId=agent.turn lists nothing. A runtime that doesn't filter by flow leaves flowId out of its page, and --flow then stops with an error rather than print every record as the flow's.

Each record carries its run's projectId. scopeKind=project&scopeId=<id> lists one project's records, and scopeKind=org&scopeId=<id> every project's in an org. The clients call it scope. A record with no project is listed only without a scope, and one whose run doesn't name its agent isn't matched by --agent.

signed is false: the records themselves aren't signed as they're written. To hand a run's provenance to someone, export it signed by the deployment, with kindgi provenance export <run-id> > provenance.json. Check it with kindgi exports verify provenance.json, as Export signed evidence shows.

Provenance in the HTTP API has every operation.

Next

Section titled “Next”