# Trace an answer and its cost > See which model calls and tool calls produced an agent's answer and what each model call cost, and which steps produced a flow run's output. This guide reads a turn's provenance record, to see where an answer came from and what it cost. Every agent turn leaves a **provenance record**: a graph of the turn, from the user's message through each model call and tool call to the answer. It names the model of each call and the version of each tool, and, beside the graph, each call's tokens and cost. Use it to answer "where did this answer come from?" and "what did it cost?". A finished flow run has a record too, of its steps and the runs they started: [A flow's answers](https://docs.kindgi.com/v0.5.0/guides/observability/trace-an-answer.md#a-flows-answers) reads it. ## Before you start - **A runtime,** such as `kindgi dev`, with Kindgi 0.2's CLI. - **The id of an agent turn's run.** `kindgi runs list` shows it. - **`KINDGI_API_URL` and `KINDGI_API_TOKEN`** set in your shell, for the HTTP requests. With `kindgi dev`, they're `apiUrl` and `token` in the pack's `.kindgirc.json`. ## Read a turn's record Fetch it by the run's id: - **TypeScript** ```sh pnpm exec kindgi provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` - **Python** ```sh uv run kindgi provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` - **Java** ```sh ./kindgiw provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` - **Scala** ```sh ./kindgiw provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` Over HTTP it's `GET /v1/provenance/`: ```sh curl "$KINDGI_API_URL/v1/provenance/a64ce71c-9cfc-4263-87b1-b5562f09079c" \ -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") ``` Both answer the record: ```json excerpt { "id": "9f0f7385-5270-484d-9ee4-13716c4edb6e", "runId": "a64ce71c-9cfc-4263-87b1-b5562f09079c", … "dag": { "nodes": [ {"id": "input:0", "kind": "input", "timestamp": "2026-10-08T15:13:29.826Z"}, {"id": "model-call:1", "kind": "model-call", "timestamp": "2026-10-08T15:13:31.345Z", "modelVersion": "anthropic/claude-sonnet-5-5", "attributes": {"step": 1, "model": "claude-sonnet-5-5", "callId": "975d170b-d14f-4fbf-b662-85bcf26b9431", "providerId": "anthropic", "finishReason": "tool-use"}}, {"id": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "tool-call", "timestamp": "2026-10-08T15:13:31.369Z", "attributes": {"toolId": "acme.echo", "toolVersion": "0.1.0", "invocationId": "toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "toolVersionRange": "0.1.0"}}, {"id": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "tool-result", "timestamp": "2026-10-08T15:13:31.369Z"}, {"id": "model-call:2", "kind": "model-call", "timestamp": "2026-10-08T15:13:32.418Z", "modelVersion": "anthropic/claude-sonnet-5-5", "attributes": {"step": 2, "model": "claude-sonnet-5-5", "callId": "99929d2e-05e2-4452-853e-b6c6f0deed6e", "providerId": "anthropic", "finishReason": "stop"}}, {"id": "model-output:2", "kind": "model-output", "timestamp": "2026-10-08T15:13:32.442Z", "modelVersion": "anthropic/claude-sonnet-5-5"} ], "edges": [ {"from": "model-call:1", "to": "input:0", "kind": "caused-by"}, {"from": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "to": "model-call:1", "kind": "invoked"}, {"from": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "to": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "produced"}, {"from": "model-call:2", "to": "input:0", "kind": "caused-by"}, {"from": "model-call:2", "to": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "influenced-by"}, {"from": "model-output:2", "to": "model-call:2", "kind": "produced"} ] }, "callUsage": { "975d170b-d14f-4fbf-b662-85bcf26b9431": {"usage": {"promptTokens": 645, "completionTokens": 132, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "costUsd": 0.0026100000000000003, "durationMs": 1460, "servedModel": "claude-sonnet-5-5"}, "99929d2e-05e2-4452-853e-b6c6f0deed6e": {"usage": {"promptTokens": 740, "completionTokens": 42, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "costUsd": 0.0019, "durationMs": 1012, "servedModel": "claude-sonnet-5-5"} } } ``` Read it from the answer backwards: - The answer, `model-output:2`, was produced by the second model call. - That call answered the user's message: it's `caused-by` `input:0`. - It also read the echo tool's result: it's `influenced-by` that `tool-result`. - The tool call that produced the result was `invoked` by the first model call. | Node | What it is | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input` | The message the turn answered. | | `model-call` | One call to a model: `modelVersion` is `provider/model`; `attributes` has its `step`, `model`, `providerId`, why it stopped (`finishReason`), its `callId`, and its `thinking` when it asked for or sent a level: `asked`, the level asked for, and `level`, the level sent ([Temperature and thinking](https://docs.kindgi.com/v0.5.0/guides/models.md#temperature-and-thinking)). | | `tool-call`, `tool-result` | A tool the model called, with the exact tool version that ran, and what it returned. | | `model-output` | The answer. | | `wait`, `resume` | A tool call that waited for an approval: the wait, and the decision that ended it. | Edges point from an effect to its cause: `caused-by`, `invoked`, `produced`, and `influenced-by` from a model call to each tool result it read. `callUsage` holds each model call's tokens, cost in USD, duration and the model version that answered, `servedModel`, by the call's `callId`. A token count the provider reports is there, 0 included, as in `cacheReadTokens: 0`. One it doesn't report is absent. A turn that waited for an approval is recorded whole, before and after the wait. The wait sits between the tool call and its result: the `tool-call` `waited-on` a `wait` node, which was `resumed-from` a `resume` node, and the `tool-result` was `caused-by` the `resume`. The `resume` node's `actor` is who decided, and its `attributes` have the decision, the rationale and the approval's id. A call a reviewer rejected: ```json {"id": "tool-hitl-gate-resume:toolu_01MAfC8Fb7e24iY12Cenoh11", "kind": "resume", "timestamp": "2026-10-05T16:26:25.504Z", "actor": "user:26fc414a-b49f-4e1f-a022-162865770178", "attributes": {"gate": "tool-call", "decision": "reject", "rationale": "Not confirmed yet", "approvalId": "12c38101-4cac-4241-8fc7-57323e4f63b9"}} ``` Its `tool-call` is there too, and its `tool-result` is the rejection the model read. `kindgi runs get ` shows the same graph, under `output.provenance`, without `callUsage`. ## From your app Read the record in your app's code. Ask your coding agent: "I want to see which model calls and tool calls produced a run's answer, and what each model call cost." For reference, the code looks like this. Each script takes the run's id as its first argument: - **TypeScript** `trace.ts` ```ts import { createClient } from '@kindgi/sdk/client'; import type { RunId } from '@kindgi/sdk/types'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const record = await kindgi.provenance.get(process.argv[2] as RunId); for (const node of record.dag.nodes) { const { callId } = (node.attributes ?? {}) as { callId?: string }; const call = typeof callId === 'string' ? record.callUsage?.[callId] : undefined; console.log(node.kind, node.modelVersion ?? '', call ? `${call.usage.promptTokens}+${call.usage.completionTokens} tokens, ${call.costUsd} USD` : ''); } ``` - **Python** `trace.py` ```python import sys from kindgi.client import Kindgi kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev record = kindgi.provenance.get(sys.argv[1]) for node in record.dag.nodes: call_id = (node.attributes or {}).get("callId") call = (record.call_usage or {}).get(call_id) if call_id else None usage = f"{call.usage.prompt_tokens}+{call.usage.completion_tokens} tokens, {call.cost_usd} USD" if call else "" print(node.kind, node.model_version or "", usage) ``` - **Java** `src/main/java/com/acme/app/Trace.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.ProvenanceRecord; import java.util.Map; import java.util.Objects; import java.util.UUID; public final class Trace { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev ProvenanceRecord record = kindgi.provenance().get(UUID.fromString(args[0])); Map usage = Objects.requireNonNullElse(record.callUsage(), Map.of()); for (ProvenanceRecord.Dag.NodesItem node : record.dag().nodes()) { Object callId = node.attributes() == null ? null : node.attributes().get("callId"); ProvenanceRecord.CallUsageValue call = callId instanceof String id ? usage.get(id) : null; System.out.println(node.kind() + " " + Objects.toString(node.modelVersion(), "") + " " + (call == null ? "" : call.usage().promptTokens() + "+" + call.usage().completionTokens() + " tokens, " + call.costUsd() + " USD")); } } } ``` - **Scala** `src/main/scala/acme/app/Trace.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.models.ProvenanceRecord import java.util.UUID import scala.jdk.CollectionConverters._ object Trace { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val record = kindgi.provenance().get(UUID.fromString(args(0))) val usage: Map[String, ProvenanceRecord.CallUsageValue] = Option(record.callUsage).map(_.asScala.toMap).getOrElse(Map.empty) for (node <- record.dag.nodes.asScala) { val callId = Option(node.attributes).flatMap(a => Option(a.get("callId"))).collect { case id: String => id } val call = callId.flatMap(usage.get) val cost = call.map(c => s"${c.usage.promptTokens}+${c.usage.completionTokens} tokens, ${c.costUsd} USD").getOrElse("") println(s"${node.kind} ${Option(node.modelVersion).getOrElse("")} $cost") } } } ``` Each prints every node, with the tokens and cost of each model call: ```text input model-call anthropic/claude-sonnet-5-5 645+132 tokens, 0.0026100000000000003 USD tool-call tool-result model-call anthropic/claude-sonnet-5-5 740+42 tokens, 0.0019 USD model-output anthropic/claude-sonnet-5-5 ``` ## What a turn cost Each model call's cost, in `callUsage`, is its tokens times the prices its provider was registered with, as [Models](https://docs.kindgi.com/v0.5.0/guides/models.md) describes. The whole turn's total is in the run's output, under `usage`. Get the run. `` is its id: - **TypeScript** ```sh pnpm exec kindgi runs get ``` - **Python** ```sh uv run kindgi runs get ``` - **Java** ```sh ./kindgiw runs get ``` - **Scala** ```sh ./kindgiw runs get ``` ```json excerpt "usage": { "steps": 2, "durationMs": 4971, "promptTokens": 1671, "totalCostUsd": 0.002156, "completionTokens": 97 }, ``` For the cost of several runs, a flow's whole run, or one customer's month, read the cost API, as [Cost per run and per customer](https://docs.kindgi.com/v0.5.0/guides/observability/cost-per-run.md) shows. ## A flow's answers A finished flow run has a record of its own: each step that ran, why it ran, the runs it started, and which steps the run's output came from. It's written just after the run ends, and from the run's end until it's written, the read says so. `` is the flow run's id: - **TypeScript** ```sh pnpm exec kindgi provenance get ``` - **Python** ```sh uv run kindgi provenance get ``` - **Java** ```sh ./kindgiw provenance get ``` - **Scala** ```sh ./kindgiw provenance get ``` ```text Error [not-found]: No provenance record for run "27882332-2f09-48fd-a607-e44a673b6a64" Record state: being-written ``` Over HTTP it's a `404 provenance-not-found` whose `details.recordState` is `being-written`: ask again in a moment. If writing the record failed, `recordState` is `failed`, and `details.reason` says which part failed. An older runtime writes no flow records, and its 404 has no `recordState`. Once it's written, the same command answers the record. This flow has three steps: a tool (`parse`), an agent (`classify`) and another tool (`count`): ```json excerpt { "id": "8379449c-327e-53d2-8fb9-ce358ff152ee", "runId": "22e085a4-103a-41a1-8667-f68796c5edb1", … "version": "1.1.0", "flowRef": {"id": "acme.typed-agent", "version": "0.1.0"}, "dag": { "nodes": [ {"id": "input", "kind": "input", "timestamp": "2026-10-28T11:20:21.844Z"}, {"id": "step:parse", "kind": "step", "timestamp": "2026-10-28T11:20:21.852Z", "attributes": {"ref": {"id": "acme.parse"}, "nodeId": "parse", "status": "completed", "endedAt": "2026-10-28T11:20:22.947Z", "attempts": 1, "nodeKind": "tool", "reachedBy": ["e0"], "startedAt": "2026-10-28T11:20:21.852Z"}}, {"id": "step:classify", "kind": "step", "timestamp": "2026-10-28T11:20:22.947Z", "attributes": {"ref": {"id": "acme.classify"}, "runs": [{"runId": "607f0079-4bba-4773-b2c1-e660ff7e7be6", "flowId": "agent.turn", "status": "completed", "provenanceId": "edbf2702-653f-4b44-b935-5134c8b092a4"}], "nodeId": "classify", "status": "completed", "endedAt": "2026-10-28T11:20:27.479Z", "attempts": 1, "nodeKind": "agent", "reachedBy": ["e1"], "startedAt": "2026-10-28T11:20:22.947Z"}}, {"id": "step:count", "kind": "step", "timestamp": "2026-10-28T11:20:27.480Z", "attributes": {"ref": {"id": "acme.count"}, "nodeId": "count", "status": "completed", "endedAt": "2026-10-28T11:20:27.547Z", "attempts": 1, "nodeKind": "tool", "reachedBy": ["e2"], "startedAt": "2026-10-28T11:20:27.480Z"}}, {"id": "output", "kind": "artifact", "timestamp": "2026-10-28T11:20:27.593Z"} ], "edges": [ {"from": "step:parse", "to": "input", "kind": "caused-by"}, {"from": "step:classify", "to": "step:parse", "kind": "caused-by"}, {"from": "step:count", "to": "step:classify", "kind": "caused-by"}, {"from": "output", "to": "step:count", "kind": "produced"}, {"from": "output", "to": "step:classify", "kind": "produced"} ] }, "stepUsage": { "step:classify": {"calls": 1, "costUsd": 0, "tokens": {"prompt": 20, "completion": 6, "cacheRead": 0, "cacheWrite": 0, "reasoning": 0}} } } ``` Read it from the output backwards: - The output was `produced` by `count` and `classify`: the flow's `output.mapping` reads both. With no mapping, it's the step that fed `$end`. - `count` was `caused-by` `classify`, `classify` by `parse`, and `parse` by `input`, the run's input. A step's `reachedBy` names the edges it was reached by. - `classify` is an agent step. Its turn ran as a run of its own, listed in `runs` with the id of the turn's own record, `provenanceId`. Fetch that record by the turn's `runId` to see its model calls and tool calls. | Attribute | What it is | | -------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `nodeId`, `nodeKind`, `ref` | The step in the flow's definition: its id, its kind (`tool`, `agent`, `loop`, …) and what it runs. | | `status`, `attempts`, `startedAt`, `endedAt` | How it ran. | | `reachedBy` | The edges it was reached by. | | `runs` | The runs it started: an agent step's turn, or a sub-flow's run. | | `reusedFrom` | The earlier results it reused, when the runtime records them. | `stepUsage` holds each step's model usage from the cost ledger, keyed by the step's node id in the record (`step:classify`): its calls, their cost in USD and their tokens, over every run it started and the runs under them. A step that made no model calls has no entry. Here the turn ran on dev-echo, which costs nothing. The record holds ids, kinds, statuses, times and codes, never a value. What a step read or returned stays in the run's journal: `kindgi runs journal `. A step in a loop has a node for each iteration it ran. A foreach over two grievances, with one agent step in its body: ```json excerpt {"id": "step:each", "kind": "step", "timestamp": "2026-10-28T11:20:32.789Z", "attributes": {"nodeId": "each", "status": "completed", "nodeKind": "loop", "iterations": {"count": 2, "stopReason": "array-exhausted"}, …}}, {"id": "step:classify-each@each#0", "kind": "step", "timestamp": "2026-10-28T11:20:32.816Z", "attributes": {"ref": {"id": "acme.classify"}, "runs": [{…}], "nodeId": "classify-each", "status": "completed", "nodeKind": "agent", …}}, {"id": "step:classify-each@each#1", "kind": "step", "timestamp": "2026-10-28T11:20:34.562Z", "attributes": {"ref": {"id": "acme.classify"}, "runs": [{…}], "nodeId": "classify-each", "status": "completed", "nodeKind": "agent", …}}, … {"from": "step:classify-each@each#0", "to": "step:each", "kind": "invoked"}, {"from": "step:classify-each@each#1", "to": "step:each", "kind": "invoked"}, ``` Each iteration's step is `step:@#`, `invoked` by its loop, with its own entry in `stepUsage`. In a nested loop, the inner loop's iteration comes first: `step:@#/#`. A long loop keeps whole iterations from both ends, at most 25 from each. The iterations between fold into one node, `step:#folded`, whose `folded` attribute has their range (`from`, `to`), their `count` and `byStatus`, and how many runs their steps started (`runs`). Its `stepUsage` entry sums theirs. A record holds at most 5,000 nodes and 4 MiB. A larger one is cut from the middle: a `truncated` node says what was left out, and a step cut with it has no `stepUsage` entry. An id over 256 characters takes a digest form, `step:~`, and keeps its start in `idHead`. ## Find records `kindgi provenance list` lists records, newest first, without their graphs. `--created-after=