# Trace an answer and its cost > See which model calls and tool calls produced an agent's answer, and what each model call cost. This guide reads a turn's provenance record, to see where an answer came from and what it cost. Every agent turn leaves a **provenance record**: a graph of the turn, from the user's message through each model call and tool call to the answer. It names the model of each call and the version of each tool, and, beside the graph, each call's tokens and cost. Use it to answer "where did this answer come from?" and "what did it cost?". ## Before you start - **A runtime,** such as `kindgi dev`, with Kindgi 0.1's CLI. - **The id of an agent turn's run.** `kindgi runs list` shows it. - **`KINDGI_API_URL` and `KINDGI_API_TOKEN`** set in your shell, for the HTTP requests. With `kindgi dev`, they're the URL and token it prints. They're also in the pack's `.kindgirc.json`. ## Read a turn's record Fetch it by the run's id: - **TypeScript** ```sh pnpm exec kindgi provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` - **Python** ```sh uv run kindgi provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` - **Java** ```sh ./kindgiw provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` - **Scala** ```sh ./kindgiw provenance get a64ce71c-9cfc-4263-87b1-b5562f09079c ``` Over HTTP it's `GET /v1/provenance/`: ```sh curl "$KINDGI_API_URL/v1/provenance/a64ce71c-9cfc-4263-87b1-b5562f09079c" \ -H "Authorization: Bearer $KINDGI_API_TOKEN" ``` Both answer the record: ```json excerpt { "id": "9f0f7385-5270-484d-9ee4-13716c4edb6e", "runId": "a64ce71c-9cfc-4263-87b1-b5562f09079c", … "dag": { "nodes": [ {"id": "input:0", "kind": "input", "timestamp": "2026-10-08T15:13:29.826Z"}, {"id": "model-call:1", "kind": "model-call", "timestamp": "2026-10-08T15:13:31.345Z", "modelVersion": "anthropic/claude-sonnet-5-5", "attributes": {"step": 1, "model": "claude-sonnet-5-5", "callId": "975d170b-d14f-4fbf-b662-85bcf26b9431", "providerId": "anthropic", "finishReason": "tool-use"}}, {"id": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "tool-call", "timestamp": "2026-10-08T15:13:31.369Z", "attributes": {"toolId": "acme.echo", "toolVersion": "0.1.0", "invocationId": "toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "toolVersionRange": "0.1.0"}}, {"id": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "tool-result", "timestamp": "2026-10-08T15:13:31.369Z"}, {"id": "model-call:2", "kind": "model-call", "timestamp": "2026-10-08T15:13:32.418Z", "modelVersion": "anthropic/claude-sonnet-5-5", "attributes": {"step": 2, "model": "claude-sonnet-5-5", "callId": "99929d2e-05e2-4452-853e-b6c6f0deed6e", "providerId": "anthropic", "finishReason": "stop"}}, {"id": "model-output:2", "kind": "model-output", "timestamp": "2026-10-08T15:13:32.442Z", "modelVersion": "anthropic/claude-sonnet-5-5"} ], "edges": [ {"from": "model-call:1", "to": "input:0", "kind": "caused-by"}, {"from": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "to": "model-call:1", "kind": "invoked"}, {"from": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "to": "tool-call:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "produced"}, {"from": "model-call:2", "to": "input:0", "kind": "caused-by"}, {"from": "model-call:2", "to": "tool-result:toolu_01LiLE7Xk9BK16Hkr2tLBEq4", "kind": "influenced-by"}, {"from": "model-output:2", "to": "model-call:2", "kind": "produced"} ] }, "callUsage": { "975d170b-d14f-4fbf-b662-85bcf26b9431": {"usage": {"promptTokens": 645, "completionTokens": 132, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "costUsd": 0.0026100000000000003, "durationMs": 1460, "servedModel": "claude-sonnet-5-5"}, "99929d2e-05e2-4452-853e-b6c6f0deed6e": {"usage": {"promptTokens": 740, "completionTokens": 42, "cacheReadTokens": 0, "cacheWriteTokens": 0}, "costUsd": 0.0019, "durationMs": 1012, "servedModel": "claude-sonnet-5-5"} } } ``` Read it from the answer backwards: - The answer, `model-output:2`, was produced by the second model call. - That call answered the user's message: it's `caused-by` `input:0`. - It also read the echo tool's result: it's `influenced-by` that `tool-result`. - The tool call that produced the result was `invoked` by the first model call. | Node | What it is | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `input` | The message the turn answered. | | `model-call` | One call to a model: `modelVersion` is `provider/model`; `attributes` has its `step`, `model`, `providerId`, why it stopped (`finishReason`) and its `callId`. | | `tool-call`, `tool-result` | A tool the model called, with the exact tool version that ran, and what it returned. | | `model-output` | The answer. | | `wait`, `resume` | A tool call that waited for an approval: the wait, and the decision that ended it. | Edges point from an effect to its cause: `caused-by`, `invoked`, `produced`, and `influenced-by` from a model call to each tool result it read. `callUsage` holds each model call's tokens, cost in USD, duration and the model version that answered, `servedModel`, by the call's `callId`. A token count the provider reports is there, 0 included, as in `cacheReadTokens: 0`. One it doesn't report is absent. A turn that waited for an approval is recorded whole, before and after the wait. The wait sits between the tool call and its result: the `tool-call` `waited-on` a `wait` node, which was `resumed-from` a `resume` node, and the `tool-result` was `caused-by` the `resume`. The `resume` node's `actor` is who decided, and its `attributes` have the decision, the rationale and the approval's id. A call a reviewer rejected: ```json {"id": "tool-hitl-gate-resume:toolu_01MAfC8Fb7e24iY12Cenoh11", "kind": "resume", "timestamp": "2026-10-05T16:26:25.504Z", "actor": "user:26fc414a-b49f-4e1f-a022-162865770178", "attributes": {"gate": "tool-call", "decision": "reject", "rationale": "Not confirmed yet", "approvalId": "12c38101-4cac-4241-8fc7-57323e4f63b9"}} ``` Its `tool-call` is there too, and its `tool-result` is the rejection the model read. `kindgi runs get ` shows the same graph, under `output.provenance`, without `callUsage`. ## From your app Read the record in your app's code. Ask your coding agent: "I want to see which model calls and tool calls produced a run's answer, and what each model call cost." For reference, the code looks like this. Each script takes the run's id as its first argument: - **TypeScript** `trace.ts` ```ts import { createClient } from '@kindgi/sdk/client'; import type { RunId } from '@kindgi/sdk/types'; const kindgi = createClient(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev const record = await kindgi.provenance.get(process.argv[2] as RunId); for (const node of record.dag.nodes) { const { callId } = (node.attributes ?? {}) as { callId?: string }; const call = typeof callId === 'string' ? record.callUsage?.[callId] : undefined; console.log(node.kind, node.modelVersion ?? '', call ? `${call.usage.promptTokens}+${call.usage.completionTokens} tokens, ${call.costUsd} USD` : ''); } ``` - **Python** `trace.py` ```python import sys from kindgi.client import Kindgi kindgi = Kindgi() # KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev record = kindgi.provenance.get(sys.argv[1]) for node in record.dag.nodes: call_id = (node.attributes or {}).get("callId") call = (record.call_usage or {}).get(call_id) if call_id else None usage = f"{call.usage.prompt_tokens}+{call.usage.completion_tokens} tokens, {call.cost_usd} USD" if call else "" print(node.kind, node.model_version or "", usage) ``` - **Java** `src/main/java/com/acme/app/Trace.java` ```java package com.acme.app; import com.kindgi.client.Kindgi; import com.kindgi.client.models.ProvenanceRecord; import java.util.Map; import java.util.Objects; import java.util.UUID; public final class Trace { public static void main(String[] args) { Kindgi kindgi = Kindgi.create(); // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev ProvenanceRecord record = kindgi.provenance().get(UUID.fromString(args[0])); Map usage = Objects.requireNonNullElse(record.callUsage(), Map.of()); for (ProvenanceRecord.Dag.NodesItem node : record.dag().nodes()) { Object callId = node.attributes() == null ? null : node.attributes().get("callId"); ProvenanceRecord.CallUsageValue call = callId instanceof String id ? usage.get(id) : null; System.out.println(node.kind() + " " + Objects.toString(node.modelVersion(), "") + " " + (call == null ? "" : call.usage().promptTokens() + "+" + call.usage().completionTokens() + " tokens, " + call.costUsd() + " USD")); } } } ``` - **Scala** `src/main/scala/acme/app/Trace.scala` ```scala package acme.app import com.kindgi.client.Kindgi import com.kindgi.client.models.ProvenanceRecord import java.util.UUID import scala.jdk.CollectionConverters._ object Trace { def main(args: Array[String]): Unit = { val kindgi = Kindgi.create() // KINDGI_API_URL, KINDGI_API_TOKEN, or the running kindgi dev val record = kindgi.provenance().get(UUID.fromString(args(0))) val usage: Map[String, ProvenanceRecord.CallUsageValue] = Option(record.callUsage).map(_.asScala.toMap).getOrElse(Map.empty) for (node <- record.dag.nodes.asScala) { val callId = Option(node.attributes).flatMap(a => Option(a.get("callId"))).collect { case id: String => id } val call = callId.flatMap(usage.get) val cost = call.map(c => s"${c.usage.promptTokens}+${c.usage.completionTokens} tokens, ${c.costUsd} USD").getOrElse("") println(s"${node.kind} ${Option(node.modelVersion).getOrElse("")} $cost") } } } ``` Each prints every node, with the tokens and cost of each model call: ```text input model-call anthropic/claude-sonnet-5-5 645+132 tokens, 0.0026100000000000003 USD tool-call tool-result model-call anthropic/claude-sonnet-5-5 740+42 tokens, 0.0019 USD model-output anthropic/claude-sonnet-5-5 ``` ## What a turn cost Each model call's cost, in `callUsage`, is its tokens times the prices its provider was registered with, as [Models](https://docs.kindgi.com/v0.1.8/guides/models.md) describes. The whole turn's total is in the run's output, under `usage`. Get the run. `` is its id: - **TypeScript** ```sh pnpm exec kindgi runs get ``` - **Python** ```sh uv run kindgi runs get ``` - **Java** ```sh ./kindgiw runs get ``` - **Scala** ```sh ./kindgiw runs get ``` ```json excerpt "usage": { "steps": 2, "durationMs": 4971, "promptTokens": 1671, "totalCostUsd": 0.002156, "completionTokens": 97 }, ``` For the cost of several runs, a flow's whole run, or one customer's month, read the cost API, as [Cost per run and per customer](https://docs.kindgi.com/v0.1.8/guides/observability/cost-per-run.md) shows. ## A flow's answers A flow run has no record of its own: ```json {"error":{"code":"provenance-not-found","message":"No provenance record for run \"41027166-71ec-4bd6-b516-cc47c0787da4\"","details":{"runId":"41027166-71ec-4bd6-b516-cc47c0787da4"},"requestId":"req-997f000a-22f6-470a-b52b-acdd21d6e228"}} ``` Each agent step is a run of its own, with its own record. The step's output in the flow's journal has that run's id. `` is the flow run's id: - **TypeScript** ```sh pnpm exec kindgi runs journal ``` - **Python** ```sh uv run kindgi runs journal ``` - **Java** ```sh ./kindgiw runs journal ``` - **Scala** ```sh ./kindgiw runs journal ``` ```json excerpt { "sequence": 9, "kind": "step.completed", "nodeId": "respond", "payload": { "output": { "text": "⚠ dev-echo isn't a real model: …\n\nTool responded: …", "runId": "4bdae2f9-ff67-48a3-9549-49028d290cc1", … ``` Fetch the record for that run: `kindgi provenance get 4bdae2f9-…`. ## Find records `kindgi provenance list` lists records, newest first, without their graphs. `--created-after=