Skip to content
Kindgi

Retry a failing step

View as Markdown

This guide retries a flow step that fails now and then, with backoff, and gives it a time limit.

A step that calls something flaky can be retried before its failure fails the run. The retry policy goes on the edge into the step.

Before you start

Section titled “Before you start”
  • A pack on a runtime such as kindgi dev, with Kindgi 0.2's CLI.
  • A step that's safe to run twice, as the caution in The policy explains.

Retry a step

Section titled “Retry a step”

Ask your coding agent: "I want a flow that reserves an order's stock at the warehouse, and tries again while the warehouse is busy." Kindgi's skills in your pack tell it how (How it knows Kindgi). For reference, the tool and the flow look like this. acme.reserve-stock calls a warehouse that is busy for the first two calls per order, and the flow's edge into the step retries it:

tools/reserve-stock/index.ts
import { defineTool } from '@kindgi/sdk/define';
import type { ToolId } from '@kindgi/sdk/types';
import { z } from 'zod';


// The warehouse API is busy for the first two calls per order.
const calls = new Map<string, number>();


const defined = defineTool({
  id: 'acme.reserve-stock' as ToolId,
  description: 'Reserves the stock for an order at the warehouse.',
  version: '0.1.0',
  input: z.object({ orderId: z.string() }),
  output: z.object({ orderId: z.string(), reserved: z.boolean(), attempt: z.number().int() }),
  effects: [],
  handler: async ({ orderId }) => {
    const attempt = (calls.get(orderId) ?? 0) + 1;
    calls.set(orderId, attempt);
    if (attempt < 3) throw new Error('Warehouse busy (503)');
    return { orderId, reserved: true, attempt };
  },
});


if (defined.kind === 'err') throw new Error(defined.error.message);


export default defined.value;
flows/reserve-order/index.ts
import { defineFlow } from '@kindgi/sdk/define';


const defined = defineFlow({
  id: 'acme.reserve-order',
  version: '0.1.0',
  name: 'Reserve an order',
  description: 'Reserves the stock for an order, retrying while the warehouse is busy.',
  nodes: [{ id: 'reserve', kind: 'tool', ref: 'acme.reserve-stock' }],
  edges: [
    {
      id: 'e1',
      from: '$start',
      to: 'reserve',
      policy: {
        retry: { maxAttempts: 4, delayMs: 500, backoff: 'exponential', maxDelayMs: 5_000 },
        timeoutMs: 10_000,
      },
    },
    { id: 'e2', from: 'reserve', to: '$end' },
  ],
});


if (defined.kind === 'err') throw new Error(defined.error.message);


export default defined.value;

Run it:

Terminal window
pnpm exec kindgi runs start --flow=acme.reserve-order --input='{"orderId":"A-200"}'

--flow is the flow, and --input its input. It prints the run, which completed at the third attempt:

{
  …
  "status": "completed",
  …
  "output": {
    "attempt": 3,
    "orderId": "A-200",
    "reserved": true
  },
  …
}
Excerpt

The journal, which kindgi runs journal <run-id> prints, has each retry. It shows the error that caused it and the wait before the next attempt:

{"sequence": 2, "kind": "step.started", "nodeId": "reserve", "payload": {"input": {"orderId": "A-200"}}, …}
{"sequence": 3, "kind": "step.retry-scheduled", "nodeId": "reserve", "payload": {"nodeId": "reserve", "attempt": 1, "nextDelayMs": 500, "previousError": "handler-error: Tool \"acme.reserve-stock\" handler threw: … Warehouse busy (503)"}, …}
{"sequence": 4, "kind": "step.retry-scheduled", "nodeId": "reserve", "payload": {"nodeId": "reserve", "attempt": 2, "nextDelayMs": 1000, "previousError": "handler-error: Tool \"acme.reserve-stock\" handler threw: … Warehouse busy (503)"}, …}
{"sequence": 5, "kind": "step.completed", "nodeId": "reserve", "payload": {"output": {"attempt": 3, "orderId": "A-200", "reserved": true}}, …}
Excerpt

The policy

Section titled “The policy”

policy.retry:

  • maxAttempts: attempts in all, the first one included, at most 10.
  • delayMs: the wait before the first retry, 0 by default.
  • backoff: how the wait grows.
    • fixed, the default: the same wait each time.
    • linear: delayMs × the retry's number.
    • exponential: doubling each time, 500, 1000, 2000 ms here. It needs maxDelayMs, the longest wait.

When the last attempt fails, the step fails and so does the run, with the last error. A step left for the run's next resume uses one of its attempts too.

policy.timeoutMs limits each attempt. An attempt that takes longer is stopped and counts as a failed attempt. Without retries, the step fails with reason: "timeout":

{"sequence": 3, "kind": "step.failed", "nodeId": "reserve", "payload": {"reason": "timeout", "limitMs": 5, "message": "Handler for node \"reserve\" exceeded timeout of 5ms", "elapsedMs": 68}, …}
Excerpt

That entry came from timeoutMs: 5, set that low to show it.

A step whose read failed

Section titled “A step whose read failed”

An agent's step first reads what its turn needs from the runtime's database: the agent's registration and live pin, the versions shipped into its project, the gate policy, and its turn's snapshot and journal. When one of those reads fails, such as when the database doesn't answer, the step doesn't fail. It's left for the run's next resume:

{"sequence": 3, "kind": "step.retry-scheduled", "nodeId": "draft", "payload": {"nodeId": "draft", "attempt": 1, "nextDelayMs": 0, "previousError": "run-version-unreadable: Could not read …", "left": true}, …}
Excerpt
  • Its step.retry-scheduled entry says left: true, with nextDelayMs 0 and the read's error in previousError.
  • The run stays running. Steps running beside it finish first.
  • The run is resumed once its lease runs out, 5 minutes by default (If the runtime stops), and the step runs again. A step in a loop, a fan-out or a subflow is left with the whole loop, fan-out or subflow step, and the resume replays from the journal what had finished.
  • The runtime logs it at WARN: flow run <run-id>'s step "<node-id>" was left for the run's next resume: the gate policy couldn't be read, naming the read.

A leave uses one of the step's attempts under retry.maxAttempts, so a retry after it has one attempt fewer. A step is left 3 times at most. Its fourth failed read is a failed attempt like any other: it's retried if the policy has attempts left, and otherwise the step fails with the read's error, and so does the run.

A step isn't left when its read finds nothing, such as an agent that isn't registered: that's a failed attempt. Nor is one whose read was cut short by its timeoutMs or by the run's cancel.

A subflow's child run that's left with its parent's subflow step is held for its parent. It stays running, and only the parent's resume resumes it: a resume of the child itself is refused with run-held-by-parent. Once the parent's subflow step has been left 3 times, the child's step fails with the read's error, and the parent's subflow step fails with child-failed.

Where a policy applies

Section titled “Where a policy applies”

A policy applies to the step its edge leads to, when that step has exactly one incoming edge. A step that joins branches, with several incoming edges, ignores the policies on them. The other policy fields, concurrencyKey and priority, are in the flow schema.

Next

Section titled “Next”