Skip to content
Kindgi

Write a guardrail

View as Markdown

This guide writes a guardrail that keeps an agent's answers short, puts it on an agent, and tests its check.

Before you start

Section titled “Before you start”
  • A pack with an agent, on a runtime such as kindgi dev, with Kindgi 0.1's CLI. This guide uses the quickstart's pack, my-pack, and its my-pack.echo-agent, from the TypeScript, Python, Java or Scala quickstart.

The guardrail

Section titled “The guardrail”

The guardrail fails a turn whose answer is longer than a limit. It's one file in the pack's guardrails folder. Ask your coding agent: "I want a guardrail that keeps my agent's answers short." Kindgi's skills in your pack tell it how (How it knows Kindgi). For reference, the guardrail looks like this. This guide calls it my-pack.answer-length:

guardrails/answer-length/index.ts
import { defineCheck } from '@kindgi/sdk/define';
import { z } from 'zod';


export const check = defineCheck({
  id: 'my-pack.checks.answer-length',
  kind: 'zero-llm',
  configSchema: z.object({ maxChars: z.number().int().positive().default(500) }),
  evaluate: async (config, trace) => {
    const length = trace.output?.length ?? 0;
    if (length > config.maxChars) {
      return {
        passed: false,
        reason: `The answer is ${length} characters; the limit is ${config.maxChars}.`,
      };
    }
    return { passed: true };
  },
});


export default {
  id: 'my-pack.answer-length',
  name: 'Answer is short',
  kind: 'zero-llm',
  check,
  config: { maxChars: 60 },
  action: { 'on-violation': 'halt' },
  severity: 'error',
};

The file has two parts:

  • The check, from defineCheck, exported as check. Its evaluate gets the guardrail's config and the turn's trace, and returns { passed, reason? }.
  • The declaration, the default export: the guardrail's id, the check it runs, the check's config, and the action on a failure.

  • The id is <pack-id>.<name>. Name the rule as what should hold: answer-length, response-not-empty.

  • kind: 'zero-llm': the check is your code, a function of the trace. It can't call a model. To have a model judge the answer, see Have a model judge.

  • The reason is what the violation reports. Say what was wrong. It shows wherever the failure does, such as the run's error and its page in the console. So describe what you blocked, never quote it: "it contains a card number", not the number.

  • The trace is the turn. RunTrace in the Python reference lists its fields. Its names are camelCase in TypeScript and snake_case in Python:

    • output: the answer it judges: the turn's final answer. When the turn has a retry guardrail at the final answer, a halt or retry guardrail there judges each answer the model writes;
    • toolCalls, or tool_calls: its tool calls, each with the tool's name and arguments. At the final answer, that's every call the model asked for. A TypeScript check can also run at a tool call or a model reply, where the trace holds other fields: Check a tool call or a reply;
    • the tools' results, its model calls, the user's input and its cost;
    • projectId, or project_id: the run's project;
    • orgId, or org_id: the project's org, when it has one.

    The runtime sets the project and org, never the input.

Put it on an agent

Section titled “Put it on an agent”

A guardrail checks only the agents that list it. Ask your coding agent: "Put the answer length guardrail on my agent." For reference, it adds it to the agent's guardrails:

// in agents/echo-agent/index.ts
guardrails: ['my-pack.response-not-empty' as GuardrailId, 'my-pack.answer-length' as GuardrailId],
Excerpt

The agent names the guardrail's id, not the check's.

Once it's saved, kindgi dev registers the guardrail with the rest of the pack. An agent that names a guardrail that isn't registered doesn't run. Its turn fails with:

Error [unresolved-guardrail]: Agent "my-pack.gated" references guardrails not in the registry: my-pack.no-such-guardrail

Run it

Section titled “Run it”

Start a turn to see the guardrail stop it. dev-echo's answer, its "isn't a real model" line and the tool's result, is longer than 60 characters, so the turn stops. --agent is the agent, and --input its input:

Terminal window
pnpm exec kindgi runs start --agent=my-pack.echo-agent --input='{"userMessage":"hi"}'
Error [guardrail-violation]: Turn blocked by guardrail 'my-pack.answer-length': The answer is 243 characters; the limit is 60.

The command exits with status 1. The run's status is failed, and it has no answer. Stop a turn or record a violation shows what the run records, and how to let the turn finish instead.

A check that throws can't run, and its action decides what happens next: with halt, the turn fails (When the check can't run). The error carries the exception's message: handler-throw: Check "…" evaluate() threw: …. Return passed: false for a rule that isn't met. Throw only when the check itself can't run.

Check a tool call or a reply

Section titled “Check a tool call or a reply”

A check can judge a step of the turn before it takes effect, instead of the final answer: a tool call before it runs, a call's result before the model sees it, or a model reply before its calls run. The check names the points it can run at, and each guardrail that runs it picks one of them with at (Where in the turn it runs).

defineCheck({ points }) names the points: 'before-tool', 'after-tool', 'model-step' and 'final-answer'. A check without points runs at the final answer only. This check refuses a refund over a limit, before the refund tool runs:

guardrails/refund-limit/index.ts
import { defineCheck } from '@kindgi/sdk/define';
import { z } from 'zod';


export const check = defineCheck({
  id: 'my-pack.checks.refund-limit',
  kind: 'zero-llm',
  points: ['before-tool'],
  configSchema: z.object({ maxAmount: z.number().positive().default(100) }),
  evaluate: async (config, trace) => {
    const amount = trace.call?.arguments.amount;
    if (typeof amount === 'number' && amount > config.maxAmount) {
      return {
        passed: false,
        reason: `The refund is ${amount}; the limit is ${config.maxAmount}.`,
      };
    }
    return { passed: true };
  },
});


export default {
  id: 'my-pack.refund-limit',
  name: 'Refunds stay under a limit',
  kind: 'zero-llm',
  check,
  config: { maxAmount: 100 },
  at: 'before-tool',
  scope: { tools: ['acme.orders.refund'] },
  action: { 'on-violation': 'halt' },
  severity: 'error',
};

At each point, the trace holds what the check judges:

  • trace.at: the point it runs at. Absent at the final answer.
  • trace.call, at 'before-tool' and 'after-tool': the call, with its callId, toolId, toolName, toolVersion and arguments.
  • trace.result, at 'after-tool': what the call came to. { output } when it returned, or { error: { code?, message } } when it threw.
  • trace.step, at 'model-step': the reply, with its text and the toolCalls it asks for. text is empty when the reply only asks for calls.

Away from the final answer, the trace has no output. Its toolCalls and toolResults hold only the calls that ran, never the one the check judges.

defineCheck throws invalid-check-definition when points is empty, names a point that doesn't exist, or names one twice.

With halt, a refund over the limit stops the turn before the call runs:

Error [guardrail-violation]: Turn blocked by guardrail 'my-pack.refund-limit' before calling acme.orders.refund: The refund is 250; the limit is 100.

With retry, the call doesn't run, and the model gets the reason as its result, so it can go on without the refund or ask for a smaller one. At a tool call or a model reply describes each action at each point.

Test the check

Section titled “Test the check”

The check is a function of the config and the trace, so a unit test calls it directly, without a run. Ask your coding agent: "Write tests for my answer length guardrail." For reference, the tests look like this, with the command that runs the pack's tests:

guardrails/answer-length/index.test.ts
import type { RunId, TenantId } from '@kindgi/sdk/types';
import { expect, test } from 'vitest';


import { check } from './index.js';


const trace = (output: string) => ({
  runId: 'run-1' as RunId,
  tenantId: 'test-tenant' as TenantId,
  output,
  toolCalls: [],
  toolResults: [],
  modelCalls: [],
  mode: 'runtime' as const,
});


test('passes a short answer', async () => {
  expect(await check.evaluate({ maxChars: 60 }, trace('Shipped.'), {})).toEqual({ passed: true });
});


test('fails a long answer, with a reason', async () => {
  const result = await check.evaluate({ maxChars: 5 }, trace('Shipped yesterday.'), {});
  expect(result.passed).toBe(false);
  expect(result.reason).toBe('The answer is 18 characters; the limit is 5.');
});
Terminal window
pnpm exec kindgi test
 ✓ guardrails/answer-length/index.test.ts (2 tests) 2ms
…
      Tests  7 passed (7)
Excerpt

Have a model judge

Section titled “Have a model judge”

A guardrail of kind llm-judge has a model judge each answer against a rubric, in place of a check you write. Use it for a rule that code can't check, such as tone. Each answer it judges costs a model call. Ask your coding agent: "I want a guardrail that has a model check my agent's answers are professional." For reference, it looks like this. This guide calls it my-pack.professional-tone:

guardrails/professional-tone/index.ts
// The runtime judges an llm-judge guardrail with a model; it never calls this check.
export const check = {
  id: 'my-pack.checks.professional-tone',
  evaluate: async () => ({ passed: true }),
};


export default {
  id: 'my-pack.professional-tone',
  name: 'Professional tone',
  kind: 'llm-judge',
  check,
  config: {
    rubric: 'The answer is professional in tone and has no slang.',
    responseFormat: 'pass-fail',
  },
  judgeCapabilities: { needs: [{ feature: 'structured-output' }] },
  action: { 'on-violation': 'log-only' },
  severity: 'warn',
};

  • What the judge sees: the rubric, the turn's final answer, and its tool calls with their arguments. It doesn't see the user's message or the tools' results, so write a rubric that the answer and the calls are enough to judge.

  • config is what it judges by:

    • rubric: what should hold.
    • responseFormat: pass-fail, the default, has the judge answer PASS or FAIL, then its reason. score has it answer a score from 0 to 1, which passes at threshold or above: 0.5 unless you set one.
    • temperature: the judge's temperature, when you set one.

  • The verdict: a FAIL, or a score below the threshold, fails the check. A reply that starts with neither PASS nor FAIL, or, for a score, whose first line doesn't start with a number, fails it too. The action handles a failure as it handles any check's (Choose the action). The judge's reason is the violation's reason, and shows wherever the failure does, so the rubric can ask for a reason that doesn't quote the answer.

  • judgeCapabilities (judge_capabilities= in Python) is what the judge's model must support, such as structured-output. The runtime picks the judge's model by it, deployed and under kindgi dev. When no model fits, the check can't run (When the check can't run). kindgi build and kindgi dev refuse an llm-judge guardrail without it, and name the guardrail:

    guardrail my-pack.professional-tone is kind 'llm-judge' with no 'judgeCapabilities': name what its judge model is routed by (for example { needs: [{ feature: 'structured-output' }] })

    In Python, @guardrail refuses it first with a DefinitionError, and in Java and Scala, check(...) throws an IllegalStateException, each with the same advice.

  • The check is still declared, because every guardrail names one, but the runtime never calls it. So it only passes, and there's nothing of yours to unit-test.

  • What it costs: a model call for each answer it judges. When the turn has a retry guardrail at the final answer, a halt or retry guardrail there judges each answer the model writes, a call each. budget.maxCostUsd isn't enforced (How long its check may take).

In the console

Section titled “In the console”

A project's Guardrails page lists its guardrails: "Checks on agents' answers: what each checks, what happens when an answer breaks it, and the agents it checks. A run checks an agent only with the guardrails its version names." Each row has:

  • Guardrail: its name, with its description under it.
  • Check: its check, such as "Forbidden substring", over how it checks: "A rule in code", "A model judges it, a call per answer" or "Another system checks it".
  • If an answer breaks it: "The turn fails" ("Its answer isn't kept"), or "Recorded only" ("The answer goes out"). One that asks for a retry, a reviewer or a compensation is recorded only too, and says that part doesn't run yet.
  • Agents it checks: the agents whose versions name it, such as "Guarded agent" or "Drafts agent, Guarded agent and 2 more", or "None yet". Under them, where it's narrowed: "Not in runs: in CI only", "Only for …" or "Only in flows …".

The list has a search box, sorts by a column's header, and Columns chooses which columns it shows.

The Guardrails page: "Checks on each agent answer: how each checks, what happens when it fires, which agents it covers.", a search box and the Columns menu, then No secrets in answers, Says thanks, No refund promises, No shouting and No placeholder text, each a rule in code with its built-in check, when it fires, where it applies and how many agents name it: 1 agent each, and No agent for No placeholder text.

A guardrail's page has its name and description, and Remove for guardrail admins. Then:

  • What it checks: How, such as "A rule in code, no model call"; Check, such as "forbidden substring (built in)", or "(from a pack)" for a pack's own; Its settings; If an answer breaks it; Severity; and Applies: "In every run", "In every run, not in CI" or "Not in runs: in CI only".
  • Agents it checks: "Each agent version below names it. It applies in every run.", then each agent version that names it and can run here, marked "(latest)", "(pinned)" or "(pinned and latest)". With none, it says "No agent version that can run here names it (each agent’s latest, every pinned one, and every one a flow version in use pins), so it checks nothing here yet."
  • What it caught: its outcomes by agent version, as Stop a turn or record a violation shows.
  • Details: its definition, as the runtime sent it.

No secrets in answers' page, with Remove: a rule in code, forbidden substring (built in), Patterns: "password", the turn fails, Critical, in runs and in CI; the agent version that names it, Guarded agent 1.0.0 (latest); and What it caught: 8 answers checked in the last 7 days, 6 passed and 2 blocked.

A flow step runs the agent version its flow version pins, so that version is checked too: "Drafts agent 1.0.0 (pinned by flow acme.drafting 1.0.0)".

No shouting's page, with Remove, whose agent version is one a flow pins: Drafts agent 1.0.0 (pinned by flow acme.drafting 1.0.0).

Remove asks first, naming the agent versions whose turns then fail, and how they come back:

Remove No secrets in answers?
Guarded agent 1.0.0 (latest) names it: its turns fail, saying it isn't registered, until it's registered again or a version that doesn't name it takes over.
There's no undo: registering it again brings it back.
  • When several versions name it, it counts them and lists each: "2 agent versions name it: their turns fail, saying it isn't registered, until it's registered again or versions that don't name it take over."
  • When a flow pins one of those versions, it adds: "The flow version that pins it fails at that step too: …".
  • When the project's agents or flows can't be read, it says so: "The project's agents or flows couldn't be read, so which versions name it isn't known. Any that do fail their turns, and the flow steps that run them, until it's registered again."
  • Keep it closes the dialog with nothing changed.
  • Once it's removed, the dialog says "Removed. Its definition, to register it again:", with the definition, and Done.
  • Past 100 agents or flows, the page and the dialog say which weren't checked: "Only the first 100 agents were checked." They say the same of the first 100 flows' steps. The list says "Only the first 100 agents were checked for which versions name each."
  • Under kindgi dev, Remove is off: "the pack's files are its guardrails. Take it out of the pack, and kindgi dev reloads it."

The Remove No secrets in answers? dialog: Guarded agent 1.0.0 (latest) names it, its turns fail until it's registered again or a version that doesn't name it takes over, and there's no undo; then Keep it and Remove.

Next

Section titled “Next”