Skip to content
Kindgi

Operate a self-hosted runtime

This page continues Self-host Kindgi: the same containers (kindgi-server, kindgi-db, kindgi-pack), the same kindgi.env and pack.env, and the same pack. Commands that take your API token read it from $KINDGI_API_TOKEN.

Restart the runtime

Section titled “Restart the runtime”

The runtime reads its settings when it starts, so most changes on this page take a restart:

Terminal window
docker stop --time 30 kindgi-server
docker rm kindgi-server
docker run -d --name kindgi-server --network kindgi --restart unless-stopped \
  --add-host registry.localhost:host-gateway \
  -p 127.0.0.1:4000:4000 --env-file kindgi.env \
  quay.io/kindgi/runtime:0.1.6

On a stop, the runtime stops taking requests and gives the runs it's executing up to 7 seconds to finish, then exits with code 0. --time 30 gives it that time before Docker kills it.

Check health and logs

Section titled “Check health and logs”

The log lines on this page are in the pretty format. A runtime in a container logs JSON unless KINDGI_LOG_FORMAT=pretty is set; Logs has both formats, the levels, and how to follow one request.

Terminal window
curl -s http://localhost:4000/health
{"ok":true}

Point a load balancer's health check at /ready (or /health), not /: / leads to the console (a 302), and without the console it answers 200 even while the database is down, so a check on / would pass a broken runtime.

/health says the process is up. /ready says its database answers too, within two seconds. Neither needs a token:

Terminal window
curl -s http://localhost:4000/ready
{"ok":true,"database":"ok"}

While Postgres is unreachable, /ready answers 503 with {"ok":false,"database":"unreachable"}, and /health still answers {"ok":true}. Use /ready for a load balancer's or platform's readiness check. The image's own health check uses it, so docker ps shows the runtime's state:

Terminal window
docker ps --filter name=kindgi-server --format 'table {{.Names}}\t{{.Status}}'
NAMES           STATUS
kindgi-server   Up 34 seconds (healthy)

docker ps shows (unhealthy) while the database is down, and the runtime's log says why:

17:37:34.151 WARN  [ready] the database doesn't answer: no answer within 2000 ms

A route that reads the database answers 500 and names the cause. With Postgres unreachable, GET /v1/deployments answers:

Terminal window
curl -s http://localhost:4000/v1/deployments -H "authorization: Bearer $KINDGI_API_TOKEN"
{"error":{"code":"internal-server-error","message":"Deployment list failed: deployments: Tenant-scoped query failed: host not found (getaddrinfo ENOTFOUND kindgi-db)","requestId":"req-…"}}

The startup log

Section titled “The startup log”

The runtime prints what it's running with when it starts (docker logs kindgi-server; in the JSON format they're the lines of its boot record). The lines to check after a change:

  Token:   kgi_bt_…3236 (provided)
  …
  Public run tokens: off (no signing key)
  …
  License: …
  …
  Env: production (tool secrets resolve in it)
  Tenant host access: deployed (stdio MCP endpoints refused; tenant-chosen hosts can't reach the metadata server or this host; KINDGI_TENANT_HOST_ACCESS)
  Pack service: http://…:8080 — acme-pack (artifact …), protocol 2, 3 tools, 1 check
  • Token: the last four characters of the API token it accepts.
  • License: who the key is for, its kind and its end date. A warning follows within 30 days of the end.
  • Pack service: the pack it reached, with its artifact version. When it can't reach it, a warning takes this line's place, with the reason.

After a crash

Section titled “After a crash”

When the runtime comes back after stopping without a shutdown, its sweep repairs the runs the stop left mid-way (If the runtime stops). Each repair is a line in its log:

<time> INFO  [leases] resuming run <run id> (tenant <tenant id>): its wait resolved and nothing resumed it
<time> INFO  [leases] woke the flow run waiting on child run <run id> …: the child had moved on
<time> INFO  [leases] ended run <run id> …: its parent run had ended

Where errors show

Section titled “Where errors show”

  • A setting the runtime refuses (a missing license key, or a log setting it can't use): it exits with code 2, and its log says what to fix.

  • A database it can't reach while starting: it exits with code 1, and its log says which database and why, never the password. Here Postgres wasn't running:

    Terminal window
    docker inspect --format '{{.State.ExitCode}}' kindgi-server
    docker logs kindgi-server
    1
    Can't connect to the database at kindgi-db:5432/kindgi: host not found (getaddrinfo ENOTFOUND kindgi-db). Check KINDGI_DATABASE_URL, and that Postgres is up and reachable from here.

  • Another failure while starting: it exits with code 1, and the log's first line starts with kindgi-runtime: fatal: and ends with the cause.

  • A run that fails: its failureMessage, in pnpm exec kindgi runs get <run id>.

Back up Postgres

Section titled “Back up Postgres”

The runtime keeps its state in Postgres: deployments, trusted keys, providers, runs and their journals. Back it up with pg_dump, while the runtime runs:

Terminal window
docker exec kindgi-db pg_dump -U kindgi -Fc kindgi > kindgi-backup.dump

-Fc is pg_dump's custom format, which pg_restore reads. The dump holds your runs' inputs and outputs: store it like the database.

If the runtime stores model keys (Models that need a key), the dump holds them encrypted, but not the keys that open them. Back up the secrets store's keys too, apart from the dump: the AAD key, and the local key (with Google Cloud KMS, that key stays in KMS). A restore needs the same ones. A runtime started with another local key stops with exit code 2:

The secrets' local key changed: tenant 207f5388-1737-45c0-a02c-b5388ca12da0's secrets were stored under local:42080424744213e5, and this key is local:f6bc4a15bef0d446. Start with the key they were stored under (KINDGI_SECRETS_LOCAL_KEY_PATH or KINDGI_SECRETS_LOCAL_KEY).

A different AAD key isn't checked when the runtime starts, so keep the two keys together.

Restore into a fresh database

Section titled “Restore into a fresh database”

Start a new Postgres, and restore the backup into it once it accepts connections:

Terminal window
docker run -d --name kindgi-db-restored --network kindgi \
  -e POSTGRES_USER=kindgi -e POSTGRES_PASSWORD="$DB_PASSWORD" -e POSTGRES_DB=kindgi \
  -v kindgi-db-restored:/var/lib/postgresql/data \
  pgvector/pgvector:pg16


docker exec kindgi-db-restored pg_isready -U kindgi -d kindgi
docker exec -i kindgi-db-restored pg_restore -U kindgi -d kindgi --no-privileges < kindgi-backup.dump

--no-privileges leaves out the backup's grants. The runtime grants its database role what it needs every time it starts, and a fresh server doesn't have that role yet.

Point the runtime at the new database in kindgi.env:

Terminal window
KINDGI_DATABASE_URL=postgres://kindgi:<DB_PASSWORD>@kindgi-db-restored:5432/kindgi

Then restart the runtime. A run from before the backup is there:

Terminal window
pnpm exec kindgi runs get <run id> --url http://localhost:4000 --token "$KINDGI_API_TOKEN"
  "status": "completed",
  …
  "output": {
    "reply": "Hello, Ada! It's great to meet you. How can I assist you today?",
    "greeting": "Hello, Ada!"
  }

A backup taken before an erasure brings back what it cleared: replay the erasures next (Erasures and backups).

Erasures and backups

Section titled “Erasures and backups”

An erasure keeps no identifier of whom it erased, only a keyed hash in the erasure ledger, so a replay after a restore can find them again. Give the runtime the ledger's key:

  • KINDGI_ERASURE_LEDGER_KEY_PATH: the absolute path of a file holding 32 random bytes (openssl rand 32), mode 0600;
  • or KINDGI_ERASURE_LEDGER_KEY: the same 32 bytes, base64, where secrets come as environment variables.

Use the same key on every replica, whatever the secrets backend, and keep it the same across a restore: losing it means losing replay. Without it, erasures still run, but each answers with an erasure-unmatchable warning, and a replay can't find whom it erased.

Three places say whether erasures can be replayed:

  • the startup log's Erasures line, with the key:

    …
      Erasures: on; the ledger is replayable after a backup restore (key from KINDGI_ERASURE_LEDGER_KEY)
    …

    and without it:

    …
      Erasures: on, but NOT replayable after a backup restore: no KINDGI_ERASURE_LEDGER_KEY, so the ledger can't keep a keyed hash
    …

  • /ready's erasures: replayable or unreplayable;

  • kindgi doctor's erasures check. It never fails, since a runtime without the key is fine for development. With the key, then without it:

    …
      ✓ Erasures: Erasures can be replayed after a backup restore: the runtime has the erasure ledger key.
    …
    …
      – Erasures: Erasures run, but a replay after a backup restore can't find whom they erased: the runtime has no KINDGI_ERASURE_LEDGER_KEY. Fine for development; set it where you run in production.
    …

Then:

  1. Export the ledger off-box, regularly: a restore rolls it back with everything else.

    Terminal window
    kindgi memory erasures export --out=erasures-$(date +%F).json

  2. After a restore, replay the latest export before the runtime serves anyone. It erases again whoever the restored database holds:

    Terminal window
    kindgi memory erasures replay erasures-<date>.json

    Its answer lists the erasures it replayed, the ones it restored to the ledger, and any it couldn't match (unmatched).

Two more things keep erasures complete:

  • End users' ids are opaque: give participantId an id your app uses for the person, never an email or a name. Ids stay on records an erasure keeps.
  • Postgres can keep a cleared row's old version on disk until it's vacuumed. Where that matters, run VACUUM (and REINDEX for indexes) on the database after erasures.

Upgrade the runtime

Section titled “Upgrade the runtime”

  1. Back up Postgres.

  2. Pull the new version, and restart with it, with the same kindgi.env:

    Terminal window
    docker pull quay.io/kindgi/runtime:<version>
    docker stop --time 30 kindgi-server
    docker rm kindgi-server
    docker run -d --name kindgi-server --network kindgi --restart unless-stopped \
      --add-host registry.localhost:host-gateway \
      -p 127.0.0.1:4000:4000 --env-file kindgi.env \
      quay.io/kindgi/runtime:<version>

When it starts, the runtime brings the database up to date: it applies the migrations the database doesn't have yet, then serves. On a database that has them all, it applies nothing, so a restart on the same version changes nothing. The log doesn't list them: once the Kindgi API server listening lines appear, they're done. If one fails, the runtime exits with code 1, and its log says kindgi-runtime: fatal: Error: migration failed for … and why.

Migrations only go forward, and an older runtime isn't guaranteed to work on a database a newer one migrated. To go back, restore the backup you took before the upgrade, and run the older version on it.

Checks you can run before upgrading

Section titled “Checks you can run before upgrading”

From 0.1.6, the runtime image runs read-only checks against a database. They read only what the previous release has, so run the new version's image against the database you're about to upgrade, with the same kindgi.env:

Terminal window
docker run --rm --network kindgi --env-file kindgi.env \
  quay.io/kindgi/runtime:<version> check provider-keys

Nothing starts and nothing is written, and they print names and ids, never a value. Each exits 0 when it finds nothing, 1 when it lists what it found, and 2 when it can't run. --tenant <uuid> checks one tenant; by default they check every tenant in the database.

Check What it lists
provider-keys Each tool version, MCP endpoint and webhook endpoint that names a model provider's key. A model provider's key is used by its provider only, so from 0.1.6 they're refused: a tool's calls fail with provider-key-refused, and so do an MCP endpoint's connections and a webhook endpoint's deliveries. For each, store the key it needs under its own name (the same value is fine), name that instead, and deploy or register it again. See Give a tool a secret.

Here check provider-keys runs on a 0.1.5 database where a tool and an MCP endpoint name Anthropic's key (exit 1):

provider-keys: 2 name a model provider's key (1 tenant checked):
  tenant=… mcp-endpoint=acme.docs secret=ANTHROPIC_API_KEY provider=anthropic
  tenant=… tool=acme.summarize@1.0.0 secret=ANTHROPIC_API_KEY provider=anthropic
A model provider's key is used by its provider only: these are refused. For each, store the key it needs under its own name (the same value is fine), name that instead, and deploy or register it again.

After each got a key of its own, the same run finds nothing (exit 0):

provider-keys: nothing names a model provider's key (1 tenant checked).

The runtime logs the same at every start, one WARN per tool version or endpoint (docker logs kindgi-server). Here it's for another tool registered on 0.1.5 with Anthropic's key, acme.translate:

{"time":"2026-10-10T10:15:40.130Z","level":"warn","severity":"WARNING","subsystem":"providers.keys","message":"provider-key-declared tenant=… tool=acme.translate@1.0.0 secret=ANTHROPIC_API_KEY provider=anthropic: a model provider's key is used by its provider only, so this tool's calls are refused. Store the key it needs under its own name and use that name","tenantId":"…","kind":"tool","id":"acme.translate","version":"1.0.0","secretName":"ANTHROPIC_API_KEY","providerId":"anthropic"}

From 0.1.5 to 0.1.6

Section titled “From 0.1.5 to 0.1.6”

The database migrates when 0.1.6 starts.

Security fixes in 0.1.6

Section titled “Security fixes in 0.1.6”
  • A guardrail could run another tenant's check. On a runtime serving several tenants, a guardrail naming a pack check its own tenant had never deployed could run another tenant's check of that id, if that tenant had deployed one. 0.1.6 looks a pack check up in the guardrail's own tenant only: such a guardrail fails with unknown-check every time, and under halt its turn fails.
  • A model provider's key could reach a tool. Anyone who could register a tool, an MCP endpoint or a webhook endpoint could name a model provider's key as its secret, and so hand the key to a pack's code or to another server. 0.1.6 refuses that (run check provider-keys first, below).
  • Who may judge as a class was shown to every reader. Anyone who could read a judge class, a project viewer included, got the ids of the people and service accounts allowed to judge as it. 0.1.6 sends those ids only to an admin of the class's scope; everyone else gets their number (assertableBy.principalCount).
  • A malformed MCP credential could appear in the runtime's log. An MCP endpoint's credential, or a value in its config.headers, with a line break or a NUL inside it could show up in the sync log, in the error the HTTP library raised when it refused the header. 0.1.6 refuses such a value before sending, and its log names only the header. A log written before keeps what it wrote: rotate an MCP credential that was ever stored with a line break inside it.
  • A secret with a line break could appear in a run's failure message. An HTTP tool's secret, or a model provider's key, containing a line break could show up in the error the HTTP library raised when it refused the header: for a tool, in the run's failure message and journal, which anyone who can read the run sees; for a provider's key, in errors and the logs. 0.1.6 refuses such a value before sending, and names only the header or the provider. A failure message or log line recorded before keeps what it recorded: rotate a secret or key that was ever stored with a line break anywhere but at its very end (a bearer secret that started with one counts). One with only a trailing line break was never affected.
  • The Python SDK held your app on old releases of cryptography. Its verify extra (pip install 'kindgi[verify]') capped cryptography below 47. So an app that installed it couldn't take the fix for the OpenSSL inside cryptography's wheels (GHSA-537c-gmf6-5ccf, fixed in 48.0.1), nor those for CVE-2026-69247, CVE-2026-69248 and CVE-2026-69249 (fixed in 49.0.0 and 50.0.0). The extra in 0.1.6 requires cryptography 50.0.0 or later and sets no upper limit, so your app can upgrade it freely. The SDK's verifier only checks signatures with a public key. It doesn't call the PKCS#7 decryption or X.509 certificate verification those three CVEs are in, but the rest of your app shares the cryptography you install.

What to check, and what's different

Section titled “What to check, and what's different”

What to check before you upgrade, and what's different after:

  • Run check provider-keys before you upgrade (see Checks you can run before upgrading). A model provider's key is now its provider's alone: a tool, an MCP endpoint or a webhook endpoint that names one is refused, and so are its calls, connections and deliveries (A model provider's key isn't a tool's).
  • Expect a slower first start on a large database. A new index on the runs table is built at the first start after the upgrade, without blocking writes; the more runs, the longer that start takes.
  • Set KINDGI_PAGINATION_KEY. List cursors are sealed, so they no longer show which row they point after. Without a key, a cursor handed out before a restart answers 400 after it, and a client starts the list again. More than one instance needs the key. When you rotate it, keep the old one as KINDGI_PAGINATION_PREVIOUS_KEY for at least a day.
  • Rebuild your packs with the 0.1.6 CLI:

    • Variables your pack doesn't declare no longer reach it. Before your pack's code loads, its service drops every variable the pack doesn't declare (env in kindgi.config, [tool.kindgi.env] in pyproject.toml, env in kindgi.config.json), except Kindgi's own (KINDGI_*) and the platform's. A key left in pack.env for something else no longer reaches your tools. After the upgrade, look in the pack service's log for a WARN line with "event":"env-dropped". It names each variable the service dropped, never its value:

      {"time":"2026-10-10T07:01:47.789Z","level":"warn","severity":"WARNING","subsystem":"pack","message":"Dropped 2 variables the pack doesn't declare: ACME_UNDECLARED_KEY, AWS_SECRET_ACCESS_KEY (declare them in the pack's env, or set KINDGI_PACK_ENV_FILTER=off)","event":"env-dropped","kind":"env-dropped","names":["ACME_UNDECLARED_KEY","AWS_SECRET_ACCESS_KEY"]}

      If your code reads one of them, declare it (Declare the environment your code reads). To keep the old behaviour meanwhile, set KINDGI_PACK_ENV_FILTER=off on the pack service. A Python pack's image always names GPG_KEY: its base image sets it, and nothing reads it.

    • A tool call has an idempotency key, ctx.idempotencyKey, the same every time the same call runs. Pass it to the system a tool writes to, so a step that runs again doesn't refund or email twice (Make a side effect happen once). A pack built before 0.1.6 doesn't get it.

    • A guardrail that names a built-in check is marked in the pack's index. One from an older CLI gets a warning in the runtime's log each time it loads; the built-in still runs. The CLI also warns when a pack's check id doesn't start with the pack's id.

    • A tool whose needsSpec schema doesn't compile is refused when it's defined, registered or deployed, instead of being left out at load (Give a tool env values). A union of types (type: ['string', 'null']) now compiles everywhere.

  • The API's own OAuth sign-in flow is removed. Sign-in runs in the runtime's browser flow, which is unchanged. POST /v1/auth/login/… and POST /v1/auth/callback/… answer 404; registering an oauth2 identity provider answers 422 (one stored before still lists); and OIDC allowedRedirectUris is refused (one stored before still works). The TypeScript and Python clients drop auth.login and auth.callback. Sessions no longer hold identity-provider tokens: 0.1.6 clears the ones stored.
  • Times are strict. Every time the API reads, list cursors included, is an RFC 3339 time with its zone (2026-10-09T12:00:00Z), or Postgres timestamptz text; anything else, a date alone included, answers 400 bad-input. The CLI's time flags still take a date and send it as that day's start in UTC.
  • member isn't a project role you can give: use viewer, which it always meant. A role given as member before still reads back as member.
  • Approvals and their runs end together (Decide an approval):

    • A reviewer's withdraw ends the run waiting on it: its turn fails with hitl-withdrawn, at once.

    • A run's end withdraws the approvals it waits on (a tool call's or a turn's), with withdrawnBecause: run-cancelled or run-ended. An approval a run asked for without waiting on it, such as a memory fact's review, stays pending.

    • Deciding one whose run has ended answers 409 run-already-terminal.

    • After the upgrade, gate approvals still open (pending, assigned or in review) from runs that had already ended or been purged are withdrawn the same way, by a pass about once a minute per tenant, up to 500 runs each time (in a test, 40 seconds after the first start). They leave the reviewer queues with a system actor in the audit (hitl-decision, via: run-end). Deciding one then answers 409 approval-terminal, for a cancelled run and for one that ended otherwise:

      Approval … was withdrawn: its run was cancelled
      Approval … was withdrawn: its run ended

      A memory fact's review left open from before stays pending, and can still be decided.

  • Who sees whom:
    • listing a project's members or team grants takes write on the project, so a viewer no longer sees who else works there;
    • the reviewer roster is for tenant admins and reviewers;
    • re-adding a member with another role answers 409 membership-exists and keeps the role they hold;
    • 409 block-project-mismatch no longer names the other project.
  • The access audit keeps every refusal of a caller the API knows, not only the authorization model's decisions, except four that don't refuse the caller (The access audit). Every secret and env write through the API is in the audit log too, never with a value.
  • A turn stops when its run is stopped: a model or tool call in flight is aborted at once on a cancel or a shutdown. A step that writes nothing for a while, such as a long model call, stops within KINDGI_RUN_ENDED_CHECK_MS of a cancel (5 s by default).
  • Cancelling a flow while it runs a loop stops the loop: no more of its steps start after the cancel.
  • Two runtime instances starting at once no longer deadlock or race each other on migrations: one waits until the other's migrations finish. One starting beside an instance that's serving writes can still fail (see Known limitations in 0.1.6).
  • Guardrail, tool and provider changes reach every runtime instance on one database at its next turn, with no restart, whichever instance (or a plain SQL change) made them.
  • Paging no longer skips or repeats rows recorded in the same millisecond as a page's last row, on the lists that page by time: organizations, projects, teams and their members, memory erasures, an agent's promotions, a schedule's fires, cost records, observations, and the versions of agents, flows, policies, test sets, blocks and tools.
  • The sign-in options lookup limit holds across instances: 30 lookups a minute per client (GET /v1/auth/sign-in-options), counted in Postgres rather than by each instance.
  • Safari signs in to kindgi dev's console. Under kindgi dev on http://localhost or 127.0.0.1, the session cookie is a plain one, since Safari keeps a Secure cookie only over https. A deployment's console still needs https.
  • The console's sign-in returns you only to a console page; a sign-in link that pointed elsewhere lands on the console's home.
  • The console has a new look, light or dark: it follows your system, or the one you choose (also in ⌘K). Its links no longer point at an address without the tenant while a page loads.
  • Every /v1 answer says Cache-Control: no-store, errors included, so no proxy or browser keeps one tenant's data or shows a kept answer again: a 410 for a flow since reinstated no longer outlives a reload (Behind a load balancer or ingress).
  • ⌘K and the breadcrumbs reach every project section; choosing Cost or Schedules there used to do nothing.
  • A record opened under another project's address moves to its own project (a run, an approval, an agent and the pages under it, a tool, and the like), keeping its tab, instead of showing under the wrong project with actions that project then refused. One in a project you can't open says so: "This agent belongs to another project, which you can't open. Ask an admin of it for access."
  • kindgi dev runs your pack's code sandboxed (macOS Seatbelt; Linux bubblewrap, which you may need to install): it can't read your keys or files outside the app, write outside it, or reach Docker, and keeps the network and the app's own files. On Linux, the UNIX sockets it closes are those under the home folder, /tmp, /var/tmp and /run. A tool that reads a path or a socket outside the app (~/.aws, a local Postgres socket) needs it opened in the pack's dev.sandbox config. Where it can't run (Windows outside WSL, a container, inside another sandbox such as a coding agent's), kindgi dev warns and runs without it; KINDGI_DEV_SANDBOX=required refuses instead (Keep your tools' code away from your keys).
  • Under kindgi dev, Kindgi keeps the secrets you store in its own file, .kindgi/secrets.env, not your app's .env.local, and neither they nor a model provider's key reach the pack service's environment (kindgi secrets copy copies keys over; Keep local values in env files).
  • The CLI: kindgi tools versions --include-tombstoned is now --include-unregistered, as blocks and flows say.
  • The clients: the TypeScript client drops eventTriggers and webhooks and the Python client their resources, which the runtime never served (it fires schedules; start a run with POST /v1/runs). Python's secrets.rotate returns its sync or async answer, and Kindgi() finds its settings when it's first used.
  • Embedding @kindgi/api: createApp's signingKey is gone (pass exportSigning), and so are refreshToken, exchangeCode and oauthStateStore. SignInOptionsRateLimit.store takes a RateLimitStore (in memory by default) to share the sign-in lookup limit across instances; a store that fails doesn't block sign-in.
  • New in 0.1.6:
    • Judging: a project's judging rules queue the runs that need a person's verdict, results per agent version, and comparisons and proposals rescored after people judge the new answers (Judge a run's output, Improve an agent).
    • What failed and why: a project's failed runs grouped by cause and version (GET /v1/runs/failures) and a failure's reason (See why runs failed), and what a suspended run waits for (waitingFor). The runs list filters by status, time and version (List runs).
    • Guardrail outcomes: how often each guardrail passed, found something, blocked a turn or couldn't run, by agent version (GET /v1/guardrails/{id}/outcomes).
    • Approvals: an approval.requested webhook event, email to reviewers (KINDGI_REVIEWER_EMAIL), and an inbox in one read (--assigned-to=me --order=asc) (Receive webhooks).
    • The console: a reviewer's Inbox across projects, where you can decide several approvals in one go or by keyboard (Decide an approval). A project gets pages for its failed runs (What failed), schedules, flows, guardrails and members, each with a page of its own, plus Quality for judging, and costs for the project and each run. The tenant gets Teams, Webhooks, Sign-in, My access and its own costs. A page or action shows only to the roles that can use it (The console, by role).
    • Access: who has access to a project and how (GET /v1/projects/{id}/access), a team's role on a project, what the caller may do (GET /v1/identity/me/permissions), the people list with their grants, and the tenant's sign-in history (GET /v1/audit/sign-ins) (Authorization).
    • Sign-in an operator manages alone: with KINDGI_AUTH_TENANT_PROVIDERS=off, a tenant can't add, change or remove identity providers.
    • Your own secret manager: Google Secret Manager, Azure Key Vault or HashiCorp Vault (KINDGI_SECRETS_BACKEND=secret-manager; Keep secrets in your own secret manager), and on Azure a Key Vault key for the database's secrets, Azure Container Registry with the server's managed identity, and export signing with a Key Vault key (Export signed evidence). Retired export keys can stay listed, so older exports still verify.
    • A tool's secret can be optional (An optional secret).
    • A retired agent, flow, tool or test set can be found and brought back (Agent and tool versions).
    • Schedules list by project and name their owners.
    • Coding agents: kindgi init keeps one out of the files that hold keys (Your coding agent).

Known limitations in 0.1.6

Section titled “Known limitations in 0.1.6”

  • Run one runtime instance. 0.1.6 fixes the known reasons several instances couldn't share a database: boot migrations wait their turn, the sign-in limit is shared, guardrail, tool and provider changes reach every instance, and page cursors work across instances with one KINDGI_PAGINATION_KEY. Several instances together aren't verified yet, so the Cloud Run module still runs one.

  • AWS Secrets Manager comes in 0.1.7. With KINDGI_SECRETS_MANAGER=aws, a 0.1.6 runtime doesn't start (exit code 2):

    KINDGI_SECRETS_MANAGER=aws is reserved: this runtime doesn't support it yet. Currently supported: azure, gcp, vault.

  • An MCP server's tool whose name contains __ can't be called. Kindgi names it <endpoint id>.<tool name>, and every model adapter turns each . into __ for the model and back again on the way in. So a name with a __ of its own comes back wrong: the model is told the agent has no such tool (unresolved-tool), and the turn fails if it calls it again, or the model answers without it. Drupal's MCP Server Tool Bridge names its tools this way (tool_api__…). There's no workaround for a server whose tool names you don't choose: 0.1.7 fixes it.

  • A runtime starting while another serves the same database can fail to start. Each start applies row-level security to every tenant table again, under locks that a write in flight can deadlock with. So a new revision starting while the old one still serves, or a second instance, can stop at start (exit code 1):

    kindgi-runtime: fatal: Error: migration failed for … (…/migrations/): Migration step "enable-rls" failed: deadlock detected

    Starting it again serves. While it happens, the runtime that's serving can answer some writes with 500 internal-server-error, "Tenant-scoped query failed: deadlock detected", and keeps serving (in our test, some project creations). This isn't new in 0.1.6, and a fix is planned for 0.1.7.

Runtime 0.1.5.1

Section titled “Runtime 0.1.5.1”

Runtime 0.1.5.1 fixes one security bug in 0.1.5 and earlier, for deployments with authorization on (KINDGI_OPENFGA_API_URL set): someone re-added to a project or a team with a different role silently got that role as well, and kept it after they were removed. The members list never showed it. Without authorization, and under kindgi dev, nothing is affected. Only the runtime changes: the 0.1.5 CLI and SDKs (npm, PyPI, Maven Central) stay as they are.

On 0.1.5.1:

  • Removing someone from a project or a team removes the roles they held there.
  • Re-adding someone who's already a member keeps their current role. To give them another, change their role (Project memberships). The answer to the re-add still echoes the role you asked for (201), as the API is unchanged in this patch; the members list shows the role they hold.
  • Removing a team's creator ends their team admin, which they hold without a membership row. This matches what the team's members list shows.

Run 0.1.5.1, pulled by its digest, with the same kindgi.env. It has no migration:

Terminal window
docker pull quay.io/kindgi/runtime:0.1.5.1@sha256:7c1b111ff22091d137f45d9770f6ff9f2521575bf28130957a5db1ce80a2e56e

On Cloud Run, copy it into your repository the same way as 0.1.5 (see The images into Artifact Registry) and set server_image to its digest.

If you removed someone from a project or a team on an earlier version, after re-adding them with another role, they may still hold that role. Remove them again on 0.1.5.1: the removal now clears it. For a leftover owner on a project, add the person as owner, then remove them. 0.1.6 adds a check that finds who needs it.

From 0.1.4 to 0.1.5

Section titled “From 0.1.4 to 0.1.5”

The database migrates when 0.1.5 starts. What to check before you upgrade, and what's different after:

  • Signing in to the console with an API token is now off by default, except in kindgi dev. If people sign in to your console by pasting an API token, set KINDGI_CONSOLE_TOKEN_SIGN_IN=on on the runtime when you upgrade (on Cloud Run, in the module's server_env: Turn on sign-in), or set up sign-in with your organization's identity provider (Turn on sign-in). Otherwise the console's sign-in page offers no way in, and the runtime's startup output says so too. API tokens keep working for the API, the CLI and the SDKs either way.

  • Reach the console over https. Every way of signing in now ends in a session cookie marked Secure, which browsers keep only over https (MDN: Set-Cookie, Secure). On the machine running Kindgi, Chrome also keeps it at http://localhost (or 127.0.0.1); Safari doesn't. At any other plain-http address, no way of signing in works. Without KINDGI_PUBLIC_URL, the runtime also accepts the console's changes only from https, or http on localhost (Requests from other sites).

  • Behind a load balancer or ingress, set KINDGI_TRUSTED_PROXIES: how many proxies are in front of the runtime (1 for one), or their addresses. Rate limits and audit records then see each client's own address. Unset, the runtime uses the connection's address and ignores X-Forwarded-For (Behind a load balancer or ingress).

  • The API token keeps its user across restarts. Without KINDGI_SEED_USER_ID, 0.1.4 made a new user at every start; 0.1.5 keeps the token's user while KINDGI_API_TOKEN stays the same. A changed token acts as a new user, and the startup output warns. With authorization on, set KINDGI_SEED_USER_ID to keep one user across token changes (Point the runtime at it). Changing KINDGI_API_TOKEN also signs out the console sessions the old token opened (Rotate the API token).

  • Signing a person out everywhere works. In 0.1.4, POST /v1/identity/users/<id>/revoke-sessions answered as if it had, and ended nothing. Revoking an API key now ends the console sessions it opened, and removing an identity provider ends the sessions opened through it (Signing out). The audit trail gains signed-out, sessions-revoked and sign-in-refused.

  • Only tenant admins list the tenant's people. GET /v1/identity/users answers anyone else 403; a person still reads their own record and sessions. A project admin adds a member by email (POST /v1/projects/<id>/memberships with email), and a project admin's member key can manage that project's members.

  • A project editor can start an eval run (compare a version on a test set); it took a project admin. Unregistering or reinstating a test set version still does.

  • The cost aggregate counts only what you may read. Across projects (no scope, the tenant, or an org), it counts the projects the caller may read, and records with no project; a tenant admin's counts every project. read on the tenant used to show every project's spend.

  • With authorization on, every route checks what it touches. Reading tenant-wide settings (providers, policies, adapters, capabilities, signing keys, deployments, sign-in providers) needs read on the tenant, and changing them admin. Webhook endpoints and compliance evidence need admin. Starting a run needs execute on what it runs, and write on a project it names. Lists hold only what the caller may read. A single admin sees no change (Authorization). Unregistering a conversation takes write on its project.

  • With authorization on, a permission change that fails no longer holds up the others. In 0.1.4, one could leave a new project unreadable by its creator (403) until an operator replayed the outbox. 0.1.5 also carries the fixes in runtime 0.1.4.1 and 0.1.4.2 (below).

  • An agent, flow, tool or test set stays in the project it was first published into. Publishing a version of one under another project is refused with a 409 (agent-project-mismatch, flow-…, tool-…, eval-suite-project-mismatch), even for an admin of both, and nothing is written; 0.1.4 accepted it. A deploy that includes one is refused whole, even when it's unchanged. The refusal carries the id, not the other project (Organize work by org and project).

  • If your Postgres DateStyle is SQL, DMY or German, check the times earlier releases saved. Run SHOW DateStyle; (DateStyle). With either style, earlier releases read a date on the 1st to the 12th of a month with day and month swapped, and saved some that way: session expiries, approval deadlines and decision times, wait timeouts, schedule fire times, memory facts' validity times and stored webhook bodies. A date after the 12th failed outright. With any other style, a time was read correctly or failed outright, so nothing was saved wrong. 0.1.5 reads and writes times correctly on any DateStyle, but it can't repair a swapped time, which looks like a real one.

  • Run OpenFGA v1.22.0. Published OpenFGA advisories affect v1.9.0 (An OpenFGA you already run).

  • Expired rows are deleted every hour: idempotency answers, sessions and sign-in state, which 0.1.4 kept for good (Retention).

  • Idempotency keys are per caller. The same key from someone else is their own request. A repeat sent while the first request still runs is refused (409 idempotency-key-in-flight) instead of running again, and the answer of a request that made a secret (a new API key, say) isn't kept, so its repeat is refused with idempotency-key-replay-withheld. Keys 0.1.4 stored aren't found again; they'd have expired within 24 hours (Retry a start safely).

  • A provider registration the runtime can't build is refused when it's registered (422 provider-config-invalid, each problem in details.issues). kindgi doctor names any registered before 0.1.5; to fix one, unregister it and register it again (Check a registration).

  • A guardrail whose config its pack check would refuse is refused when it's registered (422 guardrail-config-invalid), for a pack deployed by 0.1.5: redeploy your pack once. A guardrail already registered with such a config fails every turn it checks; unregister it, and register it again with a config that fits (Configure a guardrail).

  • A halting guardrail whose check can't run now stops the turn; it used to let it through. That's a check that can't run, for any reason: no check by that name is registered, its configuration is invalid, an llm-judge guardrail's judge can't be routed to a model, or the check throws (check-failed: pack code that crashed, a pack service that couldn't be reached, a judge call that failed). With halt, the turn fails with guardrail-violation, and evaluationErrors says which guardrail and why. With any other action, the turn goes on; in 0.1.4 a check that threw failed the turn whatever the action. Either way, the error is in the run's provenance and journal (When the check can't run).

  • The built-in guardrail checks run. A guardrail that names one (must-cite, never-call-tool, max-tool-calls, output-matches, tool-order, required-substring, forbidden-substring) runs it, from a pack file or POST /v1/guardrails. In 0.1.4 it never ran (Use a built-in check).

  • A pack can't ship its own guardrail check under a built-in check's id: the runtime runs the built-in for a guardrail naming one, so a pack's implementation under that id would be silently replaced. Building the pack refuses it with reserved-check-id (Python: DefinitionError), saying to rename the check. Rebuild your packs with the 0.1.5 CLI: a pack built with an earlier one that ships such a check runs the built-in instead.

  • Before you upgrade, check every guardrail that names a built-in check. In 0.1.4 such a guardrail never ran. In 0.1.5 the built-ins run and check their config, but a guardrail registered before the upgrade isn't checked again: if its config doesn't fit the built-in's settings, its check can't run at any turn, and with halt it blocks every turn of the agents that list it. The runtime warns about each one when it loads it:

    20:40:17.669 WARN  [guardrails] guardrail config invalid traceId=… tenantId=… guardrailId=acme.tool-budget check=max-tool-calls problems="config must NOT have additional properties" fix="unregister it (kindgi guardrails unregister <id>), then register it again or redeploy, with a config that fits; with several runtime instances, restart the others"

    List your guardrails (kindgi guardrails list), and check each one whose check is a built-in against its settings. To fix one, unregister it (kindgi guardrails unregister <id>), then deploy your pack, or register it again, with a config that fits: a deploy doesn't change a registered guardrail's config. If you run several runtime instances, restart them afterwards (below).

  • The built-in guardrail checks check their config. A guardrail naming one with a config the check doesn't take is refused when it's registered (422 guardrail-config-invalid, each problem in details.issues) or deployed (deployment-validation-failed); one that still reaches a turn is a check that can't run, so a halt guardrail stops the turn.

  • A deploy keeps a registered guardrail only if it's the deploy's own: in the project the deploy registers into, with the same definition. A pack whose guardrail changed is refused (409 guardrail-already-registered): unregister the guardrail, and deploy again. A guardrail with that id in another project is refused too (409 guardrail-project-mismatch, without naming the project). Either way nothing is deployed. Both used to pass silently, and the old definition stayed in force. An unchanged pack still redeploys, from a new image and across releases.

  • The runtime signs exports (audit bundles, provenance, compliance evidence) with the deployment's export key: set KINDGI_EXPORT_SIGNING_KEY_PATH, KINDGI_EXPORT_SIGNING_KEY or KINDGI_EXPORT_SIGNING_KMS_KEY. 0.1.4's runtime didn't sign them. Without a key, outside development, the exports answer 404 signing-not-configured. kindgi exports verify also checks audit bundles made with @kindgi/api 0.1.4 (Export signed evidence).

  • The runtime's own address, /, leads to the console, or lists what it serves; it answered 404. Health checks stay on /ready (Check health and logs).

  • The console: a run whose answer a guardrail blocked shows that answer as not sent, with the guardrail; it used to look sent (A halted turn). The dashboard's API keys, Service accounts and People cards open their own pages (they opened Deleted data), and ⌘K and the breadcrumbs reach them. Access audit shows to tenant admins only, the people its API answers.

  • Anthropic retires Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) on 2026-11-30. A provider registration that names it should move to claude-sonnet-5-5, the anthropic preset's default. No preset lists it, so only a registration made with a spec is affected (Anthropic: model deprecations).

  • Claude agents use Anthropic's prompt cache: a turn's later calls read the prompt they repeat at a fraction of the input price, and the first write costs a little more. Register the anthropic preset again for the 5.5 models' cache-read rate (Prompt caching).

  • Retrieved memory reaches the model as data, in a <memory> block in a user message instead of a second system message, so what models see changes. A retrieval that searches by meaning (semantic) on a runtime without embeddings fails the turn with semantic-unavailable; 0.1.4 skipped that search without a word.

  • same-user memory is the run's end user's, never the key's user. A run that names no participantId reads no same-user memory, isn't offered kindgi_remember, and its result warns memory-needs-participant. Such a run used to read and keep that memory as the user its key acts for, so an app's service account serving many customers mixed their memory; facts kept that way are no longer read as anyone's. Pass the person's participantId on each run (Give an agent memory).

  • A comparison's replays run for the past turn's end user, with the same tools, kindgi_remember included (a replayed one stores nothing). A replay's conversation is never recalled as an earlier conversation.

  • Your pack's service writes log records on stderr, as the runtime does: one per tool call, at the levels KINDGI_LOG_LEVEL and KINDGI_LOG_LEVELS set (Logs). Printing a tool's context (console.log(ctx), print(ctx)) no longer shows its secrets.

  • A turn's warnings are logged, at WARN under [runs], once per tenant, agent and warning while the runtime runs, as well as in the run's result (Logs).

  • The clients read every 409 as a conflict (TypeScript code: 'conflict', Python ConflictError). Twenty codes used to come back as a server error, among them run-lease-lost, agent-version-mismatch and secret-write-conflict. Code that matched one of them by its class should match its code alone: err.serverCode (Python e.server_code).

  • TypeScript's runs.follow and runs.followProgress follow a run to its end, as in Python, which gains runs.follow too. runs.stream and runs.streamProgress are deprecated (Follow a run).

  • Paging conversations, approvals and runs no longer skips rows. A row created in the same millisecond as a page's last row (for approvals, at the same instant) could be left out of the next page. Cursors you hold keep working. A cursor whose time isn't a time is now 400 bad-input on conversations and runs, as it already was on approvals.

  • The CLI:

    • a usage error (a missing argument, a bad flag value) exits 2; 1 is for a call that failed;
    • kindgi build versions each build by its time (YYYYMMDD.HHMMSS, UTC), so a build without --artifact-version and --published-at is no longer reproducible;
    • KEY=${KEY} in a pack's .env takes the shell's value, as docker compose does; it used to come out empty;
    • kindgi dev gives the runtime your Google credentials only when KINDGI_DEV_GOOGLE_CREDENTIALS names them: a pack that uses Vertex AI adds KINDGI_DEV_GOOGLE_CREDENTIALS=adc to its .env (Gemini on Vertex AI);
    • kindgi console opens the console, and so does kindgi dev --open;
    • under kindgi dev, Sign in as seeded user shows only when the console is at a loopback address (localhost, 127.0.0.1 or [::1]); from another address, sign in with the dev token. A deployed console no longer asks for it.

  • New in 0.1.5:

    • Sign-in with your organization's identity provider (Set up SSO), and with Google, Microsoft or GitHub accounts or an emailed link through the deployment's own apps (Turn on sign-in).
    • People, API keys and service accounts: each person and pipeline acts with its own key and grants (People, API keys and service accounts).
    • Schedules: an agent or a flow at set times, as you, with a history of each time it ran (Run on a schedule).
    • Memory: an agent remembers what its declaration allows (kindgi_remember), with the scope and how long chosen by you, not the model, and can recall earlier conversations. Retrieval can search by meaning through an embeddings endpoint (KINDGI_MEMORY_EMBEDDINGS). A tenant admin can erase an end user's words, their conversations and the runs that served them (kindgi memory erasures).
    • Improvement passes: propose new settings or a new prompt for an agent version, compare the candidate on a test set, and promote it through the scope's gate after review (kindgi proposals). A pass can look for better settings on its own, and a schedule can start one.
    • Files kept with a run (artifacts) and the capability catalog (Keep files with a run).
    • A tool's env values per project (ctx.env) (Give a tool env values).
    • A failed run says why, as data: failure, with its code and message, on the run.
    • Logs: a pack's service writes log records, kindgi dev shows them, and records from a run carry its ids; providers and MCP endpoints can opt in to the run's trace (Logs).
    • Java and Scala, as a preview (a Java or Scala pack needs Jackson 2.18 or later in your app, which is Spring Boot 3.4 or later): Quickstart: Java, Quickstart: Scala and Call Kindgi from a Java app.
    • The Cloud Run module: an image repository you already have, no KMS key, calling your app's Cloud Run services from a tool, client addresses, and the runtime's own settings for sign-in (Deploy on Google Cloud Run).

Known limitations in 0.1.5

Section titled “Known limitations in 0.1.5”
  • Safari can't sign in to kindgi dev's console. The console's sign-in is a Secure cookie, and Safari doesn't keep one over plain http, even at http://localhost, where Chrome and Firefox do. Open the local console in Chrome or Firefox. A fix is planned. A deployment's console needs https in every browser (above).
  • Run one runtime instance. Several aren't supported yet (the Cloud Run module runs one). For one thing, a guardrail change reaches other instances only when they restart: registering, unregistering or deploying a guardrail takes effect at once on the instance that took the request, and other instances keep the guardrails they had. If you run several anyway, restart the others after changing a guardrail. A fix is planned.
  • Cancelling a flow while it runs a loop can let a few more of the loop's steps start before it stops: in our tests up to a few dozen, within seconds. The run still ends cancelled. A later release fixes it.

Runtime 0.1.4.2

Section titled “Runtime 0.1.4.2”

Runtime 0.1.4.2 fixes one bug in 0.1.4 and 0.1.4.1, for every deployment: when the database drops its connections (a restart, a failover, a network blip), the runtime could exit instead of reconnecting. Its log then ends like this:

file:///app/node_modules/.pnpm/postgres@3.4.9/node_modules/postgres/src/connection.js:255
    const x = socket.write(chunk, fn)
                     ^


TypeError: Cannot read properties of null (reading 'write')
    at Immediate.nextWrite (file:///app/node_modules/.pnpm/postgres@3.4.9/node_modules/postgres/src/connection.js:255:22)

On 0.1.4.2, requests that need the database fail while it's down, and the runtime keeps running and answers again once it's back. Only the runtime changes: the 0.1.4 CLI and SDKs (npm, PyPI) stay as they are. kindgi dev keeps its pinned 0.1.4 runtime, so if your local database restarts under it, restart kindgi dev.

Run 0.1.4.2, pulled by its digest, with the same kindgi.env. It has no migration, and it carries 0.1.4.1's fix (Runtime 0.1.4.1). Keep --restart unless-stopped on the runtime's container either way (Restart the runtime):

Terminal window
docker pull quay.io/kindgi/runtime:0.1.4.2@sha256:420826ad9bac0c2fdb021c49517aabebeac1ff7ec236e02af47e90a31f5b825e

On Cloud Run, copy it into your repository the same way as 0.1.4 (see The images into Artifact Registry) and set server_image to its digest.

Runtime 0.1.4.1

Section titled “Runtime 0.1.4.1”

Runtime 0.1.4.1 fixes one bug in 0.1.4, for deployments with authorization on (KINDGI_OPENFGA_API_URL set): a redeploy that published a new version of an existing agent could leave other permission changes made in the same few seconds unapplied. A newly published agent or project could then answer 403 to the person who made it. Without authorization, and under kindgi dev, 0.1.4 is unaffected. Only the runtime changes: the 0.1.4 CLI and SDKs (npm, PyPI) stay as they are.

With authorization on, run 0.1.4.1, pulled by its digest, with the same kindgi.env. It has no migration, so going back to 0.1.4 works, but the bug comes back with it:

Terminal window
docker pull quay.io/kindgi/runtime:0.1.4.1@sha256:3f14fcf7336c846b276c6119bc8dc96eaf44faee6dacfd45ba40b2e7c08d555f

On Cloud Run, copy it into your repository the same way as 0.1.4 (see The images into Artifact Registry) and set server_image to its digest.

0.1.4.1 and later don't retry a change 0.1.4 already lost. What comes back, and when:

  • A project or agent that answered 403 reads again from the upgrade on, since the upgrade restarts the runtime. At every start, the runtime writes again the permissions that place each project, agent, flow, tool, guardrail and test set of its tenant (KINDGI_TENANT_ID).
  • The creator's own rights on an agent come back when a new version of it is published on 0.1.4.1 or later.
  • A project membership added while the bug hit stays missing, through restarts and publishes, though the project's member list still shows the person. Add them again on 0.1.4.1 or later with the same call (POST /v1/projects/<project-id>/memberships, see Project memberships), and their access applies within seconds.

On 0.1.4.1, re-publishing an agent can log a warning like this one:

WARN  [authz.outbox] drain: FGA refused a batch; trying its tuples one by one tenantId=<tenant> rowCount=2 error="cannot write a tuple which already exists: user: 'project:<id>', relation: 'parent', object: 'agent:acme.alpha': tuple to be written already existed or the tuple to be deleted did not exist"

It's expected: the runtime then applies the batch's changes one at a time, and a change that's already there counts as applied. From 0.1.5, with OpenFGA v1.22.0, the warning doesn't appear: the runtime asks OpenFGA to ignore changes already in place, and v1.22.0 does, so it takes the batch whole. With an older OpenFGA that doesn't, the runtime still applies them one at a time, and logs it at info.

From 0.1.3 to 0.1.4

Section titled “From 0.1.3 to 0.1.4”

kindgi.env needs no change: the database migrates when 0.1.4 starts. What's different after:

  • Agent versions are pinned. A version published from 0.1.4 on resolves its tool ranges once, when it's published, and keeps those versions (Agent and tool versions). Versions published before have no pins and keep resolving each run. The first kindgi deploy after the upgrade registers a pinned next version of each agent, and says so:

    agent acme.matcher: registered new version 1.4.1 (1.4.0 was published before pins; 1.4.1 pins its tools); set version: '1.4.1' in acme.matcher to match

    Set that version in the agent's code. An app that runs an exact version (agentVersion: '1.4.0') keeps running the unpinned one until you change it; an app that runs the latest gets the pinned one.

  • With authorization on (KINDGI_OPENFGA_API_URL), the first start brings each tenant's authorization store up to 0.1.4's model, with a line per store before the banner:

    <time> INFO  [authz] tenant 8c3b6779-c02b-47e1-a9e1-4d0cd7f0c5a7: store 01M48C11G6C00WQXYMAJEWH7DW now has the current model (01M48C11V76E0C6APGAKKPRXXK → 01M48C24EGSZQHYJMEE2ZD88S2)

    Later starts print nothing. A store it can't reach gets a warning instead (could not bring store … up to the current model), the runtime starts anyway, and the next start tries again.

  • Deleting an org keeps its record until a retention policy on org purges it. It still answers 404 org-not-found, its projects and teams stay without an org, and its slug is free at once (Delete an org). 0.1.3 removed the record at once.

  • Unregistering a provider keeps its record until a retention policy on provider purges it; its id is free to register again at once (List, change and remove). 0.1.3 removed the record at once.

  • KINDGI_PUBLIC_URL is new: the address clients use when it isn't the one the runtime binds, behind a proxy or a load balancer (Check it).

  • The runtime's logs are structured. Each line is a record with a level and a subsystem: JSON when the output isn't a terminal (as on a server), readable lines on one. KINDGI_LOG_LEVEL, KINDGI_LOG_LEVELS and KINDGI_LOG_FORMAT choose what's written, and a request's traceparent is joined (Logs).

  • A conversation's turns run the agent version it was opened with. A turn that names another version is refused (agent-version-mismatch); one whose version is no longer registered answers 404 (A conversation keeps its agent version). 0.1.3 ran the current version and refused the turn when it differed.

  • kindgi runs start exits 1 for a failed run, with Error [<code>]: <message> naming the error's own code, such as budget-exceeded or model-invocation-failed. 0.1.3 printed Error [server] and exited 0 for a run that failed. A script that checks the exit code now sees the failure; the run itself is still printed on stdout.

  • dev-echo says it isn't a real model. Its answers start with ⚠ dev-echo isn't a real model, and a turn it answers carries a dev-echo-not-a-model warning.

  • On Cloud Run, the Terraform module pins the version of the secrets' AAD key it reads: add secrets_aad_key_version (normally "1") to your .tfvars before you apply (Deploy on Google Cloud Run).

  • Gemini 2.5 retires on Vertex AI. Vertex AI retires gemini-2.5-pro and gemini-2.5-flash on 2026-10-20; a turn routed to them fails after. The 0.1.4 gemini preset lists gemini-3.8-flash with gemini-3.5-flash-lite instead (If you registered Gemini 2.5). The anthropic preset adds claude-haiku-5-5 beside claude-haiku-4-5; Anthropic lists each model's status on its model deprecations page.

  • Register presets again. A provider registered from a preset before 0.1.4 keeps what it had: no default model, so an agent that names none gets the first model by name (claude-haiku-4-5 for anthropic); none of the models' temperature and thinking marks; and only the two base prices. Unregister it and register the preset again with the 0.1.4 CLI, or restart kindgi dev for one the pack's config declares. kindgi doctor warns (!) about one that's stale.

  • Each provider can name a default model (metadata.defaultModel): when nothing else decides, an agent gets it rather than the first model by name. Each preset names a mid-priced one, such as claude-sonnet-5-5 (How Kindgi picks).

  • Models that take no temperature get none. A model registered with "sampling": false (the Claude 5.5 and GPT-6 models) is called without one, and the turn carries a sampling-unsupported warning instead of failing. A model's thinking says how it thinks; thinking counts against its output limit and bills as output (Temperature and thinking).

  • Costs follow what providers bill. A registration keeps its model's extra rates: cached prompts, cache writes, long prompts and data residency. A long prompt prices the whole call at the long rates (Claude Haiku 5.5 past 100,000 tokens, Gemini 3.1 Pro past 200,000, GPT-6 past 272,000). A rate that isn't a non-negative number is refused when you register. Kindgi's costs stay estimates from published prices; your provider's invoice is what you pay.

  • OpenAI calls use OpenAI's Responses API, which GPT-6 models need to call tools: agents with tools now work on them. Every call sends store: false. An existing OpenAI registration moves over when you upgrade, with nothing to register again; "api": "chat-completions" in its adapter_config keeps the older API (OpenAI's own API).

  • Gemini calls retry a temporary failure (a rate limit 429, an overloaded model 503, and the like), three attempts in all, as the Anthropic and OpenAI-compatible adapters already did.

  • A waited start that outlasts the client's timeout (30 seconds in TypeScript, now settable with timeoutMs; 60 seconds in Python) fails on the client without the run's id, and says to start the run in the background and follow it (Wait for the result).

  • In 0.1.2 and 0.1.3, the Python client could start a waited run up to three times when it outlasted the client's timeout. 0.1.4's client never sends a call again once it may be running: upgrade the client. A start you repeat yourself with the same idempotency key still returns the first run only once the first request has answered (Retry a start safely).

Rotate the API token

Section titled “Rotate the API token”

Make a new token, replace KINDGI_API_TOKEN in kindgi.env with it, and restart:

Terminal window
printf 'kgi_bt_%s\n' "$(openssl rand -hex 32)"

The Token line in the log now shows the new token's last four characters, and the old token is refused:

{"error":{"code":"auth-missing","message":"Bearer token is not recognized","requestId":"req-…"}}

KINDGI_API_TOKEN is one token: switch your CLI and apps to the new one when you restart.

Console sessions signed in with the old token end at the restart, and people sign in again. The startup log counts them:

  Signed out: 1 console session an earlier token opened

A request with such a session afterwards answers 401.

Rotate the pack service token

Section titled “Rotate the pack service token”

The runtime and your pack's service share this token. Make a new one:

Terminal window
openssl rand -base64 32

Put it in both files, KINDGI_PACK_SERVICE_TOKEN in pack.env and in kindgi.env. Then restart the pack's service with the same image:

Terminal window
docker stop --time 30 kindgi-pack
docker rm kindgi-pack
docker run -d --name kindgi-pack --network kindgi --env-file pack.env \
  registry.localhost:5050/acme-pack@sha256:<your pack's digest>

And restart the runtime. While the two tokens differ, the pack's service refuses the runtime's calls, and your tools fail. The runtime's log says so:

  ⚠ Pack service at http://kindgi-pack:8080 isn't answering (pack-service-unauthorized: The pack service rejected the pack token). The server is up; pack tools and checks fail until it answers.

Once both sides have the same token, the runtime's calls go through again, with no further restart.

Rotate the license key

Section titled “Rotate the license key”

Replace KINDGI_LICENSE_KEY in kindgi.env with the new key, restart, and check the License line in the log:

  License: Docs example · non-production · until 2026-11-02

Rotate a signing key

Section titled “Rotate a signing key”

Rotate under a new key id. A key id stays bound to its public key, and a revoked id can't be trusted again.

  1. Create a key, and trust it:

    Terminal window
    pnpm exec kindgi key create acme-selfhost-2 --env selfhost
    pnpm exec kindgi key trust acme-selfhost-2 --url http://localhost:4000 --token "$KINDGI_API_TOKEN"

  2. Sign your next release with it. In the selfhost block of kindgi.config.ts:

    signingKey: '~/.kindgi/keys/acme-selfhost-2.pem',
    signerKeyId: 'acme-selfhost-2',

    Then build and deploy as usual, and run the new image as your pack's service (step 4). The first line keeps the old key's envelope, to check the revocation below:

    Terminal window
    cp .kindgi/build/deploy-envelope.json old-envelope.json
    pnpm exec kindgi build --local --push --env selfhost
    pnpm exec kindgi deploy --env selfhost --token "$KINDGI_API_TOKEN"
      Registering deployment
        ✓ POST /v1/deployments  →  201 Created
          deploymentId:    3c4f4d76-f4a4-4d5f-8339-f05b0497b462
          artifactVersion: 20261008.193855

    The artifact version defaults to the build time, YYYYMMDD.HHMMSS in UTC, so a second build the same day gets its own tag.

  3. Revoke the old key:

    Terminal window
    pnpm exec kindgi key revoke acme-selfhost --reason "rotated to acme-selfhost-2" \
      --url http://localhost:4000 --token "$KINDGI_API_TOKEN"
      ✓ Revoked acme-selfhost

    A revoked id can't be trusted again: kindgi key trust says so, and how to trust another key.

From then on, the runtime refuses a deploy signed by the old key. Here, an envelope it signed earlier:

Terminal window
pnpm exec kindgi deploy --env selfhost --from-envelope old-envelope.json \
  --idempotency-key redeploy-old --token "$KINDGI_API_TOKEN"
  Registering deployment
    ✗ POST /v1/deployments  →  HTTP 403
      code:    signer-not-trusted
      message: Signer key "acme-selfhost" is not on this tenant's trust list

Without --idempotency-key, the CLI's key is a hash of the envelope. Sending an envelope you already deployed then returns the first answer again (for 24 hours), and deploys nothing.

The runtime checks signatures when you deploy. A deployment the revoked key signed keeps running, through restarts too. The revoked key stays listed for audit:

Terminal window
curl -s "http://localhost:4000/v1/signing-keys?includeRevoked=true" -H "authorization: Bearer $KINDGI_API_TOKEN"

Each key in the list has its revokedAt and revokedReason, if it was revoked.

Rotate the public run token key

Section titled “Rotate the public run token key”

If browsers follow runs, the runtime signs their public run tokens with a key of its own. Make one:

Terminal window
openssl genpkey -algorithm ed25519 -out public-token-signing.pem
chmod 600 public-token-signing.pem

Give it to the runtime as a file: add this line to kindgi.env, and restart with the file mounted:

Terminal window
KINDGI_PUBLIC_TOKEN_SIGNING_KEY_PATH=/etc/kindgi/public-token-signing.pem
Terminal window
docker run -d --name kindgi-server --network kindgi --restart unless-stopped \
  --add-host registry.localhost:host-gateway \
  -v "$PWD/public-token-signing.pem:/etc/kindgi/public-token-signing.pem:ro" \
  -p 127.0.0.1:4000:4000 --env-file kindgi.env \
  quay.io/kindgi/runtime:0.1.6

The file must have mode 0600, and the runtime's user in the container (uid 10001) must be able to read it. The log says:

  Public run tokens: on (browsers follow runs; no CORS origins)

To rotate it, replace the file with a new key (the same two commands), and restart with the same mount. Tokens signed with the old key are refused from then on:

{"error":{"code":"auth-missing","message":"Bearer token is not recognized","requestId":"req-…"}}

Tokens minted after the restart work as before.