Ship a pack from CI

When you finish this page, every pull request runs your pack's tests, and every merge to main deploys the pack to staging. Prod then takes the same signed build once it passes your test cases and a reviewer approves. Each key CI holds reaches only what its step needs.

The example uses GitHub Actions and a TypeScript pack. Python, Java and Scala packs and Other CI systems say what changes.

On Kindgi Cloud

Kindgi Cloud builds the image, so CI needs no Docker and no registry login. The build and deploy steps take --hosted, and both get the deploy key. A hosted build takes a tenant admin's key, so on Kindgi Cloud the deploy key is one. It reaches prod too, so main's own protection is what guards prod there (The deploy key). Everything else is the same.

What CI ships, and what you set up once

CI ships what's in your pack: its agents, tools, flows, blocks and guardrails, built into one image and signed. Git is where they're kept. A change made on the runtime comes back to git as a pull request (Changes made on the runtime).

A person sets up the rest once, not CI: the environment order, trusting CI's signing key, providers, release gates, test cases and triggers. Projects, packs and values has how a pack, its projects and its values fit together.

Environments in order

One runtime holds a project for each environment, in an order: staging, then prod. Each environment is in the pack's kindgi.config.ts, under environments, with its endpoint and its project. A tenant admin sets the order once, by the pack's environment names:

pnpm exec kindgi environments set staging prod

Only the first environment takes a deploy. Each later one takes a promotion of the build already in the one before it, so prod runs exactly what ran in staging:

pnpm exec kindgi promote --from=staging --to=prod --check
pnpm exec kindgi promote --from=staging --to=prod --reason="Passed in staging"

• --check says what would move, and which values the build needs aren't set in prod. It changes nothing.
• Without it, the build moves to prod, with its agents' and flows' versions. Runs of each agent in prod keep the version Pinned for it there, until you pin the new one (Make the new version Pinned in prod).
• While a value isn't set in prod, both exit 1 and nothing moves. Set it, then promote again.

These are the steps CI runs, and without CI a person runs them the same way. kindgi environments and kindgi promote have every flag.

Set up each environment once

A person does these, with a key of their own:

• Trust CI's signing key, once for the tenant, with a key that may trust signing keys. The command is kindgi key trust <key-id>, as Trust your key and deploy shows; on Kindgi Cloud, Sign the pack. CI never holds a key that trusts signing keys.
• Register the providers each environment's agents call (kindgi providers).
• In prod, once the first build has reached it, pin a version of each agent, and publish its release gate (Gate a pin). Until kindgi promote has moved a build to prod, no version runs there, so a pin is refused with 409 nothing-runs-in-scope. After it, each agent's Latest in prod is the version the build ships, so pinning that version is a pin in place: no test run and no approval. A version the build only carries counts as Latest there when it was published in full, made Latest in staging, or published before the runtime recorded Latest. So an agent the build carries only as a version derived in staging, and never made Latest, has no Latest in prod until a build ships a version of it. A scope with a release gate needs a pinned version of its own. Without one, prod's runs take each agent's Latest in prod: the version the newest build promoted there shipped.
• Correct runs in prod's project: the agent's test set there is what prod's test run uses (An agent's test set).
• Create the triggers: schedules, webhooks and mail triggers. A trigger's toolsAllowedUnattended is checked against what runs in its project, so in prod, create one that lists an agent's tools once the first build has reached it.

Registering a provider again under its id, publishing a release gate's version again, or making a test set of test cases again under its id and version is refused, as already registered. A trigger has no name to match, so a second create of a schedule or a mail trigger, or a second register of a webhook, makes a second trigger. A script that sets these up checks first with each kind's list, and tells its triggers apart by --label.

The keys CI uses

Make each key for a service account of its own, made for that one key and given nothing else. A key reaches whatever its account may. So an account with other grants, or one key used for two environments, reaches all of them.

Each key expires in 90 days. To rotate one, make the new key, put it in CI, then revoke the old one with kindgi tokens revoke <token-id>. Service accounts has the rest about them.

The deploy key

The deploy key is for the first environment:

pnpm exec kindgi service-accounts create acme-ci-deploy --description="Deploys to staging from CI" --project=<staging-project-id>:admin
pnpm exec kindgi tokens create --for=sa:<service-account-id> --project=<staging-project-id> --label=ci-deploy --expires=90d

• A deploy into a project takes admin on it. The pack's staging environment names its project, so the account needs admin on that project, not on the tenant.
• --project limits the key to staging.
• Add --capability=secrets:write to the key only when CI sets secrets with kindgi deploy --sync-secrets.

On Kindgi Cloud, a hosted build takes a tenant admin's key, so the deploy key is one. Make its service account a tenant admin, and its key an admin key with no --project:

pnpm exec kindgi service-accounts create acme-ci-deploy --description="Builds and deploys to staging from CI" --tenant-admin
pnpm exec kindgi tokens create --for=sa:<service-account-id> --role=admin --label=ci-deploy --expires=90d

That key reaches the whole tenant, prod included. Even with no capability, it can change the environment order, replace prod's release gate, promote to prod, and make service accounts, tenant admins among them, and keys for them. So on Kindgi Cloud, prod's required reviewers don't guard prod from a job that holds it. Main's own protection does: a required review of each pull request before it merges.

• It's a secret of the staging environment only, and that environment takes only main.
• If it leaks, revoke it, then look for the keys and accounts it made with kindgi tokens list and kindgi service-accounts list, and revoke those too. Revoking doesn't undo what it changed, so check that kindgi environments list and kindgi release-gates list still show the order and the release gates you set.

A promote key for each later environment

pnpm exec kindgi service-accounts create acme-ci-promote-prod --description="Promotes staging to prod from CI" --tenant-member --project=<staging-project-id>:viewer --project=<prod-project-id>:admin
pnpm exec kindgi tokens create --for=sa:<service-account-id> --label=ci-promote-prod --expires=90d

• The key has no --project. A key limited to a project is refused on every promotion, because a promotion reads one project and writes another. This key reaches what its account may: read on staging, and admin on prod.
• --tenant-member, because kindgi promote lists staging's deployments, and that read takes read on the tenant.
• Admin on prod doesn't let it skip staging. Prod comes later in the order, so it refuses a deploy. That's why the order is set first.
• It can't approve its own promotion. A service account's key has no reviewer role.
• It's an admin of prod in everything except changing env values and secrets, which take capabilities it doesn't have. That includes reading prod's conversations. So its job waits for GitHub's required reviewers, and the key expires in 90 days.

Where the keys live

• Each key, and the signing key, is a secret of its own GitHub environment, staging or prod, never of the repository.
• Each environment takes only main, with its deployment branches set to main, and prod has required reviewers (Managing environments for deployment).
• The signing key is the pipeline's most valuable secret. It's trusted across the tenant, so a build it signs passes the signature check in every project. Whoever holds it can sign a build of any code, not only main's, and any key that can deploy then deploys it there. If it leaks, rotate it at once (Rotate a signing key).
• KINDGI_API_TOKEN is set on the kindgi steps only, never on the job, so no key is in the environment of pnpm install or of the install scripts it runs.
• What's installed is what you reviewed. A job's steps share one machine, so a process an install script leaves running could read what later steps write, such as the signing key's file. --frozen-lockfile installs only what the lockfile, reviewed in each pull request, names. pnpm runs a dependency's install script only when allowBuilds in pnpm-workspace.yaml allows it, and a pack kindgi init makes allows none.
• Each action is pinned to a commit, with its version in a comment. A tag can move to other code, and a commit can't. That holds in test too: it writes the pnpm cache that the jobs holding a key install from.

The signing key

CI signs each build with an Ed25519 key of its own. Make it once, on your machine:

pnpm exec kindgi key create acme-ci

It writes the private key to ~/.kindgi/keys/acme-ci.pem. Put that file's contents in the staging environment's KINDGI_SIGNING_KEY secret, and trust the key as Set up each environment once says.

In CI, the job writes the secret to a file for kindgi build. The signature names its key by the file's name without its extension, so the file is named after the key the tenant trusts: acme-ci.pem for acme-ci.

To replace the key, follow Rotate a signing key, then put the new key in the secret, and name the file the job writes after the new key.

The pipeline

• On a pull request: kindgi test runs the pack's tests. It needs no runtime and no Kindgi key. A test that calls a model provider needs that provider's key as a secret of the job. Make it a key of its own, with a spending limit, not the one your runtime uses: a pull request from any branch of the repository can read it before anyone reviews it.
• On a merge to main: build and sign the image, then deploy it to staging.
• Then prod, once GitHub's required reviewers let its job start: promote the build, run the test cases on the new version in prod, and make it Pinned there through its release gate.

Exit codes

Every step that asks a question exits 3 when the answer is no, so the job stops on it, and 1 when it couldn't ask:

Command	0	3	1
kindgi deploy, and with --dry-run	it deployed, or would	never	it would put back an older block than runs now, so nothing was deployed (Changes made on the runtime), or an error
kindgi promote --from --to, and with --check	it moved, or would	never	a value the build needs isn't set in the target, or an error
kindgi test-runs start --wait	the run completed	it failed or was cancelled	it's still running after 30 minutes, or an error
kindgi agents pin --wait	the version is Pinned	the release gate refused it, its approval was rejected or expired, the request was superseded, or the wait ran out	an error, or 3 reads of the pin in a row failed
kindgi promote --pin, and with --wait	every version it carries is Pinned	one isn't	a newer version of one of the build's blocks runs in the target, so nothing moved (Changes made on the runtime), an error, or 3 reads of a promotion in a row failed

A usage error exits 2.

The example

The pack's kindgi.config.ts has environments.staging and environments.prod, each with its endpoint, its project, its registry and its tenant id. The CLI is a dev dependency of the pack, and package.json pins its pnpm version (packageManager), as kindgi init sets it.

name: ship-pack
on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read # every job's GITHUB_TOKEN only reads the repository

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with: { persist-credentials: false }
      - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
        with: { node-version: 22, cache: pnpm }
      - run: pnpm install --frozen-lockfile
      - run: pnpm exec kindgi test

  deploy-staging:
    needs: test
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: staging # its own secrets; deployment branches: main
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with: { persist-credentials: false }
      - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
        with: { node-version: 22, cache: pnpm }
      - run: pnpm install --frozen-lockfile
      - uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
        with:
          registry: registry.acme.example # the staging environment's registry
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      - name: Signing key to a file
        env:
          SIGNING_KEY: ${{ secrets.KINDGI_SIGNING_KEY }}
        run: |
          umask 077
          printf '%s\n' "$SIGNING_KEY" > "$RUNNER_TEMP/acme-ci.pem"
      - run: pnpm exec kindgi build --local --push --env staging --signing-key "$RUNNER_TEMP/acme-ci.pem"
      - run: pnpm exec kindgi deploy --env staging --from-envelope .kindgi/build/deploy-envelope.json
        env:
          KINDGI_API_TOKEN: ${{ secrets.KINDGI_DEPLOY_KEY }}

  promote-prod:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: prod # required reviewers; deployment branches: main
    timeout-minutes: 120 # longer than the steps' own waits
    env:
      AGENT: acme.order-desk
      TEST_SET: acme.order-checks # the test cases, in prod's project
      PROD_PROJECT: <prod-project-id>
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with: { persist-credentials: false }
      - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
        with: { node-version: 22, cache: pnpm }
      - run: pnpm install --frozen-lockfile
      - id: promoted
        env:
          KINDGI_API_TOKEN: ${{ secrets.KINDGI_PROMOTE_KEY }}
        run: |
          pnpm exec kindgi promote --from=staging --to=prod --reason="${{ github.sha }} passed in staging" --json > promoted.json
          echo "version=$(jq -r --arg agent "$AGENT" '.promotion.moved.agents[] | select(.id == $agent and .carried != true) | .version' promoted.json)" >> "$GITHUB_OUTPUT"
      - id: tested
        if: steps.promoted.outputs.version != ''
        env:
          KINDGI_API_TOKEN: ${{ secrets.KINDGI_PROMOTE_KEY }}
          VERSION: ${{ steps.promoted.outputs.version }}
        run: |
          pnpm exec kindgi test-runs start "$TEST_SET" --project="$PROD_PROJECT" --agent="$AGENT" --agent-version="$VERSION" --wait > run.json
          echo "run=$(jq -r .testRunId run.json)" >> "$GITHUB_OUTPUT"
      - if: steps.promoted.outputs.version != ''
        env:
          KINDGI_API_TOKEN: ${{ secrets.KINDGI_PROMOTE_KEY }}
          VERSION: ${{ steps.promoted.outputs.version }}
          RUN: ${{ steps.tested.outputs.run }}
        run: pnpm exec kindgi agents pin "$AGENT" "$VERSION" --project="$PROD_PROJECT" --test-run="$RUN" --reason="${{ github.sha }} passed its test cases" --wait=1h

Excerpt

Make the new version Pinned in prod

The last three steps of promote-prod do it for one agent:

1. promote moves the build to prod. jq reads, from its JSON, the version of acme.order-desk the build ships. When the build carries it unchanged, the next two steps skip.
2. test-runs start --wait runs the test cases on that version in prod. The job stops when the run fails, and jq reads its id.
3. agents pin --wait asks prod's release gate, with that run. When the gate asks for an approval, the step prints the approval's id and the command that approves it, then waits for a reviewer, up to an hour. It exits 0 once the version is Pinned.

• A step that waits ends before the CI system ends it. test-runs start --wait waits up to 30 minutes, and exits 1 when the run is still going then. agents pin --wait=1h waits up to an hour: when it runs out, it exits 3, and when 3 reads of the pin in a row fail, it exits 1. Either way the approval stays open: once a reviewer approves it, the version is Pinned. Keep the job's timeout-minutes longer than the two waits together, as the example's 120 is.
• A carried agent the workflow doesn't name stays on the version Pinned for it in prod. Name each agent that should go further, with its own test run and agents pin steps.
• When prod's release gate asks only for an approval, with no test run, kindgi promote --from=staging --to=prod --pin --wait does the whole step for every agent the build carries. --pin can't name a test run, so a release gate that needs one holds every version it carries.
• --pin takes prod off Latest. A project in prod that ran its Latest runs the version --pin pins from then on, as with any pin. --pin is the ask, so nothing more is confirmed. A later build's version reaches its runs when a pin moves to it, or an unpin puts the project back on Latest.

Gate a pin has what a release gate checks, and how a reviewer approves. kindgi agents and kindgi test-runs have every flag.

Versions in git

• Bump an agent's or a flow's version in git when you change it.
• When its content changed under the same version, a deploy registers the next free version, and says so for each agent and flow (Deploys make the new version for you).
• A tool whose definition changed under the same version is refused, with 409 tool-already-registered, naming what differs. Give the tool a new version.

When a release goes wrong

• Unpin an agent in prod: one step, with no release gate in the way (Unpin).
• For anything else, revert the change in git and let the pipeline ship it. That rolls forward: each agent and flow whose content changed gets a new version.
• A tool goes back with the agent version that pins it.

Changes made on the runtime

Improve, suggested changes and edits in the console change the runtime, not git, so the next deploy from git doesn't have them. Bring each one back as a pull request: copy the change into the pack's files, and open the pull request. For blocks, kindgi blocks pull writes the newest version running anywhere into each block's file. When a block's file holds edits that aren't in its version on the runtime, or a version the runtime doesn't have, it names the file and writes nothing, so it never drops them. Improve an agent has what Improve changes.

kindgi deploy run in a git work tree, as CI's is, also says where the pack lives: the origin remote with no user or token in it, the pack's folder in the repository, the commit and branch, and whether the pack's folder had changes the commit doesn't hold. Each deployment keeps that as source, with each block's file, and lists the block versions it shipped in contents.blocks (GET /v1/deployments/:deploymentId). A deploy from outside git keeps no source.

A deploy doesn't undo one without saying so. Before an image is built, kindgi deploy compares each block in the pack's files with the newest version of it that runs now, in any project of the pack's agents: pinned, for a segment too, or Latest. When the files' version is older, the deploy stops, deploys nothing, and exits 1:

  Checking returns-helper against what runs…
✗ Stopped: this deploy would put back older blocks than run now.

  Block                 In your files   Runs now
  returns-helper.tone   3.0.0           4.0.0  latest in Support sandbox (Ana, 11-05)

  Nothing was deployed.
  To keep the newer versions:  merge https://github.com/acme/returns/pull/214, or run `kindgi blocks pull`, then deploy again.
  To drop them on purpose:  kindgi deploy --env=prod --replace=returns-helper.tone --reason=<why>

Excerpt

• Every project counts, whoever deploys. For a project the deploy key can't read, the row names the version only: "in a project you can't read".
• The same version passes, and so do a newer one in the files and a block that runs nowhere. Code isn't compared.
• To drop a newer version on purpose, name its block with --replace (repeatable), and say why with --reason, in 1 to 500 characters. The deployment keeps each block it put back (replaced: the version that ran, the files' one, the reason and who deployed), and the audit log keeps one deployment-replaced-block event for each. A --replace that nothing newer runs for says "nothing newer runs; --replace not needed".
• --dry-run checks too, and a stop exits 1 there as well, so a job can check before it deploys.
• The runtime holds the same rule. A deploy that calls POST /v1/deployments itself stops with 409 deploy-would-put-back-older-blocks, each block in details.blocks, unless its body names the block in replace. POST /v1/deployments/check gives the same answer without deploying. A runtime that doesn't read pins and Latest answers it with 501 runs-in-unsupported, and doesn't check deploys either: there, kindgi deploy deploys as before, and says so in one line. A key the check refuses (403) goes on without it too, and says so: the deploy asks the same right as the check, so a runtime with the check refuses the deploy the same way.
• A promotion that pins is checked too, against what runs in the target environment only: kindgi promote --pin puts the build's blocks there. A stop prints the same table ("Nothing was promoted.") and exits 1, with nothing moved. --replace and --reason go with --pin there as well. The deployment the promotion makes keeps the blocks its pins put back (replaced, with who promoted), never the ones the build put back where it was deployed. The audit log keeps one deployment-replaced-block event for each block. A build deployed before deployments kept contents.blocks isn't compared, and kindgi promote says so.

Separate runtimes

There's no promotion between runtimes. A signed build names its tenant, and a runtime checks the signature against its own tenant. So build and sign from the same commit for each runtime, and deploy each build to its runtime's first environment. Set up each runtime once: its trusted signing key, providers, release gates, test cases and triggers.

Python, Java and Scala packs

The jobs are the same. What changes is how each job sets up the pack and runs the CLI:

• Python: set up uv in place of pnpm and Node, and run uv sync --locked. Run the tests with uv run pytest, and uv run kindgi where this page runs pnpm exec kindgi.
• Java: set up a JDK 17 or later. Run the tests with ./mvnw test, and ./kindgiw where this page runs pnpm exec kindgi.
• Scala: set up a JDK 17 or later and sbt. Run the tests with sbt test, and ./kindgiw where this page runs pnpm exec kindgi.

Other CI systems

Any CI system runs the same jobs. Keep each key in a secret that only main's protected jobs read, and set KINDGI_API_TOKEN on the kindgi steps only.

Next

• People, API keys and service accounts: the service accounts and keys CI uses, and how to revoke one.
• Release an agent version: pin, unpin and gate a release by hand.