Deploy on Azure Container Apps

When you finish this page, the Kindgi runtime and your pack's service run as two container apps in an Azure resource group you already have, with everything around them, from Kindgi's Terraform module. You then deploy your pack to it and run a flow.

In the verification run, the first apply took about seven minutes, most of it PostgreSQL. The services took under three more. A new runtime replica was ready 10 seconds after its container started, its first migrations included.

Access to the runtime image

Sign in at access.kindgi.com with GitHub or your email address for a license key. The key is also the runtime image's password (Pull the runtime image). Questions or trouble: contact@kindgi.com.

What you'll have

• The runtime (<name_prefix>-server): Kindgi's server, a container app with its own user-assigned managed identity. External ingress, port 4000, one replica.
• Your pack's service (<name_prefix>-pack): your tools' code. It runs only the tool versions in its image's build (A self-hosted runtime runs what your pack has). Environment-internal ingress: only apps in the same Container Apps environment reach it, and the runtime sends the shared pack token on every call.
• PostgreSQL Flexible Server (16, with pgvector) for the runtime's data, on a private subnet with no public access.
• Azure Container Registry for both images. The runtime also reads your pack's image from it, with its own identity, when you deploy.
• Key Vault for the secrets both apps read, and an RSA key that wraps the secrets the runtime stores (model API keys, webhook secrets).
• A VNet with a subnet for the environment and a delegated one for PostgreSQL, or two subnets in a VNet you already have (A new VNet, or yours).
• Log Analytics for both apps' logs.

One tenant: the runtime serves the tenant it seeds (seed_tenant_id), and the plan refuses KINDGI_TENANCY or KINDGI_PACK_ROUTING other than single. Serving many tenants needs a container app for each deployment's pack service, run by an orchestrator of your own, which this module doesn't make.

Who calls whom:

• your app calls the runtime, with its API token;
• the runtime calls your pack's service, with the pack token, and PostgreSQL, Key Vault and model APIs;
• kindgi deploy calls the runtime with a signed envelope, and the runtime reads the pack's image from the registry.

Before you start

• An Azure resource group, and Owner on it, or Contributor plus User Access Administrator: the module grants roles to its identities. It never creates the group, and never writes at the subscription level.

• The resource providers registered, once per subscription, by someone allowed to:

  for p in Microsoft.App Microsoft.OperationalInsights Microsoft.ContainerRegistry \
           Microsoft.DBforPostgreSQL Microsoft.KeyVault Microsoft.ManagedIdentity Microsoft.Network; do
    az provider register -n $p
  done

• A second resource group, made by Azure: a Container Apps environment on a VNet keeps its infrastructure in a platform-managed group named <your group>-<name_prefix>-infra. Azure creates it with the environment and deletes it with the environment. If your policies forbid new resource groups, exempt this one.

• Terraform 1.11 or later, since the database's admin password is write-only. Also the Azure CLI, Docker with buildx, jq, and your pack's kindgi CLI, 0.1.6 or later. The module needs the azurerm provider 5.9 or later, and its lock file pins it.

• For kindgi commands: Kindgi 0.2's CLI, which runs from any folder as npx --yes @kindgi/cli@0.2 (Install). Inside a pack project, pnpm exec kindgi runs the project's pinned CLI instead. The build, env plan and deploy steps need the pack's folder, so they use pnpm exec kindgi.

• Runtime 0.1.6 or later: the first with Key Vault keys and the registry's managed identity.

• A license key: a non-production key covers staging, and it also pulls the runtime image. Get one at access.kindgi.com, or from contact@kindgi.com.

• Docker logged in to registry.kindgi.com with that key (Pull the runtime image).

• A pack that kindgi build --local --push builds, with an environment block for this deployment in its config, whose registry is the module's registry (step 2 prints it).

• Your Azure Policy assignments checked, with az policy assignment list -g <rg> --disable-scope-strict-search. Allowed locations, required tags, or a rule against public network access change the recipe.

1. The foundation

This step creates everything the two apps need, before their images and secrets exist. The module is deploy/azure-container-apps/ in the kindgi-sdk repository: copy the folder from the release you run. Its README.md lists every variable.

Terraform's state goes in a storage container of its own:

az storage account create -n <state account> -g <rg> --sku Standard_LRS \
  --allow-blob-public-access false --min-tls-version TLS1_2
az storage container create -n tfstate --account-name <state account> --auth-mode login
az role assignment create --assignee <you> --role "Storage Blob Data Contributor" \
  --scope "$(az storage account show -n <state account> -g <rg> --query id -o tsv)"

• <state account> is a new storage account's name, and <rg> your resource group.
• --allow-blob-public-access false and --min-tls-version TLS1_2 keep the state private, and over TLS 1.2 or later.
• --auth-mode login creates the container as you, with your Entra ID sign-in.
• <you> is your own user, such as your email: the role lets you read and write the state.

Give each deployment its own state key, never shared with your app's own infrastructure.

The services need images and secret values that the next steps make, so the first apply creates everything else:

cp example.tfvars dev.tfvars   # subscription_id, resource_group_name, kindgi_env, the seed ids, …
terraform init -backend-config=resource_group_name=<rg> -backend-config=storage_account_name=<state account> \
  -backend-config=container_name=tfstate -backend-config=key=<this deployment>.tfstate -backend-config=use_azuread_auth=true
terraform apply -var-file=dev.tfvars \
  -target=azurerm_container_app_environment.kindgi \
  -target=azurerm_postgresql_flexible_server_database.kindgi \
  -target=azurerm_postgresql_flexible_server_configuration.extensions \
  -target=azurerm_role_assignment.server_wraps \
  -target=azurerm_role_assignment.server_pulls \
  -target=azurerm_role_assignment.pack_pulls

Excerpt

• dev.tfvars holds this deployment's variables. example.tfvars lists them.
• -backend-config tells init where the state goes: the storage account and container you made, and <this deployment>.tfstate, its own key. use_azuread_auth=true reads it as you, with no storage key.
• -target limits this apply to the environment, the database and the roles, with what they need.

The image variables need only a digest-shaped placeholder here. secrets_aad_key_version and erasure_ledger_key_version may stay empty. The plan creates 22 resources. In the verification run, the apply took 6 minutes 47 seconds. The database's admin password is write-only: Terraform sends a random one that nobody keeps, and it never reaches the state. You set the real one in step 3.

The vault has purge protection on: key_vault_purge_protection, true by default. Every secret the runtime stores is wrapped by the vault's key. So a deleted key, secret or vault stays recoverable for key_vault_soft_delete_retention_days, 90 by default, and nobody can purge it sooner, an owner included. Once on, purge protection can't be turned off.

A new VNet, or yours

• A new VNet, the default in example.tfvars: the module creates the VNet, a subnet for the environment, and a delegated subnet and a private DNS zone for PostgreSQL. Pick it when your tools reach only the internet, or nothing outside your pack.
• Yours, in example-existing-vnet.tfvars: give it two empty delegated subnets and a private DNS zone, as environment_subnet_id, database_subnet_id and database_private_dns_zone_id. The module then creates no network. Pick it when your pack's tools reach private resources in that VNet: an app database, an internal service, a private endpoint.

In both, the environment holds only Kindgi's two apps. The private DNS zone and its link get your tags except any whose name has a colon: Azure ignores those there.

2. The images into the registry

The two apps run from images in the module's registry, which they pull with their managed identities, never a password. With Docker logged in to registry.kindgi.com, copy the runtime image into the module's registry by digest, with every platform it was built for:

ACR=$(terraform output -raw image_registry)
az acr login -n "${ACR%%.*}" --expose-token --only-show-errors --query accessToken -o tsv \
  | docker login "$ACR" --username 00000000-0000-0000-0000-000000000000 --password-stdin
docker buildx imagetools create --tag "$ACR/runtime:0.3.0" \
  registry.kindgi.com/kindgi/runtime:0.3.0@sha256:1f5cfe0032bb82b467781949978719d6a3da4d500cd994471f2f2e009c9a1b9f

• ACR is the registry's address, from the foundation's output, and ${ACR%%.*} its name, for az acr login.
• --expose-token has az print the registry's token instead of logging Docker in, and --query accessToken -o tsv keeps the token alone, which goes straight to docker login's stdin. A plain az acr login runs docker login --password <token>, which puts the token on a command line other processes on the machine can read. 00000000-0000-0000-0000-000000000000 is the username az's token goes with.
• imagetools create --tag copies the image by digest, every platform included, under the tag you give.

The copy keeps the release's digest. A plain docker pull, tag and push from an Apple silicon machine pushes only the arm64 image, which Container Apps can't run.

The copy goes through the machine you run it on, every layer down and back up, and takes both platforms (amd64 and arm64), so on a slow uplink it can take most of an hour.

Then build and push your pack's image, signed:

pnpm exec kindgi build --local --push --env dev

• --local builds the image on your machine.
• --push pushes it to the registry of your pack's dev environment block, here the module's.
• --env dev names that environment block.

It prints the build as JSON, then each step. The last steps:

…
    ✓ Pushed acmeep0y.azurecr.io/acme@sha256:4d18e695…

  Verifying build output
    ✓ /app/index.json in the image matches the local index byte for byte

  Signing locally
    ✓ Ed25519 signature over (imageDigest, artifactVersion, indexHash, tenantId, publishedAt)
      key: acme-dev

  Deploy envelope written to …/.kindgi/build/deploy-envelope.json
  Run `kindgi deploy --env=dev` to POST it.

Excerpt

Set server_image and pack_image in dev.tfvars to the two images by digest, …@sha256:….

3. The secrets

The apps read their secrets from Key Vault. Make each value here and pipe it in, straight into Key Vault or, for the database password, into az: it's never on disk, on a command line or in Terraform's state, and Kindgi generates none of them. Don't run these commands with --debug: it prints each request's body, values included.

# Off in this shell: az's log file, if it's on, records each request's body.
export AZURE_LOGGING_ENABLE_LOG_FILE=no

V=$(terraform output -raw key_vault_name)
PG=$(terraform output -raw database_server_name)
FQDN=$(terraform output -raw database_fqdn)
put() { az keyvault secret set --vault-name "$V" -n "$1" --file /dev/stdin --query id -o tsv; }

# Kindgi's database login: set its password, then the URL with it.
DBPW=$(openssl rand -hex 24)
printf '%s' "$DBPW" | az postgres flexible-server update -g <rg> -n "$PG" --admin-password @- -o none
printf 'postgres://kindgiadmin:%s@%s:5432/kindgi?sslmode=require' "$DBPW" "$FQDN" | put database-url
unset DBPW

# The token the runtime and the pack's service share.
openssl rand -hex 32 | tr -d '\n' | put pack-service-token

# The first API token.
printf 'kgi_bt_%s' "$(openssl rand -hex 32)" | put api-token

# The AAD key and the erasure ledger's key: 32 bytes, base64. Each prints the
# secret's id; its last segment is the version to pin.
openssl rand 32 | base64 | tr -d '\n' | put secrets-aad-key
openssl rand 32 | base64 | tr -d '\n' | put erasure-ledger-key

# The key that signs public run tokens.
openssl genpkey -algorithm ed25519 | base64 | tr -d '\n' | put public-token-key

# Only with export_signing = "secret": the key that signs exports, base64.
openssl genpkey -algorithm ed25519 | base64 | tr -d '\n' | put export-signing-key

# The license key, pasted, never echoed.
read -rs LICENSE_KEY && printf '%s' "$LICENSE_KEY" | put license-key && unset LICENSE_KEY

• put stores what it reads on its input as the Key Vault secret its argument names, and prints the secret's id.
• --admin-password @- reads the database password from its input.
• AZURE_LOGGING_ENABLE_LOG_FILE=no keeps az's log file off in this shell, whatever az's configuration says.
• openssl rand -hex 24 and openssl rand -hex 32 make random values, and openssl rand 32 | base64 32 random bytes in base64.
• openssl genpkey -algorithm ed25519 makes an Ed25519 private key.
• read -rs reads the license key you paste, without showing it.

Put the two keys' versions in secrets_aad_key_version and erasure_ledger_key_version. The server reads those versions, never the latest. Every secret stored in Postgres is bound to the AAD key, and erasures replay after a backup restore only with the same ledger key. The module refuses anything else:

Error: Invalid value for variable
…
secrets_aad_key_version: a 32-hex Key Vault secret version id.

Excerpt

and, left empty, at the services' apply:

Error: Resource precondition failed
…
secrets_aad_key_version is needed for the services: the version `az keyvault
secret set` printed for secrets-aad-key (README, step 3).

Excerpt

Signed exports are off by default, with export_signing = "none":

• With "secret", the server signs with the key you put in the vault as export-signing-key. It's Ed25519, or an EC P-256 key from openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256.
• With "kms", the module makes an EC P-256 key in the vault, and the server signs there, so the private key never leaves Key Vault. Exports are then ecdsa-p256-sha256, since Key Vault has no Ed25519.

See Export signed evidence.

The database login is the server's admin, a member of azure_pg_admin with CREATEROLE, which is what the runtime needs.

Your pack's own secrets: kindgi env plan --env dev lists what your pack needs. Create each one in the same vault with put. The module passes them to the pack's service, which gets read access to exactly those:

pnpm exec kindgi env plan --env dev > pack-env.json
jq '{pack_env: .env, pack_secret_env: .secret_env}' pack-env.json > dev.pack-env.auto.tfvars.json

jq turns the plan's env and secret_env into the module's pack_env and pack_secret_env variables. Terraform reads *.auto.tfvars.json only from the module's directory, and there it reads each such file by itself, for every plan and apply. So keep the name's ending: a file such as dev.pack-env.tfvars.json is read only with -var-file, and a plan made without it takes the pack's env away.

4. The services

With the images and the secrets in place, apply the whole module. It creates the two container apps:

terraform apply -var-file=dev.tfvars

The module waits a minute after granting the apps' roles before it creates them, so they start with the roles in place. In the verification run, the apply took 2 minutes 43 seconds. The runtime reached Key Vault and the pack's service on its first start, with no warning.

The runtime logs JSON, so its startup lines are the lines of one record, Kindgi runtime ready, in Log Analytics:

WS=$(az monitor log-analytics workspace list -g <rg> --query "[0].customerId" -o tsv)
az monitor log-analytics query -w "$WS" --analytics-query \
  "ContainerAppConsoleLogs_CL | where ContainerAppName_s == '<name_prefix>-server' and Log_s contains 'Kindgi runtime ready' | top 1 by TimeGenerated" \
  --query "[0].Log_s" -o tsv | jq -r '.lines[]'

• WS is the Log Analytics workspace's id.
• --analytics-query finds the runtime's newest Kindgi runtime ready record. <name_prefix> is your name_prefix variable.
• --query "[0].Log_s" -o tsv prints that record, and jq -r '.lines[]' its lines.

The first time, the CLI installs its log-analytics extension. A new replica's lines take a minute or more to reach the workspace. az containerapp logs show -n <name_prefix>-server -g <rg> --type console streams the running replica's lines at once.

Before them, the runtime checks the key:

KMS probe OK (azure-key-vault): azure-key-vault 7.5 (wrap/unwrap round trip, RSA-OAEP-256, key acme-secrets version 28fedcfb…) (999ms)

Excerpt

Then the lines name the license and the pack's service it reached:

  License: … · non-production · until 2026-11-20
  Env: dev (tool secrets resolve in it)
  …
  Pack service: https://acme-pack.internal.… — acme (artifact 20261008.211554), protocol 2, 3 tools, 1 check

Excerpt

Keep the pack's service warm when response time matters. The default, pack_min_instances = 1, keeps one replica running, so the first tool call after a quiet spell doesn't wait for one to start. That replica runs around the clock (Azure Container Apps pricing). pack_min_instances = 0 lets the service scale to zero when it's idle, which suits a dev deployment: the first tool call after that waits for a replica to start.

How the runtime calls your pack's service

The pack's service has environment-internal ingress, and the environment holds only Kindgi's two apps. From outside, its address answers 404. Inside, a call without the pack token is refused by the pack's service:

{"error":"Bad pack token"}

with status 401. KINDGI_PACK_SERVICE_AUTH=token makes the runtime send the token on every call.

Requests longer than 240 seconds

Container Apps ends HTTP requests at 240 seconds (Microsoft: Ingress, HTTP). A start that waits for a run longer than that gets this at 240 seconds:

HTTP/2 504
content-length: 14
content-type: text/plain
…

stream timeout

Excerpt

The run itself carries on: in the verification run, it completed 30 seconds later. Start a run that can take longer in the background, with wait: false, and follow it, as kindgi runs start does. It followed another run of the same flow to its end, 273 seconds later. For the same reason, the module refuses a pack_call_timeout_ms of 240000 or more: the runtime's wait for one tool call.

Client addresses

Container Apps' ingress appends the caller's address to X-Forwarded-For (Microsoft: HTTP headers). The module sets KINDGI_TRUSTED_PROXIES=1, from trusted_proxies, so rate limits and audit records see the caller, not the ingress. If you put Front Door or Application Gateway in front, set trusted_proxies = "2" (Behind a load balancer or ingress).

5. Trust your key, and deploy

The runtime accepts only deploys signed by a key it trusts. Trust your pack's key, then deploy:

URL=$(terraform output -raw server_url)
export KINDGI_API_TOKEN=$(az keyvault secret show --vault-name "$V" -n api-token --query value -o tsv)
npx --yes @kindgi/cli@0.2 key trust acme-dev --label dev --url "$URL"

It prints the key the runtime now trusts, then says so:

{
  "keyId": "acme-dev",
  "tenantId": "722e23b8-bbb4-43dc-9a37-9640e369fa7d",
  "algorithm": "ed25519",
  "publicKey": "YoQxkUZcrXvo7NKCJmalfcMcGWMfNJzBhk+nhNctQBE=",
  "label": "dev",
  "createdAt": "2026-10-12T14:28:40.122Z"
}

  ✓ Trusted acme-dev (sha256:1b5dcdf07dacf4e91c1bab5b)
    The runtime now accepts deploys this key signs.

• acme-dev is your signing key's id, from ~/.kindgi/keys/. --label is a label the runtime keeps with it.
• --url is the runtime's address. The commands send the API token from the vault, exported as KINDGI_API_TOKEN, which keeps it off the command line. kindgi deploy takes the address as --endpoint.
• --env dev deploys the pack's dev environment block.

pnpm exec kindgi deploy --env dev --endpoint "$URL"

It prints the deployment as JSON, then each step. The last steps:

…
  Registering deployment
    ✓ POST /v1/deployments  →  201 Created
      deploymentId:    8ae8c6d0-…
      artifactVersion: 20261008.211554
      primitives:      3 tools, 1 guardrail, 1 agent, 2 flows
      activatedAt:     2026-10-08T21:22:04.191Z

  Deploy complete.
  Runs started from now use image sha256:4d18e69581b56… (artifactVersion 20261008.211554).
  With one pack service, runs already in flight call that service's current build, as new runs do: a call naming a tool version that build doesn't have fails with tool-version-mismatch.

Excerpt

To deploy a client's pack into its own project, add --project <id or slug> (Deploy a client's pack into its project).

The runtime read your pack's image from the registry with its own identity (KINDGI_IMAGE_REGISTRY_AUTH=azure) to check it before registering it. The module sets KINDGI_IMAGE_REGISTRY_HOST to the registry's login server. If you set it yourself, give the login server alone: <name>.azurecr.io, or .azurecr.cn / .azurecr.us in those clouds. The runtime exchanges its Entra token at https://<host>/oauth2/exchange, so the host decides where that token goes. Anything else, such as a scheme, a port or a path, stops the runtime at startup with exit code 2:

KINDGI_IMAGE_REGISTRY_AUTH=azure signs in to an Azure Container Registry: KINDGI_IMAGE_REGISTRY_HOST must be its login server (<name>.azurecr.io, or .azurecr.cn / .azurecr.us in those clouds), with no scheme, port or path. Got "https://myregistry.azurecr.io".

6. Check it

Check the runtime answers, lists your pack's tools, and runs a flow:

curl "$URL/health"                                  # {"ok":true}
npx --yes @kindgi/cli@0.2 tools list --url "$URL"   # your pack's tools
npx --yes @kindgi/cli@0.2 runs start --flow=acme.greet-flow --input='{"name":"Azure"}' --url "$URL"

--flow names the flow to run, and --input is its input, as JSON.

In the verification run, that flow (one tool step) completed in 399 ms, start to finish, with the output {"greeting":"Hello, Azure!"}.

7. Turn on sign-in

Nobody can sign in to the console until you turn sign-in on. The API, the CLI and the SDKs take API tokens either way. Until then, the startup lines say so:

  ⚠ Console sign-in: nobody can sign in to the console. KINDGI_CONSOLE_TOKEN_SIGN_IN=on allows an API token; KINDGI_AUTH_SECRET_PATH turns on sign-in with identity providers.

The runtime's own settings go in two variables, as the pack's do: server_env for plain values, and server_secret_env for Key Vault secrets, read with the runtime's identity. You create each secret in the module's vault with put (3. The secrets), and the runtime gets read access to exactly those. The plan refuses a name the module sets itself (its own variables cover those, such as public_url below), KINDGI_DEV, a name in both maps, and a secret given as a plain value: a name with SECRET in it, or the SMTP URL.

With an API token, which the console's sign-in page then takes:

server_env = {
  KINDGI_CONSOLE_TOKEN_SIGN_IN = "on"
}

Excerpt

With an emailed link, you need the runtime's sign-in secret, an SMTP server and public_url:

# In the shell from step 3: it defines `put` and keeps az's log file off. No `--debug`.
# The sign-in secret: 32 bytes, base64.
openssl rand 32 | base64 | tr -d '\n' | put auth-secret
# The SMTP URL, password included: paste it; it never goes in a file.
read -rs SMTP_URL && printf '%s' "$SMTP_URL" | put smtp-url && unset SMTP_URL

public_url = "https://kindgi-server.<environment domain>" # terraform output -raw server_url
server_env = {
  KINDGI_CONSOLE_TOKEN_SIGN_IN = "on"
  KINDGI_AUTH_EMAIL_FROM       = "kindgi@acme.example"
}
server_secret_env = {
  KINDGI_AUTH_SECRET         = { secret = "auth-secret", version = "<the version put printed>" }
  KINDGI_AUTH_EMAIL_SMTP_URL = { secret = "smtp-url", version = "<the version put printed>" }
}

Excerpt

Pin each secret to a version, as here, rather than latest: a new version then reaches the runtime only when you change version, not whenever Container Apps picks it up.

• public_url (KINDGI_PUBLIC_URL) is the address people open the console at: terraform output -raw server_url after the first apply, so the emailed link comes with a second apply, or your own domain in front. Console sessions are then accepted from that address only. The plan refuses KINDGI_AUTH_SECRET without it, because the runtime wouldn't start. A tool that pauses hands other systems a link made from it too (On a server you run).

With Continue with Google, you need the sign-in secret and public_url, as for the emailed link, and an OAuth client of type Web application in the Google Cloud console (Google). Its authorized redirect URI is public_url followed by /auth/kindgi/social/callback/google:

https://kindgi-server.<environment domain>/auth/kindgi/social/callback/google

Store the client's secret, then give the runtime the client's ID and the secret:

# In the shell from step 3. The Google client's secret: paste it; it never goes in a file.
read -rs GOOGLE_SECRET && printf '%s' "$GOOGLE_SECRET" | put google-client-secret && unset GOOGLE_SECRET

public_url = "https://kindgi-server.<environment domain>"
server_env = {
  KINDGI_AUTH_GOOGLE_CLIENT_ID = "<client id>"
}
server_secret_env = {
  KINDGI_AUTH_SECRET               = { secret = "auth-secret", version = "<the version put printed>" }
  KINDGI_AUTH_GOOGLE_CLIENT_SECRET = { secret = "google-client-secret", version = "<the version put printed>" }
}

Excerpt

• <client id> is the client's ID in the Google Cloud console, ending in .apps.googleusercontent.com.
• Microsoft and GitHub go the same way, with MICROSOFT or GITHUB in the names, and a redirect URI ending in microsoft or github (Continue with Google, Microsoft or GitHub). So do verified domains and Turnstile: KINDGI_AUTH_VERIFIED_DOMAINS and the Turnstile site key in server_env, and KINDGI_AUTH_TURNSTILE_SECRET in server_secret_env.
• Several ways in share the two maps: with the emailed link too, they hold both sets of settings.
• The module mounts no files, so use the value forms of the settings, such as KINDGI_AUTH_GOOGLE_CLIENT_SECRET, not their …_PATH forms.

Check it worked: after the apply, the new revision's startup lines (4. The services) say which ways in the console has, instead of the warning. With Continue with Google alone:

  Console sign-in: identity providers (KINDGI_CONSOLE_TOKEN_SIGN_IN=on also allows an API token)
  …
  Continue with: Google (the deployment's own apps)

Excerpt

What each setting does, and the ways in: Turn on sign-in.

The roles it sets up

Who	Role	On
The runtime's identity	Key Vault Crypto Service Encryption User	the key that wraps stored secrets (the startup check wraps and unwraps with it)
	Key Vault Secrets User	each of its secrets
	AcrPull	the registry
	Key Vault Crypto User, only with export_signing = "kms"	the export signing key
The pack's identity	Key Vault Secrets User	the pack token and your pack's secrets
	AcrPull	the registry
Whoever runs Terraform, and key_vault_admins	Key Vault Secrets Officer, Key Vault Crypto Officer	the vault

Operate it

• Upgrade at least every 90 days. A preview release stops starting 90 days after it was released. Container Apps starts new replicas on its own, and after that day none comes up. The runtime warns from 30 days before, at startup and once a day, in a log record with "event": "kindgi.preview_expiry": alert on it in Log Analytics (Licensing).

• Upgrade: Upgrade has the steps. Migrations only go forward: never run two runtime versions on one database, and go back by restoring the backup.

• Rotate a secret: add a version with put again. The apps read their secrets when they start. The module references every secret but the two pinned keys without a version. So a replica that starts after Container Apps picks up the new version gets it (Microsoft: Key Vault secret URI and secret rotation says when that happens). Never rotate the AAD key or the erasure ledger's key this way: they're pinned.

• Rotate the wrapping key: with key_rotation_days, 90 by default, Key Vault adds a key version on schedule. New secrets use it, and the ones stored before keep unwrapping with theirs. In the verification run, a secret stored before az keyvault key rotate read back the same after it.

• Renew the license key: enroll once, then run $(terraform output -raw license_renew_command) by hand or from your scheduler. Or set license_renewal_schedule, a cron in UTC, and the module runs it as a job of its own. It comes with an alert for a failed run, and one for a key within 7 days of its end (Renew the license key). With license_usage_key_secret too, its renewals carry counts of the deployment's use (A key for the counts), and an alert says when a run sent none.

• A key the runtime can't use: a new replica exits with code 2 and says why, and what to check:

  KMS probe failed at boot: kms-unauthorized: Azure Key Vault refused probe (HTTP 403): Forbidden: Caller is not authorized to perform action on resource.
  …
  . Check KINDGI_SECRETS_AZURE_KEY_ID (an RSA key's URL, without a version) and KINDGI_AZURE_CLIENT_ID (the runtime's managed identity), that the identity has the Key Vault Crypto Service Encryption User role on the key or its vault, and that the vault's network rules let the runtime in.

  Excerpt

  In the verification run, the replica that couldn't start didn't replace the one serving. Its lines are in Log Analytics:

  az monitor log-analytics query -w "$WS" --analytics-query \
    "ContainerAppConsoleLogs_CL | where ContainerAppName_s == '<name_prefix>-server' | order by TimeGenerated desc | take 50"

• Logs: Log Analytics, per app, in ContainerAppConsoleLogs_CL. The platform's events, such as image pulls and restarts, are in ContainerAppSystemLogs_CL. Filter the runtime's records by traceId to follow one request or run (Logs).

Upgrade

The module's upgrade.sh runs these steps for you: a deployment of the module on runtime 0.1.8 or later, in either network shape, can run it to 0.2.0 or later. Use the upgrade.sh as of the @kindgi/sdk@<version> tag of the release you're upgrading to: it knows the steps up to its own release. It first came with the @kindgi/sdk@0.4.0 tag, so to keep the module at an earlier release, upgrade by hand. The module's README has what it checks and changes, under Upgrading, and ./upgrade.sh --help lists its options.

Upgrade by hand instead if you'd rather: run these steps on a staging deployment first if you have one. Run them in the module's folder, as the identity that deploys the module. <rg> is the resource group, and <name_prefix> your name_prefix.

1. Read the upgrade steps of each release you're upgrading across, in its release notes (Upgrade the runtime links them), and note the new release's digest: docker pull registry.kindgi.com/kindgi/runtime:<version> prints it (Pull the runtime image).

2. Copy the new runtime image into the module's registry by digest, every platform included, as in 2. The images into the registry, with Docker logged in to registry.kindgi.com (it has 0.2.0 and later):

   ACR=$(terraform output -raw image_registry)
   az acr login -n "${ACR%%.*}" --expose-token --only-show-errors --query accessToken -o tsv \
     | docker login "$ACR" --username 00000000-0000-0000-0000-000000000000 --password-stdin
   docker buildx imagetools create --tag "$ACR/runtime:<version>" \
     registry.kindgi.com/kindgi/runtime:<version>@sha256:<digest>

3. Check the database can go back. The module keeps automatic backups with point-in-time restore, for database_backup_retention_days (7 by default). This prints the earliest time a restore can go back to:

   PG=$(terraform output -raw database_server_name)
   az postgres flexible-server show -g <rg> -n "$PG" --query backup.earliestRestoreDate -o tsv

4. Set the new image: server_image in dev.tfvars, to $ACR/runtime@sha256:<digest>. If the release notes say to rebuild the pack, rebuild it as in 2. The images into the registry and set pack_image too. Keep a copy of the file as it was: it's your way back.

5. Plan, and save the plan:

   (umask 077 && terraform plan -var-file=dev.tfvars -out=upgrade.tfplan)

   Terraform also reads every *.auto.tfvars.json in the folder, your pack's env among them. Read the plan: it should change the server's image, the license renewal job's too with license_renewal_schedule, and what the release notes say. A plan that removes the pack's env was made without its file (3. The secrets).

   The plan holds your variables' values: umask 077 keeps it readable by you alone, and you delete it once it's applied.

6. Note the time, then apply exactly that plan:

   date -u +%Y-%m-%dT%H:%M:%SZ
   terraform apply upgrade.tfplan
   rm upgrade.tfplan

   • date -u prints the time, in UTC, that a restore goes back to if you go back. Write it down. Writes made after it aren't in the restore.
   • A saved plan applies as you read it, with the variables it was made with, so it takes no -var-file. If the state changed since, Terraform refuses it as stale: plan again.
   • The new revision migrates the database as it starts. Once it's ready, it takes all the traffic, and the one before it stops.

7. Check it:

   az containerapp revision list -n <name_prefix>-server -g <rg> -o table

   It lists the active revisions. Once the new one is ready, it's the only one, with all the traffic, and healthy. Then read its startup lines, as in 4. The services, and run a flow, as in 6. Check it.

If it doesn't come up, go back. Nothing rolls back on its own, and migrations only go forward, so never point the old version at the migrated database. A point-in-time restore makes a new server, and never overwrites the module's. So the runtime moves onto the restored one, through its database URL in Key Vault. Container Apps picks up a new version of that URL within 30 minutes, and a revision made sooner may start with the one before it (Microsoft: Key Vault secret URI and secret rotation). So nothing runs while it changes:

1. First, export the erasure ledger from the server serving now (Erasures and backups). It holds every erasure so far, those made after the time you noted too. If no server answers, the restored database's own ledger is all there is.

2. Restore the database as it was at the time you noted, into a new server:

   PG=$(terraform output -raw database_server_name)
   az postgres flexible-server restore -g <rg> --name "$PG-back" \
     --source-server "$PG" --restore-time <the time you noted>
   az postgres flexible-server parameter set -g <rg> -s "$PG-back" \
     --name azure.extensions --value VECTOR

   • With no network options, az puts the new server in the module server's subnet and private DNS zone.
   • The new server keeps the module server's tier, storage and backup retention, but none of its server parameters (Microsoft: Backup and restore). The second command allows pgvector on it, as the module does on its own.
   • Nothing reads it yet: the new version keeps serving from the module's server.

3. Save the plan that goes back. Put server_image, and pack_image, back from the copy of dev.tfvars you kept, then plan, and read it:

   (umask 077 && terraform plan -var-file=dev.tfvars -out=back.tfplan)

   Plan now: a plan reads the module's server, which the next step stops.

4. Stop the new version. Deactivate its revision, then stop the module's database server. PG is from step 2: in a new shell, set it again first.

   REV=$(az containerapp revision list -n <name_prefix>-server -g <rg> --query "[?properties.active].name" -o tsv)
   az containerapp revision deactivate -n <name_prefix>-server -g <rg> --revision "$REV"
   az postgres flexible-server stop -g <rg> -n "$PG"
   az postgres flexible-server show -g <rg> -n "$PG" --query state -o tsv
   az containerapp revision list -n <name_prefix>-server -g <rg> -o table

   • Deactivating a revision stops its replicas, so the new version can't start on the restored server once the URL names it.
   • With the module's server stopped, no revision can reach the migrated database, whichever URL it starts with.
   • The last two commands check it: the server is Stopped, and no revision is listed, since the list shows only active ones. If it lists any revision, stop here, and don't change the URL while one can run.

5. Point the runtime at the restored server. Set the restored server's admin password and the database URL with it, as in 3. The secrets: the password goes through a pipe, never on a command line.

   # Off in this shell: az's log file, if it's on, records each request's body.
   export AZURE_LOGGING_ENABLE_LOG_FILE=no
   
   V=$(terraform output -raw key_vault_name)
   put() { az keyvault secret set --vault-name "$V" -n "$1" --file /dev/stdin --query id -o tsv; }
   
   DBPW=$(openssl rand -hex 24)
   BACK=$(az postgres flexible-server show -g <rg> -n "$PG-back" --query fullyQualifiedDomainName -o tsv) &&
     [ -n "$BACK" ] &&
     printf '%s' "$DBPW" | az postgres flexible-server update -g <rg> -n "$PG-back" --admin-password @- -o none &&
     printf 'postgres://kindgiadmin:%s@%s:5432/kindgi?sslmode=require' "$DBPW" "$BACK" | put database-url
   unset DBPW

   Each command runs only if the one before it worked, so the vault never gets a URL for a server or a password that isn't there.

6. Wait 30 minutes, then apply the plan that goes back:

   terraform apply back.tfplan && rm back.tfplan

   • Nothing runs while you wait, and by the end of it Container Apps has the URL's new version.
   • The plan's revision can reach only the restored server. One that starts with the URL before can't connect, and Container Apps restarts it with the new one.
   • If Terraform refuses the plan as stale, start the module's server, plan again as in step 3, stop it as in step 4, and apply.
   • Check it as the upgrade's step 7 does: once the plan's revision is the only active one, healthy, it serves from the restored server.

7. Replay the erasures before anyone uses it (Erasures and backups). A replay matches only with the same ledger key, which the module pins (erasure_ledger_key_version).

From then on, the runtime serves from the restored server, which the module doesn't manage. The module's own server, which database_server_name names, stays: Terraform keeps it, and would make it again if you deleted it by hand. Start it again before your next plan; a stopped server also starts again on its own after seven days. Its database, the migrated one, doesn't get the erasures made since, so once the restored server is checked, delete it (Erasures and backups):

az postgres flexible-server start -g <rg> -n "$PG"
az postgres flexible-server db delete -g <rg> -s "$PG" -d kindgi --yes

The module's next apply makes it again, empty. At the next upgrade, check and restore the server the URL names, not the module's.

Tear it down

terraform destroy -var-file=dev.tfvars

• The vault: with key_vault_purge_protection = true, the default, the deleted vault and its keys stay recoverable for key_vault_soft_delete_retention_days, and its name stays taken until then (Microsoft: Key Vault soft-delete). A sandbox can leave purge protection off from the start, with key_vault_purge_protection = false before the vault exists. It can then set key_vault_purge_on_destroy = true, if whoever runs Terraform may purge.
• The infrastructure group goes with the environment. Check: az group exists -n <rg>-<name_prefix>-infra.
• How long: in the verification run, one destroy took 25 minutes, most of it the environment.
• On azurerm 5.8, the destroy stopped after each container app and the environment were already deleted, with polling support for the Content-Type "" was not implemented. 5.9 fixes it, which is why the module needs it.

Limits

• One runtime replica. Several aren't supported.
• Requests end at 240 seconds: follow long runs instead of waiting (Requests longer than 240 seconds).

Next

• Operate a self-hosted runtime: health, backups, upgrades and rotating keys.
• Renew the license key: on the module's schedule.
• Logs: what the runtime logs, and how to follow a run.
• Turn on sign-in: every way in, and what each setting does.