Serve many tenants

View as Markdown

A runtime serves one tenant by default, KINDGI_TENANT_ID's. With KINDGI_TENANCY=multi it serves many. Each tenant is created through the system API, and each request's tenant comes from its key or session. This page is for the operator: whoever runs the runtime, its database and the services around it, for tenants who don't trust one another. The runtime's license applies as on any deployment (Licensing).

Commands that take the operator's token read it from $KINDGI_API_TOKEN, and the runtime's address from $KINDGI_URL.

What changes

No tenant is set on the server. KINDGI_TENANT_ID is refused: "KINDGI_TENANT_ID (or --tenant, or the config file's tenant) is refused with KINDGI_TENANCY=multi: a runtime serving many tenants takes each request's tenant from its key or session. Unset it."
Nothing else that names one tenant is taken either: KINDGI_SEED_USER_ID, KINDGI_PACK_DIR, KINDGI_DEV_CONSOLE_LOGIN, the dev echo provider and the config file's mcp.endpoints are each refused, by name: "KINDGI_TENANCY=multi serves many tenants, so it refuses what names one: seedUserId (KINDGI_SEED_USER_ID)."
The deployment's own token is the operator's. KINDGI_API_TOKEN has the full system capability, kindgi:system. It acts in no tenant and reaches /v1/system/* only.
A tenant's own identity provider is offered only on the domains you list in KINDGI_AUTH_VERIFIED_DOMAINS (Which domains find a workspace's identity provider). A runtime serving one tenant routes that tenant's domains with nothing to set. On a domain not listed, its people reach the provider only through its own link, and GitHub and the emailed link aren't refused to them (Who gets in).
The emailed sign-in link needs a captcha. Without the Turnstile keys (Sign-in links by email), the start stops: "The emailed sign-in link on a runtime that serves several tenants needs a captcha: set KINDGI_AUTH_TURNSTILE_SECRET (or KINDGI_AUTH_TURNSTILE_SECRET_PATH) and KINDGI_AUTH_TURNSTILE_SITE_KEY."
Each tenant is created through POST /v1/system/tenants, with its setup: its default project and its authorization store.
The start sets nothing up per tenant. After an upgrade that adds a step for each tenant, run tenants repair.
Background work keeps up every tenant that's set up, and starts new work (schedules, deliveries, notices) only in the active ones.

The banner says so:

  Tenants: many (KINDGI_TENANCY=multi), each created through POST /v1/system/tenants
  Token:   kgi_bt_… (provided), the operator's: kindgi:system, in no tenant, /v1/system/* only
  …
  Background work: upkeep in every tenant set up; new work (schedules, deliveries, notices) in the active ones
  …
  Tenant host access: isolated (stdio MCP endpoints refused; tenant-chosen hosts can't reach the metadata server, this host or private networks; providers can't sign in with this server's own credentials; KINDGI_TENANT_HOST_ACCESS)
  OpenFGA calls carry its preshared key (KINDGI_OPENFGA_PRESHARED_KEY)
  Pack services: one per deployment, at the endpoints recorded through the system API (1 allowed host pattern)

Excerpt

The settings it starts only with

A runtime serving many tenants starts only with every setting that keeps them apart, and the two that let you allow and count them:

Setting	What it does
`KINDGI_TENANT_HOST_ACCESS=isolated`	What a tenant registers reaches public hosts only: not the metadata server, this host or private networks, and never with the server's own cloud credentials. The server then refuses `KINDGI_DEV`, `KINDGI_WEBHOOK_PRIVATE_NETWORKS=allow` and `KINDGI_AUTH_PRIVATE_IDP_ORIGINS`.
`KINDGI_PACK_ROUTING=per-deployment`	Each call goes to the pack service of the newest deployment in the run's project that shipped the tool or check, never to one pack service every tenant shares. A run with no project uses the default project, whose deployments include each one that named no project. It needs a key to seal each endpoint's token under: `KINDGI_PACK_ENDPOINT_KMS_KEY`, or else the secrets' own, with `KINDGI_SECRETS_BACKEND=postgres`. `KINDGI_PACK_SERVICE_URL`, `KINDGI_PACK_SERVICE_TOKEN` and `KINDGI_PACK_SERVICE_AUTH` are refused.
`KINDGI_PACK_ENDPOINT_HOSTS`	The hosts a deployment's pack service may be on, comma-separated: a host name, `.` and a domain, or `-`, a tail and a domain.
`KINDGI_OPENFGA_API_URL` and `KINDGI_OPENFGA_PRESHARED_KEY`	The authorization store, and the key it requires of its callers, so only the server reads and writes tenants' grants (Require a key for OpenFGA).
`KINDGI_PRODUCTION_GATE=on`	A tenant puts a project in production only once you allow it (Limits and production).
`KINDGI_USAGE_METERING=on`	Each run is counted when it's created and when it ends, so tenants' limits read what each one did.

A start that lacks any of them is refused, naming each, and exits 2:

KINDGI_TENANCY=multi serves tenants who don't trust one another, so it starts only with the hosted profile's settings: KINDGI_PACK_ENDPOINT_HOSTS must name the hosts a deployment's endpoint may be on (it's empty, so every endpoint would be refused).

Environment variables has each setting in full.

At every start, the server checks that OpenFGA takes its key. If OpenFGA refuses the server's call, the start stops, exits 1 and names the settings to fix. If OpenFGA doesn't answer, the start goes on and logs a warning: "OpenFGA at isn't answering (…). The server is up; requests that need authorization fail until it answers."

The operator's keys

Keep the deployment's own token for yourself. Give each service that acts for you, such as sign-up, the service that runs pack services, or billing, its own system key (kgi_sy_…), with the least scope it needs:

Scope	What it reaches
`system:tenants`	Creating, listing, reading, renaming and closing tenants
`system:tenants:create`	Creating tenants only: each with a first admin, production blocked, no image repository, not a test tenant
`system:tenants:image-prefix`	Setting a tenant's image repository
`system:tenants:production`	A tenant's production switch
`system:tenants:limits`	A tenant's limits
`system:tenants:contacts`	A tenant's admins' contact addresses, for service notices
`system:tenants:purge`	Purging a closed tenant, once its window has passed (runtime 0.1.10 or later). No other scope includes it
`system:suspend`	Suspending and restoring a tenant, a project or a pack
`system:deployments`	The deployments and their pack services' endpoints
`system:pack-env`	The values each deployment's pack declares, for its pack service, and the revision it runs (runtime 0.1.9 or later). On a key of its own: the deployments key reads no value, and only a key holding this scope itself, not `kindgi:system`, reads a secret's
`system:env-refs`	Setting a tenant's env names to a cloud secret, for its pack services (runtime 0.4.0 or later)
`system:license`	Reading the deployment's license, for its renewal
`kindgi:system`	All of them: a full key

curl -sS -X POST "$KINDGI_URL/v1/system/keys" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") \
  -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"scopes": ["system:deployments"], "label": "pack services", "expiresAt": "2027-10-01T00:00:00Z"}'

The token comes back once, in the answer's token.
A system key acts in no tenant. Every tenant route refuses it (system-key-on-tenant-route). GET /v1/system/whoami tells a service which key it holds.
Only the full capability mints, lists and revokes keys. A key can't outlive the key minting it. A full key minted by another system key needs an expiresAt, so only the deployment's own token mints one that never expires.
Revoking a key revokes every key minted from it, however deep: POST /v1/system/keys/{keyId}/revoke.
Every request a system key makes is recorded, and a request to the system routes that changes something is recorded before it acts. When that record can't be written, nothing is done (access-record-unavailable).

Create a tenant

curl -sS -X POST "$KINDGI_URL/v1/system/tenants" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_API_TOKEN") \
  -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "name": "Acme Store",
    "slug": "acme-store",
    "imagePrefix": "registry.acme.com/tenants/acme-store",
    "firstAdmin": { "email": "pat@acme-store.example", "name": "Pat" }
  }'

With firstAdmin, that person becomes the tenant's admin and member by email, and signs in with the runtime's own sign-in (Turn on sign-in). No key comes back: once signed in, they make the tenant's keys (API keys).
Without firstAdmin, the answer carries the tenant's first admin key, once. Store it, because nothing shows it again. Only the full capability creates a tenant that way.
A tenant whose admins can't sign in any more (runtime 0.1.9 or later): kindgi-runtime tenants make-admin makes one of its people its admin, or a new person by email, with what a first admin holds: every capability but secrets:reveal, which no person is given. They then sign in and make the tenant's keys, so it needs sign-in turned on (Turn on sign-in). Like tenants repair, it runs against the runtime's database and authorization store. Run it with --dry-run first, which says what would change and changes nothing:
Terminal window
```
kindgi-runtime tenants make-admin <tenant id> \
  --email pat@acme-store.example --name Pat \
  --reason "Pat can't sign in; checked with Acme Store" --dry-run
```
--person <user id> names one of its people instead, who must be a tenant member. Nothing is taken from anyone. If it stops partway, run it again with the same arguments to finish it. What it does is in the tenant's audit log, by system, which the tenant's admins can read: making them admin and what that gives them carry your --reason.
A tenant created without firstAdmin whose admin key is lost (runtime 0.2.0 or later): kindgi-runtime tenants admin-key mints it again. The live keys of the tenant's "First admin" service account are revoked, and the console sessions they opened end. Then one key is minted for it, carrying what the first one did: tenant admin, and the tenant's root capabilities, secrets:reveal among them. It runs against the runtime's database. Run it with --dry-run first:
Terminal window
```
kindgi-runtime tenants admin-key <tenant id> \
  --reason "Acme Store's admin key was lost; checked with Acme Store" --dry-run
```
The key is printed once, alone, on stdout, so it can go straight into a secret store; the rest is said on stderr. With --dry-run, nothing is minted, and what would change is said on stdout. It's refused when one of those keys has a use recorded, unless --revoke-used: a key that was used was received, and someone may be holding it. A use is recorded at most once a minute, so a key used just now may not show it yet. A suspended tenant gets the key too, as does a closing one inside its 30-day export window; a closing tenant past that window is refused. Each key revoked, and the key minted, is in the tenant's audit log, by system, with your --reason. It exits 0 when the key is minted, 1 when it's minted but an audit entry couldn't be written or a revoked key's sessions couldn't be ended, and 2 when it's refused and nothing changed. A session that wasn't ended is refused at its next request anyway: each one checks its key. To end it, revoke that key again, with the new key as KINDGI_API_TOKEN, as the command's last line says. <key id> is the revoked key's id, from its revoked: line:
Terminal window
```
npx --yes @kindgi/cli@0.2 tokens revoke <key id>
```
A key that's already revoked stays revoked, and the sessions it left end, in the audit log as an api-key-revoked entry with sessionsEnded. A tenant created with firstAdmin has no such account: tenants make-admin makes a person its admin.
Production starts blocked. "productionAllowed": true allows it from the start, and takes system:tenants:production.

imagePrefix is the repository every image the tenant deploys must sit in, <host>/<path>. Without one, the tenant can't register a deployment. You can set it later, with PUT /v1/system/tenants/{tenantId}/image-prefix:

{
  "error": {
    "code": "image-prefix-unset",
    "message": "This workspace has no image repository set, so it can't register a deployment yet: its operator sets one.",
    "requestId": "req-f5000e71-d1ec-4ae8-984b-1778ece7b597"
  }
}

Send an Idempotency-Key with every create. If the setup didn't finish, the tenant waits, frozen, and the same request finishes it. A repeat with the first request's key gets the first answer again, except an admin key, which is never sent twice. Without the key, a lost answer can't be told apart from another tenant's slug (tenant-slug-taken).
A setup left pending shows at every start: the log names each tenant (the first 10; it counts the rest, and tenants repair --all names them all) and says to retry the same create, or to run kindgi-runtime tenants repair --tenant <id> if its create named a first admin.

Rename a tenant (runtime 0.1.10 or later) with PATCH /v1/system/tenants/<tenant id> and { "name": …, "slug": … }, either or both, under system:tenants. It answers 204, and its id stays. Its own admins can do the same with PATCH /v1/tenant. The slug follows the create's rules, and a taken one is refused with tenant-slug-taken. A closing tenant, or one whose setup is pending, can't be renamed (409 tenant-closing, 409 tenant-setup-pending). Neither can a slug that KINDGI_AUTH_VERIFIED_DOMAINS names (409 tenant-slug-in-config): see Turn on sign-in.

A pack service for each deployment

Each deployment runs in its own pack service, which you run: an orchestrator, or a script, with a system:deployments key. The runtime records where each service is, and checks it. It never creates or deletes one.

A pack's code may read whatever its service's account may. On Cloud Run, every tenant's pack services run as the module's one pack account, $N-pack. So each tenant's pack code can read every kp- secret, other tenants' pack service tokens and secret env values among them, and every secret in pack_secret_readers. One such installation is for tenants that trust one another's pack code. Give tenants that don't an installation each (Your tools' code).

List the deployments that need a service: GET /v1/system/deployments?endpoint=missing, oldest first across tenants. A deployment needs one while it serves something a call can still ask for that no newer deployment in its project serves: a tool, agent or flow version that's registered, pinned by a flow or agent version a run can reach, or used by an unfinished run; a tool deployed without a version; or a pack check a registered guardrail runs in its project (runtime 0.4.0 or later; before, while it was the newest in its project with routes for some tool version or pack check). One whose endpoint an earlier runtime made or last set also needs the earlier rule: once newer deployments in its project ship every tool version and check it does, and deploy each tool it deployed without a version the same way, it's retired for good. It also has no endpoint or a cleared one, and nothing about it is closing or suspended. Each entry has the tenant's id, the deployment's id, its pack and project (the one it named, else the tenant's default), and its image: the digest-pinned imageRef, its digest, the artifact version and when it went live, and whether to start its service warm (step 7). It has nothing else of the tenant.

A deployment that ships no tools and no guardrail checks of its own, such as one of agents and flows whose guardrails use only built-in checks, has nothing to route. It gets no endpoint, so it's never listed, and the routes on its endpoint (PUT and DELETE …/endpoint, PUT …/endpoint/problem and PUT …/suspended) answer 404. Its agents and flows reach its project's runs without a pack service. One registered by an earlier runtime may still hold a pending endpoint from then. It's never listed or shown, though those routes still find it. Leave it: clearing it retires the deployment, and a version that reaches the project's runs only through it, as a promoted one does, stops reaching them.
Run the image as a service whose host begins with the deployment's own service name, packServiceName(tenantId, deploymentId) from @kindgi/api (kp- and 20 characters), then - or .. Give it a token of its own, its KINDGI_PACK_SERVICE_TOKEN: 16 to 512 printable characters, no spaces.

If its pack declares an environment (runtime 0.1.9 or later), read it first, with a key of its own holding system:pack-env: GET …/deployments/{deploymentId}/pack-env gives each declared name's plain value (env) and the revision they make. Ask with ?secrets=values, and a name set only as a secret is given too, in secrets, read at the version the revision names. Give the service those. A name your operator set to a cloud secret (runtime 0.4.0 or later) comes in cloudSecrets, by its path, and never its value: give the service that name from that version of the secret, read as the service's own account, as a Cloud Run secretKeyRef does. If it names a problem instead, it gives no value: start nothing, and say why (step 4).
Record its endpoint:
Terminal window
```
curl -sS -X PUT "$KINDGI_URL/v1/system/tenants/$TENANT_ID/deployments/$DEPLOYMENT_ID/endpoint" \
  -H @<(printf 'Authorization: Bearer %s\n' "$KINDGI_SYSTEM_KEY") \
  -H 'Content-Type: application/json' \
  -d @<(printf '{"url": "https://%s", "auth": "token", "token": "%s"}' "$SERVICE_HOST" "$PACK_TOKEN")
```
auth is token, or google-id-token to add an ID token for the URL (Cloud Run's invoker check). The runtime records the endpoint only once every check passes:
- the URL is https://<host> (endpoint-url-invalid);
- its host is in KINDGI_PACK_ENDPOINT_HOSTS (endpoint-host-not-allowed);
- its host names the deployment's service (endpoint-name-mismatch);
- the image is in the tenant's repository (image-not-allowed);
- the service answers /v1/info as this deployment, with the kindgi-tenant-id header (endpoint-unreachable, endpoint-info-mismatch).
The deployment is then ready. A refusal is also recorded as the deployment's problem, and the tenant sees its code. For a pack that declares an environment, also send the revision you gave the service, as packEnvRevision.
While you can't make a service, say why: PUT …/deployments/{deploymentId}/endpoint/problem with a code: tenant-not-provisioned, capacity-unavailable, service-start-failed or image-not-allowed. For its environment (runtime 0.1.9 or later): pack-env-missing (a required name is set nowhere), pack-env-ambiguous (set more than one way: as a plain value, a secret or a cloud secret) or pack-env-secret-not-yet (set only as a secret, read without secrets=values). Where you run the service can't give it a cloud secret, record pack-env-cloud-secret-unsupported (runtime 0.4.0 or later). With secrets=values, every secret value the route gives is audited, one row per secret, before the answer. When those rows can't be written, or the secret store is out, it gives none (503 pack-env-unavailable): try again. A secret that can't be read for a reason the platform fixes is pack-env-secret-unreadable, a runtime with nowhere to audit one refuses with 501 pack-env-secrets-unaudited, and a secret over 32768 bytes is more than a pack service can be given (pack-env-secret-too-large): record the code. So is a name set as a secret that's too long for the secret you keep it in (pack-env-name-too-long), which the owner shortens in the pack. Record pack-env-secret-capacity when there's no room just now for a service that can be given a secret, kept apart from other projects': try again. The tenant sees the code and when, so a deploy that waits for its pack service says why at once.
Give a running service its changed environment (runtime 0.1.9 or later): GET /v1/system/pack-env lists the deployments with a ready service whose values changed since the revision it runs: ids, the image and the revision each runs, never a value. Read each one's values, start its service again with them, and record it: PUT …/pack-env with the new revision and the appliedRevision it was listed with. Another revision recorded since is pack-env-conflict. If the values can't be given, or the new revision didn't start, send held with the code instead (pack-env-update-failed for one that didn't start): the service keeps what it runs, and the tenant sees the code. One held as pack-env-update-failed stays listed for an hour after it was first held, so you try it again. After that it's listed again only when its values change. One held as pack-env-secret-unreadable or pack-env-secrets-unaudited is listed every time, with its held, since the platform fixes those and its revision doesn't move: read it again on your own back-off. If a running service can't be given a secret where you run it, and a name is now set only as one, hold it as pack-env-secret-redeploy: a new deployment's service can be given it, from a new build of the pack.
Retire what should go: GET /v1/system/deployments?endpoint=retire lists deployments with an endpoint whose service should go, each with its reason. Clear the endpoint first, with DELETE …/endpoint, then delete the service. A deployment is superseded when nothing a call can still ask for needs it: for each tool version, check, agent and flow version it serves, a newer deployment in its project serves it too, or it's unregistered, not pinned by a flow or agent version a run can reach and not used by an unfinished run (a check: no registered guardrail runs it in its project); and each tool it deployed without a version, a newer deployment in its project deploys without one too (runtime 0.4.0 or later; before, superseded meant newer deployments in its project shipped every tool version and check it does and deployed each tool it deployed without a version the same way). One whose endpoint an earlier runtime made or last set is superseded as well, for good, once newer deployments in its project ship every tool version and check it does and deploy each tool it deployed without a version the same way (step 1). Otherwise it's listed as needing a service again once something it serves that no newer deployment in its project serves is wanted again: a tool, agent or flow version registered again, pinned by a flow or agent version a run can reach or used by an unfinished run, or a check a newly registered guardrail runs in its project. A tenant sees what keeps each one in its endpoint's keptFor.
Keep the warm ones up: GET /v1/system/deployments?endpoint=ready lists deployments whose service is ready and stays, oldest first across tenants. Here and in endpoint=missing, warm: true means calls in its project route to it for something always on: a tool version reached from a version a pin holds (an agent's, a flow's or a tool's) or from the Latest of each agent no pin covers there, or a pack check a registered guardrail runs there. Keep an instance of a warm service running, and let the others scale to zero. A runtime that doesn't send warm keeps none warm.

A call to a deployment's tool never goes anywhere else. With no endpoint yet it fails with pack-endpoint-pending, and after a clear with pack-endpoint-cleared. A tenant sees its deployment's endpoint: pending (with the problem, if one is recorded), ready or cleared.

A call never goes to a deployment in another project, either. When no deployment in the run's project ships what it names, it fails with pack-route-missing:

No deployment in project "<project name>" ships tool acme.lookup@1.0.0.

A call run again stays on the deployment it first ran on. If that deployment isn't in the run's project, or doesn't ship what the call names:

Deployment <deployment-id> isn't in project "<project name>", or doesn't ship tool acme.lookup@1.0.0. This call first ran on deployment <deployment-id>, and a re-run doesn't move to another.

Set a name to a cloud secret

A name a pack reads from its environment can be set, at a tenant's scope and under one env, to a version of a secret your cloud holds instead of to a value (runtime 0.4.0 or later): a Google Cloud Secret Manager version, projects/<project>/secrets/<secret>/versions/<number>. Turn it on with KINDGI_ENV_CLOUD_SECRETS=on, which needs KINDGI_PACK_ROUTING=per-deployment. The runtime keeps that path and never reads the value. The pack service is given the name, and Cloud Run reads the value as the pack service's own account, so that account needs roles/secretmanager.secretAccessor on the secret. Every deployment's pack service runs as that one account, so any of them could read it: set one only for a value every pack may see.

The version is a number, never latest or an alias, so what a pack service runs changes only when you set another version. A secret whose id starts with kp- is refused: those are the pack services' own copies.

Only the operator sets one, with a system key holding system:env-refs as KINDGI_API_TOKEN:

npx --yes @kindgi/cli@0.4 env set STRIPE_KEY \
  --cloud-secret projects/acme-prod/secrets/stripe-key/versions/3 \
  --tenant="$TENANT_ID" --scope="project:$PROJECT_ID" --env=prod
npx --yes @kindgi/cli@0.4 env list --cloud-secret \
  --tenant="$TENANT_ID" --scope="project:$PROJECT_ID" --env=prod
npx --yes @kindgi/cli@0.4 env unset STRIPE_KEY --cloud-secret \
  --tenant="$TENANT_ID" --scope="project:$PROJECT_ID" --env=prod

kindgi env set refuses to change one that's set to another version without --force. Over HTTP, those are PUT, GET and DELETE on /v1/system/tenants/{tenantId}/env[/{name}], with envName, scopeKind and scopeId in the query, and {"cloudSecret": "<path>"} as the PUT's body. A path not of that form is 400 cloud-secret-invalid, and a tenant's own key is refused, on kindgi env set too (403 cloud-secret-system-only). Each change is in the tenant's audit record (env-set, env-deleted), naming the system key that made it and marked cloudSecret, never with the path, and in the system access record. Setting, changing or removing one moves the pack env revision, so the pack services that declare the name are given the change as for any value (step 5).

The tenant sees which names are set this way, never the path: kindgi env list --scope shows each as NAME (cloud secret), and kindgi deployments get names it set. A name set to a cloud secret and also as a value or a secret is pack-env-ambiguous: its pack service gets neither until one is removed. With KINDGI_ENV_CLOUD_SECRETS off, the default, the runtime keeps none and answers 501 cloud-secrets-unsupported, as Kindgi Cloud does. A runtime serving one tenant with a pack service per deployment mints system:env-refs keys too, from its own token, for that tenant alone.

Limits and production

Limits: GET and PUT /v1/system/tenants/{tenantId}/limits (system:tenants:limits). PUT replaces them all: a limit left out isn't set, and isn't enforced. Read them, change one, and put them all back. The tenant reads its own on GET /v1/tenant.
Production: PUT /v1/system/tenants/{tenantId}/production-allowed with {"allowed": true} (system:tenants:production). Until then, putting a project in production is refused with 403 production-not-allowed, and projects already in production stay so. KINDGI_PRODUCTION_NEXT_URL names where a refused tenant goes to be allowed (Put a project in production). The projectsInProduction limit caps how many of its projects are in production at once.

Suspend and restore

Suspending takes system:suspend, which no other scope includes, and can record a reason. A second suspend keeps the first one's time and reason, and restoring clears both. A closing tenant can't be suspended (409 tenant-closing).

A tenant: PUT /v1/system/tenants/{tenantId}/suspended with {"suspended": true, "reason": "…"}. It takes hold within 5 seconds. Its own keys and sessions still read and export it. Nothing runs (its unfinished runs are cancelled, waiting ones included), and any other write is refused (409 tenant-suspended), but for deletes, signing in and out, revoking, and erasing or searching memory. Its schedules and triggers are skipped, not changed, and a webhook whose signature checks is answered 202 and skipped. At the restore, nothing that came due meanwhile runs late: a schedule fires next at its first time after the restore.
A project: PUT /v1/system/tenants/{tenantId}/projects/{projectId}/suspended. Nothing new runs in it (409 project-suspended), and no pack service's endpoint is recorded for it. Its unfinished runs are cancelled, waiting ones included. Its schedules and webhooks are skipped, and its deployments list to retire. The rest of the tenant is untouched. While the default project is suspended, a pack call that names no project is refused too.
A pack: PUT /v1/system/tenants/{tenantId}/deployments/{deploymentId}/suspended. Its calls fail with pack-suspended, never going to an older deployment, and it lists to retire. Restored, it serves again from the endpoint recorded for it. If its endpoint is pending, or was cleared meanwhile, it lists as needing a pack service.

Close and purge a tenant

POST /v1/system/tenants/{tenantId}/close starts the end of the tenant's contract, and there's no way back. The tenant is closing. For its 30-day export window, its own keys and sessions still read and export it, and its people can still sign in. Nothing runs (its unfinished runs are cancelled, waiting ones included), and writes are refused as for a suspended tenant, with 409 tenant-closing. Its deployments list to retire.

Once the window has passed, purge it. Its keys and sessions go with its rows. Until then, nothing of the tenant runs, and it writes only deletes, revokes, erasures, exports and sign-ins. One of those during the purge can leave a row: the purge names it, and running it again finishes it.

When nothing of the tenant is left, the purge writes its record and marks it purged. Each server keeps a tenant's state for about 5 seconds, so within about 5 seconds of the mark nothing of it signs in or is served. The purge waits three times that, then deletes again: work of the tenant under way at the mark, such as a request that started before it, can still write a row.

Purge it through the system API, or with kindgi-runtime tenants purge where the runtime's database is reachable. Both run the same purge.

Through the system API

POST /v1/system/tenants/{tenantId}/purge (runtime 0.1.10 or later) takes a system key holding system:tenants:purge, or kindgi:system. No other scope includes it: give it to the tooling that ends accounts, and to nothing else.

curl -sS -X POST "$KINDGI_URL/v1/system/tenants/$TENANT_ID/purge" \
  -H "Authorization: Bearer $KINDGI_SYSTEM_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"confirm": "acme-shop"}'

confirm is the tenant's slug, a second check on the id. Another value is 400 tenant-slug-mismatch, and nothing is changed.
The servers need KINDGI_TENANT_PURGE_RECORD: where purge records are kept, local:<absolute dir>, gcs:<bucket>[/<prefix>] or s3:<bucket>[/<prefix>] (runtime 0.4.0 or later), the same place as the command's --record (Choose where the runtime keeps files). Unset, every purge is refused with 409 purge-record-unset: without a record, restoring a backup taken before the purge would bring the tenant back, with nothing to purge it again (kindgi-runtime tenants repurge reads the records after a restore).
It answers when it's done, after its wait. The answer names each table's rows it deleted (deleted), the objects deleted from the artifacts' store (objectsDeleted), and whether its authorization store went (authzStoreDeleted).
One purge of a tenant runs at a time, through this route or with kindgi-runtime tenants purge: a second waits for the first to finish, then runs. Every 30 seconds of waiting, the server logs tenant purge: waiting for another purge of the tenant to finish, with tenantId and waitedSeconds, under the subsystem tenants.purge.
complete: true: nothing of it is left, its record is written, and it's marked purged at purgedAt. complete: false: left and objectsLeft say what is, and leftBecause why, when it's a setting the runtime lacks. Run it again. A call that ended before its answer is finished the same way. Each run again is a new request: a new Idempotency-Key, or none.
Refused, with nothing changed: a tenant that isn't closing (409 tenant-not-closing), or whose window is still open (409 tenant-export-window-open, with windowEndsAt), and one with artifacts or an authorization store when the server has no KINDGI_ARTIFACTS or KINDGI_OPENFGA_API_URL (409 purge-setting-missing).
A failed delete of its OpenFGA store is 502 authz-store-delete-failed, and its message says what that left. A tenant's rows go only after its store, so by then its artifacts' bytes may be deleted, but none of its rows. Run it again once OpenFGA answers.
A day later, run it again. On a tenant already purged, it deletes anything of it written since its mark (alreadyPurged: true), and its mark and its record stay as they were. Nothing should write to a tenant a day after its mark, so report any tables it names.
Every call is on the operator's access record, with its key, the tenant and its outcome, a refusal too.

With kindgi-runtime

kindgi-runtime tenants purge <tenant id> --confirm <tenant slug> --record <where>

tenants purge: tenant 1ccae8e9-3dc8-4db9-b393-ba3d44e7f86e (acme-shop)
  artifacts: 1 row deleted
  audit_events: 4 rows deleted
  authz_store_registry: 1 row deleted
  identity_users: 1 row deleted
  adapter_registry: 6 rows deleted
  memory_erasures: 1 row deleted
  memory_facts: 2 rows deleted
  projects: 1 row deleted
  artifacts' bytes: 1 deleted
  OpenFGA store: 01M4WPFNJJ7YY6RXSJK1KB9W7D deleted
  No table holds a row of it now, and no bytes are under its prefix; its tenants row stays, marked purged.
  record: purged/1ccae8e9-3dc8-4db9-b393-ba3d44e7f86e.json written

It deletes the tenant for good: every row of it, its artifacts' bytes and its OpenFGA store. Its tenants row stays, marked purged, with nothing left that signs in.
--dry-run says what would be deleted, and changes nothing.
Refused (exit 2) for a tenant that isn't closing or is still in its window, or one with artifacts or an OpenFGA store when KINDGI_ARTIFACTS or KINDGI_OPENFGA_API_URL isn't set as the server has it. Exit 1 when rows or bytes of it are left, named: run it again.
With a key for OpenFGA (Require a key for OpenFGA), purging a tenant with an OpenFGA store needs runtime 0.1.9 or later. An earlier one doesn't send the key, or the ID token behind Cloud Run IAM, so the store's delete is refused, and the purge exits 2 with OpenFGA answered <status> to deleting store <id>. By then the tenant's artifacts' bytes are deleted, but none of its rows.
Exit 1, too, when work in flight at the mark wrote rows of it: each is named (written after its mark: …) and deleted. One that needs a setting the run doesn't have is left, with why (written after its mark, not deleted: …): run it again with that setting, or report it.
A day later, run it again with --dry-run, and without it if it lists anything. On a tenant already purged, the purge deletes what's of it, and its mark and its record stay as they were. Nothing should write to a tenant a day after its mark, so report the tables it names. kindgi-runtime check purged-tenants lists every tenant marked purged that still holds rows, by table and count: rows only, not the bytes under its prefix.
--record (local:<dir>, gcs:<bucket>[/<prefix>], or s3:<bucket>[/<prefix>] with runtime 0.4.0 or later) keeps a record of each purge, purged/<tenant id>.json under it. After a restore, tenants repurge reads them.
One purge of a tenant runs at a time (runtime 0.1.10 or later): a second, here or through the system API, waits for the first to finish. Every 30 seconds of waiting it says tenants purge: waiting for another purge of tenant <id> to finish (30 s so far) on stderr. tenants repurge waits the same way, tenant by tenant: tenant <id>: waiting for another purge of it to finish (30 s so far).

After a restore: tenants repurge

A backup taken before a tenant's purge brings the tenant back with its data, and replaying the erasure ledger doesn't purge it. After a restore, before the runtime serves anyone, purge every purged tenant again:

kindgi-runtime tenants repurge --from <where> --ledger <where>

Two sources name the tenants: the purge records under --from (local:<dir>, gcs:<bucket>[/<prefix>] or s3:<bucket>[/<prefix>], where --record and KINDGI_TENANT_PURGE_RECORD write them), and the newest copy of the erasure ledger under --ledger (runtime 0.1.10 or later), which names every tenant purged before it was written, however long ago. A record can be gone, deleted by a bucket's retention rule, say; the copy still names its tenant. A tenant both name is purged once.
--ledger takes what erasures replay --from takes: a directory, a bucket, or one file by its absolute path. Left out, it's KINDGI_ERASURE_LEDGER_EXPORT, where the server writes its copies. With neither, it goes by the records alone, and says so: No --ledger or KINDGI_ERASURE_LEDGER_EXPORT: the records alone, so a tenant whose record is gone isn't found. With no copy where it looks, it stops: tenants repurge: no copy of the erasure ledger in <where>.
One bucket can hold both, for example the records under purged/ (--from gcs:<bucket>) and the server's copies under runtime/ (--ledger gcs:<bucket>/runtime). Reading a bucket, it lists and downloads, so whatever runs it needs read access to it, as replay does (the permissions).
Run it with --dry-run first, and nothing changes. Its first line says what it read: tenants repurge --dry-run: <n> purge records, and the ledger's copy <file> (written <when>; <n> purged tenants). Nothing will be changed. A copy names purged tenants when runtime 0.1.10 or later wrote it. An earlier one says version 1, which names no purged tenant, and only the records name tenants then. A line for each tenant follows, by id and slug, with what named it (its record, the ledger's copy, or both) and what a run would do: would be purged again, with its state and what it holds; already purged; what was written of it after its mark would be deleted; or already purged, and nothing of it is left.
Each tenant is purged again, whatever its restored row says: a restore can bring it back active, or closing. On one still marked purged, it deletes what's of it, and leaves its mark and its record as they were.
Skipped or refused: a tenant the database doesn't have is skipped (not in this database, skipped). One whose record or entry in the copy isn't its own is refused, and named. That's checked by when the tenant was created, which a rename doesn't change (refused: its record's createdAt isn't this tenant's.), and, for a record without a creation time, by the tenant's slug.
Exit 0: done. 1: rows of a tenant are left, named (tenant <id>: left: <table>: <n> rows): run it again. 2: a tenant was refused, as above or for a setting the run lacks (artifacts or an OpenFGA store, and no KINDGI_ARTIFACTS or KINDGI_OPENFGA_API_URL), or it couldn't run.

After each upgrade: tenants repair

A runtime serving many tenants sets nothing up for each tenant when it starts. After every upgrade that adds a step for each tenant, run this once:

kindgi-runtime tenants repair --all

tenant 5b52fd42-bd2b-4597-b324-84ec1a7033a4 (workspace-1792242381807-ed3afb): Default project kept; authorization store kept; 0 tuples added; 0 memberships and 0 service account grants put back; 0 removals finished and 0 grants taken away; adapters: 6 kept up to date, 0 removed; sign-in providers reconciled
1 tenant repaired.

It adds what's missing: the default project, the authorization store's tuples and current model, the access each active person and service account has on record (a person's tenant membership, a service account's grants), the shipped adapters, and the sign-in providers. It finishes a create whose setup didn't finish, if the create named a first admin.

It also takes away access the tenant's record doesn't back (runtime 0.3.0 or later): it finishes a person's removal, or a service account's unregister, that stopped part way, and revokes a service account's admin, membership or project role that its grants don't give. And it takes away the access an older runtime gave 00000000-0000-0000-0000-000000000000, the id it named as the creator of what a key with no person behind it made; no person has that id. Each removal it finishes and each grant it takes away is named under the tenant's line. --dry-run shows what it would put back and take away, per tenant, and changes nothing.

It also sets right the parents in the authorization store that a project's or a team's move into or out of an org, or an org's delete, left behind on an older runtime. That runtime changed only the row, so the old org's admins kept their way in, and the new org's had none. For each project, team and deleted org, it reads the parents the store has, and queues the changes the row calls for, which the server applies. The tenant's line counts them after the tuples added, only when there are any, such as 1 parent change queued for the server to apply: a corrected parent is one change. --dry-run doesn't preview them. A runtime serving one tenant sets them right at each start.

Run it again at will. --tenant <id> repairs one tenant. --providers redoes only sign-in's providers, after KINDGI_PUBLIC_URL changes. It exits 0 when every tenant is set up, and 1 when something is left for you, named:

tenant f309aaca-024a-4fb9-aff8-ef76c4c40e16 (tenant-a): Default project created; no authorization store, and none can be made without naming its first admin: left for the operator; 0 tuples added; adapters: 6 kept up to date, 0 removed; sign-in providers reconciled
1 tenant gone through; 1 left for the operator (named above).

Run it with the version you upgraded to, after its server has started. The tenants commands, repair among them, run only on a database migrated to their own version, and exit 2 on any other, with Nothing was changed.:

A database a newer version migrated: This database was migrated by a newer Kindgi version: it records a migration this version doesn't have (…). This version's `tenants repair` doesn't run on it: run it with the version that migrated it.
A database not migrated to this version yet: This database isn't migrated to this Kindgi version yet: it doesn't record one of this version's migrations (…). This version's `tenants repair` doesn't run on it until it's migrated: start this version's server on it first, which migrates it. For tenants repurge after a restore, it says to run this version's erasures replay on it first instead.

--all checks again before each tenant. When it's refused partway, it names the tenants it went through first. A command waits while a migration or another admin command runs, and says so on stderr every 30 seconds, as tenants repair: waiting for a migration, or another admin command, to finish (30 s so far). A server that starts while a command runs waits for it the same way, and logs it at WARN: waiting for another instance's migrations, or an admin command, to finish (30 s so far).

A tenant id given in upper case

A runtime serving one tenant takes KINDGI_TENANT_ID in lower case. An earlier runtime given it in upper case wrote under that spelling. Each start that finds something written there warns, once, at WARN: KINDGI_TENANT_ID is set as <TENANT ID>, and is taken in lower case (<tenant id>)., followed by the steps for that runtime:

Move the artifact bytes kept under the old spelling to the same place under tenants/<tenant id>/. The release notes' upgrade steps have the commands.
Once every replica runs the new version, repair the tenant with the old spelling:
Terminal window
```
kindgi-runtime tenants repair --tenant <tenant id> --old-spelling <TENANT ID>
```
<tenant id> is the id in lower case, and <TENANT ID> the id as it was given, as the warning names them.
Set KINDGI_TENANT_ID in lower case.

--old-spelling moves tenant admin given on the old spelling to the tenant, for the people still in it and the system's. A removed person's is only taken. Then it repairs the tenant, and deletes every other authorization tuple that names the old spelling. Its line lists the ids it moved (tenant admin moved for …), the ids it only took (taken only, as they're not in the tenant: …), and how many tuples it deleted. A second run finds nothing to do. It's refused, with exit 2, without --tenant, with --providers or --dry-run, with an id that isn't the tenant's in another case, and without KINDGI_OPENFGA_API_URL.

Until it runs, a tenant admin given on the old spelling keeps it on what was made before the upgrade, and the console doesn't list them as one. Taking tenant admin from someone, or removing them, takes it on both spellings.

Switch a one-tenant runtime to many

The first start with many tenants keeps the tenant it served, with its data, keys and people, as the first of many.

Run OpenFGA with a preshared key, and point the runtime at it (Require a key for OpenFGA).
Upgrade and start it still as one tenant, KINDGI_TENANT_ID kept. If the tenant has no authorization store yet, because it ran without OpenFGA, that start sets one up. Going straight to many tenants would leave it without one, and tenants repair can't make one.
Start it with many tenants: KINDGI_TENANCY=multi and the settings above, with KINDGI_TENANT_ID, KINDGI_SEED_USER_ID and the rest that name one tenant removed (What changes). If the tenant has its own identity provider, list its domains in KINDGI_AUTH_VERIFIED_DOMAINS. If the emailed sign-in link is on, set the Turnstile keys: the start stops without them. KINDGI_API_TOKEN is now the operator's and acts in no tenant, so whoever used it as the tenant's admin needs another way in first. A tenant key minted before the switch keeps working, and so does the tenant's sign-in.
Run kindgi-runtime tenants repair --all once. Until it runs, the people and service accounts added while the tenant ran without OpenFGA are refused: the start in step 2 makes only the seed user an admin. The repair gives them back the access their records show.
Set the tenant's image repository: PUT /v1/system/tenants/{tenantId}/image-prefix with {"prefix": "<host>/<path>"}.
Deploy each pack again. Its deployments from before have no routes, and the banner says how many:
```
  ⚠ Per-deployment routing: 1 deployment registered before it has no routes; deploy it again. The same image is enough.
```
Deploy it with --idempotency-key <new value>, so the runtime doesn't answer it as it did the earlier deploy.

Its endpoint is then pending until you record its pack service.

The system routes in full: system tenants, system keys and system deployments.
Back up and restore, then tenants repurge.