# Serve many tenants > Run one Kindgi runtime for many tenants who don't trust one another. Covers the settings it starts only with, the operator's system keys, creating tenants, a pack service for each deployment, and suspending, closing and purging a tenant. A runtime serves one tenant by default, `KINDGI_TENANT_ID`'s. With `KINDGI_TENANCY=multi` it serves many. Each tenant is created through the system API, and each request's tenant comes from its key or session. This page is for the operator: whoever runs the runtime, its database and the services around it, for tenants who don't trust one another. The runtime's license applies as on any deployment ([Licensing](https://docs.kindgi.com/v0.1.8/concepts/licensing.md)). Commands that take the operator's token read it from `$KINDGI_API_TOKEN`, and the runtime's address from `$KINDGI_URL`. ## What changes - **No tenant is set on the server.** `KINDGI_TENANT_ID` is refused: "KINDGI_TENANT_ID (or --tenant, or the config file's `tenant`) is refused with KINDGI_TENANCY=multi: a runtime serving many tenants takes each request's tenant from its key or session. Unset it." - **Nothing else that names one tenant is taken either:** `KINDGI_SEED_USER_ID`, `KINDGI_PACK_DIR`, `KINDGI_DEV_CONSOLE_LOGIN`, the dev echo provider and the config file's `mcp.endpoints` are each refused, by name: "KINDGI_TENANCY=multi serves many tenants, so it refuses what names one: seedUserId (KINDGI_SEED_USER_ID)." - **The deployment's own token is the operator's.** `KINDGI_API_TOKEN` has the full system capability, `kindgi:system`. It acts in no tenant and reaches `/v1/system/*` only. - **A tenant's own identity provider is offered only on the domains you list** in `KINDGI_AUTH_VERIFIED_DOMAINS` ([Which domains find a workspace's identity provider](https://docs.kindgi.com/v0.1.8/deploy/sign-in.md#which-domains-find-a-workspaces-identity-provider)). A runtime serving one tenant routes that tenant's domains with nothing to set. On a domain not listed, its people reach the provider only through its own link, and GitHub and the emailed link aren't refused to them ([Who gets in](https://docs.kindgi.com/v0.1.8/deploy/sign-in.md#who-gets-in)). - **The emailed sign-in link needs a captcha.** Without the Turnstile keys ([Sign-in links by email](https://docs.kindgi.com/v0.1.8/deploy/sign-in.md#sign-in-links-by-email)), the start stops: "The emailed sign-in link on a runtime that serves several tenants needs a captcha: set KINDGI_AUTH_TURNSTILE_SECRET (or KINDGI_AUTH_TURNSTILE_SECRET_PATH) and KINDGI_AUTH_TURNSTILE_SITE_KEY." - **Each tenant is created through `POST /v1/system/tenants`**, with its setup: its Default project and its authorization store. - **The start sets nothing up per tenant.** After an upgrade that adds a step for each tenant, run [`tenants repair`](https://docs.kindgi.com/v0.1.8/deploy/serve-many-tenants.md#after-each-upgrade-tenants-repair). - **Background work** keeps up every tenant that's set up, and starts new work (schedules, deliveries, notices) only in the active ones. The banner says so: ```text excerpt Tenants: many (KINDGI_TENANCY=multi), each created through POST /v1/system/tenants Token: kgi_bt_… (provided), the operator's: kindgi:system, in no tenant, /v1/system/* only … Background work: upkeep in every tenant set up; new work (schedules, deliveries, notices) in the active ones … Tenant host access: isolated (stdio MCP endpoints refused; tenant-chosen hosts can't reach the metadata server, this host or private networks; providers can't sign in with this server's own credentials; KINDGI_TENANT_HOST_ACCESS) OpenFGA calls carry its preshared key (KINDGI_OPENFGA_PRESHARED_KEY) Pack services: one per deployment, at the endpoints recorded through the system API (1 allowed host pattern) ``` ## The settings it starts only with A runtime serving many tenants starts only with every setting that keeps them apart, and the two that let you allow and count them: | Setting | What it does | | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `KINDGI_TENANT_HOST_ACCESS=isolated` | What a tenant registers reaches public hosts only: not the metadata server, this host or private networks, and never with the server's own cloud credentials. The server then refuses `KINDGI_DEV`, `KINDGI_WEBHOOK_PRIVATE_NETWORKS=allow` and `KINDGI_AUTH_PRIVATE_IDP_ORIGINS`. | | `KINDGI_PACK_ROUTING=per-deployment` | Each call goes to the pack service of the deployment that shipped the tool or check, never to one pack service every tenant shares. It needs `KINDGI_SECRETS_BACKEND=postgres`, because each endpoint's token is sealed under the secrets' key. `KINDGI_PACK_SERVICE_URL`, `KINDGI_PACK_SERVICE_TOKEN` and `KINDGI_PACK_SERVICE_AUTH` are refused. | | `KINDGI_PACK_ENDPOINT_HOSTS` | The hosts a deployment's pack service may be on, comma-separated: a host name, `*.` and a domain, or `*-`, a tail and a domain. | | `KINDGI_OPENFGA_API_URL` and `KINDGI_OPENFGA_PRESHARED_KEY` | The authorization store, and the key it requires of its callers, so only the server reads and writes tenants' grants ([Require a key for OpenFGA](https://docs.kindgi.com/v0.1.8/deploy/authorization.md#require-a-key-for-openfga)). | | `KINDGI_PRODUCTION_GATE=on` | A tenant puts a project in production only once you allow it ([Limits and production](https://docs.kindgi.com/v0.1.8/deploy/serve-many-tenants.md#limits-and-production)). | | `KINDGI_USAGE_METERING=on` | Each run is counted when it's created and when it ends, so tenants' limits read what each one did. | A start that lacks any of them is refused, naming each, and exits 2: ```text KINDGI_TENANCY=multi serves tenants who don't trust one another, so it starts only with the hosted profile's settings: KINDGI_PACK_ENDPOINT_HOSTS must name the hosts a deployment's endpoint may be on (it's empty, so every endpoint would be refused). ``` [Environment variables](https://docs.kindgi.com/v0.1.8/reference/env-vars.md) has each setting in full. At every start, the server checks that OpenFGA takes its key. If OpenFGA refuses the server's call, the start stops, exits 1 and names the settings to fix. If OpenFGA doesn't answer, the start goes on and logs a warning: "OpenFGA at isn't answering (…). The server is up; requests that need authorization fail until it answers." ## The operator's keys Keep the deployment's own token for yourself. Give each service that acts for you, such as sign-up, the service that runs pack services, or billing, its own **system key** (`kgi_sy_…`), with the least scope it needs: | Scope | What it reaches | | ----------------------------- | ---------------------------------------------------------------------------------------------------------- | | `system:tenants` | Creating, listing, reading and closing tenants | | `system:tenants:create` | Creating tenants only: each with a first admin, production blocked, no image repository, not a test tenant | | `system:tenants:image-prefix` | Setting a tenant's image repository | | `system:tenants:production` | A tenant's production switch | | `system:tenants:limits` | A tenant's limits | | `system:tenants:contacts` | A tenant's admins' contact addresses, for service notices | | `system:suspend` | Suspending and restoring a tenant, a project or a pack | | `system:deployments` | The deployments and their pack services' endpoints | | `system:license` | Reading the deployment's license, for its renewal | | `kindgi:system` | All of them: a full key | ```sh curl -sS -X POST "$KINDGI_URL/v1/system/keys" \ -H "Authorization: Bearer $KINDGI_API_TOKEN" \ -H 'Content-Type: application/json' \ -H "Idempotency-Key: $(uuidgen)" \ -d '{"scopes": ["system:deployments"], "label": "pack services", "expiresAt": "2027-10-01T00:00:00Z"}' ``` - **The token comes back once,** in the answer's `token`. - **A system key acts in no tenant.** Every tenant route refuses it (`system-key-on-tenant-route`). `GET /v1/system/whoami` tells a service which key it holds. - **Only the full capability mints, lists and revokes keys.** A key can't outlive the key minting it. A full key minted by another system key needs an `expiresAt`, so only the deployment's own token mints one that never expires. - **Revoking a key revokes every key minted from it,** however deep: `POST /v1/system/keys/{keyId}/revoke`. - **Every request a system key makes is recorded,** and a request to the system routes that changes something is recorded before it acts. When that record can't be written, nothing is done (`access-record-unavailable`). ## Create a tenant ```sh curl -sS -X POST "$KINDGI_URL/v1/system/tenants" \ -H "Authorization: Bearer $KINDGI_API_TOKEN" \ -H 'Content-Type: application/json' \ -H "Idempotency-Key: $(uuidgen)" \ -d '{ "name": "Acme Store", "slug": "acme-store", "imagePrefix": "registry.acme.com/tenants/acme-store", "firstAdmin": { "email": "pat@acme-store.example", "name": "Pat" } }' ``` - **With `firstAdmin`,** that person becomes the tenant's admin and member by email, and signs in with the runtime's own sign-in ([Turn on sign-in](https://docs.kindgi.com/v0.1.8/deploy/sign-in.md)). No key comes back: once signed in, they make the tenant's keys ([API keys](https://docs.kindgi.com/v0.1.8/deploy/people-and-keys.md#api-keys)). - **Without `firstAdmin`,** the answer carries the tenant's first admin key, once. Store it, because nothing shows it again, and without it nothing gives the tenant another admin key. Only the full capability creates a tenant that way. - **Production starts blocked.** `"productionAllowed": true` allows it from the start, and takes `system:tenants:production`. - **`imagePrefix`** is the repository every image the tenant deploys must sit in, `/`. Without one, the tenant can't register a deployment. You can set it later, with `PUT /v1/system/tenants/{tenantId}/image-prefix`: ```json { "error": { "code": "image-prefix-unset", "message": "This tenant has no image repository set, so it can't register a deployment yet: its operator sets one.", "requestId": "req-f5000e71-d1ec-4ae8-984b-1778ece7b597" } } ``` - **Send an `Idempotency-Key` with every create.** If the setup didn't finish, the tenant waits, frozen, and the same request finishes it. A repeat with the first request's key gets the first answer again, except an admin key, which is never sent twice. Without the key, a lost answer can't be told apart from another tenant's slug (`tenant-slug-taken`). - **A setup left pending shows at every start:** the log names each tenant (the first 10; it counts the rest, and `tenants repair --all` names them all) and says to retry the same create, or to run `kindgi-runtime tenants repair --tenant ` if its create named a first admin. ## A pack service for each deployment Each deployment runs in its own pack service, which you run: an orchestrator, or a script, with a `system:deployments` key. The runtime records where each service is, and checks it. It never creates or deletes one. 1. **List the deployments that need a service:** `GET /v1/system/deployments?endpoint=missing`, oldest first across tenants. A deployment needs one while it's routed: it's the newest with routes for some tool version or pack check, it has no endpoint or a cleared one, and nothing about it is closing or suspended. Each entry has the tenant's id, the deployment's id, its pack (and project, if it has one), and its image: the digest-pinned `imageRef`, its digest, the artifact version and when it went live. It has nothing else of the tenant. 2. **Run the image as a service** whose host begins with the deployment's own service name, `packServiceName(tenantId, deploymentId)` from `@kindgi/api` (`kp-` and 20 characters), then `-` or `.`. Give it a token of its own, its `KINDGI_PACK_SERVICE_TOKEN`: 16 to 512 printable characters, no spaces. 3. **Record its endpoint:** ```sh curl -sS -X PUT "$KINDGI_URL/v1/system/tenants/$TENANT_ID/deployments/$DEPLOYMENT_ID/endpoint" \ -H "Authorization: Bearer $KINDGI_SYSTEM_KEY" \ -H 'Content-Type: application/json' \ -d "{\"url\": \"https://$SERVICE_HOST\", \"auth\": \"token\", \"token\": \"$PACK_TOKEN\"}" ``` `auth` is `token`, or `google-id-token` to add an ID token for the URL (Cloud Run's invoker check). The runtime records the endpoint only once every check passes: - the URL is `https://` (`endpoint-url-invalid`); - its host is in `KINDGI_PACK_ENDPOINT_HOSTS` (`endpoint-host-not-allowed`); - its host names the deployment's service (`endpoint-name-mismatch`); - the image is in the tenant's repository (`image-not-allowed`); - the service answers `/v1/info` as this deployment, with the `kindgi-tenant-id` header (`endpoint-unreachable`, `endpoint-info-mismatch`). The deployment is then `ready`. A refusal is also recorded as the deployment's problem, and the tenant sees its code. 4. **While you can't make a service, say why:** `PUT …/deployments/{deploymentId}/endpoint/problem` with a `code`: `tenant-not-provisioned`, `capacity-unavailable`, `service-start-failed` or `image-not-allowed`. The tenant sees the code and when, so a deploy that waits for its pack service says why at once. 5. **Retire what should go:** `GET /v1/system/deployments?endpoint=retire` lists deployments with an endpoint whose service should go, each with its `reason`. Clear the endpoint first, with `DELETE …/endpoint`, then delete the service. A `superseded` deployment is one whose every tool version and check newer deployments ship. A call to a deployment's tool never goes anywhere else. With no endpoint yet it fails with `pack-endpoint-pending`, and after a clear with `pack-endpoint-cleared`. A tenant sees its deployment's `endpoint`: `pending` (with the problem, if one is recorded), `ready` or `cleared`. ## Limits and production - **Limits:** `GET` and `PUT /v1/system/tenants/{tenantId}/limits` (`system:tenants:limits`). `PUT` replaces them all: a limit left out isn't set, and isn't enforced. Read them, change one, and put them all back. The tenant reads its own on `GET /v1/tenant`. - **Production:** `PUT /v1/system/tenants/{tenantId}/production-allowed` with `{"allowed": true}` (`system:tenants:production`). Until then, putting a project in production is refused with `403 production-not-allowed`, and projects already in production stay so. `KINDGI_PRODUCTION_NEXT_URL` names where a refused tenant goes to be allowed ([Put a project in production](https://docs.kindgi.com/v0.1.8/guides/projects/organize-by-org-and-project.md#put-a-project-in-production)). The `projectsInProduction` limit caps how many of its projects are in production at once. ## Suspend and restore Suspending takes `system:suspend`, which no other scope includes, and can record a `reason`. A second suspend keeps the first one's time and reason, and restoring clears both. A closing tenant can't be suspended (`409 tenant-closing`). - **A tenant:** `PUT /v1/system/tenants/{tenantId}/suspended` with `{"suspended": true, "reason": "…"}`. It takes hold within 5 seconds. Its own keys and sessions still read and export it. Nothing runs (its running runs are cancelled), and any other write is refused (`409 tenant-suspended`), but for deletes, signing in and out, revoking, and erasing or searching memory. Its schedules and triggers are skipped, not changed, and a webhook whose signature checks is answered `202` and skipped. At the restore, nothing that came due meanwhile runs late: a schedule fires next at its first time after the restore. - **A project:** `PUT /v1/system/tenants/{tenantId}/projects/{projectId}/suspended`. Nothing new runs in it (`409 project-suspended`), and no pack service's endpoint is recorded for it. Its running runs are cancelled, its schedules and webhooks are skipped, and its deployments list to retire. The rest of the tenant is untouched. - **A pack:** `PUT /v1/system/tenants/{tenantId}/deployments/{deploymentId}/suspended`. Its calls fail with `pack-suspended`, never going to an older deployment, and it lists to retire. Restored, it serves again from the endpoint recorded for it. If its endpoint is pending, or was cleared meanwhile, it lists as needing a pack service. ## Close and purge a tenant `POST /v1/system/tenants/{tenantId}/close` starts the end of the tenant's contract, and there's no way back. The tenant is `closing`. For its 30-day export window, its own keys and sessions still read and export it, and its people can still sign in. Nothing runs (its running runs are cancelled), and writes are refused as for a suspended tenant, with `409 tenant-closing`. Its deployments list to retire. Once the window has passed, purge it. Its keys and sessions go with its rows. Until then, nothing of the tenant runs, and it writes only deletes, revokes, erasures, exports and sign-ins. One of those during the purge can leave a row: the purge exits `1` and names it, and running it again finishes it. When nothing of the tenant is left, the purge marks it purged. Each server keeps a tenant's state for about 5 seconds, so within about 5 seconds of the mark nothing of it signs in or is served. The purge waits three times that, then deletes again: work of the tenant under way at the mark, such as a request that started before it, can still write a row. ```sh kindgi-runtime tenants purge --confirm --record ``` ```text tenants purge: tenant 1ccae8e9-3dc8-4db9-b393-ba3d44e7f86e (acme-shop) artifacts: 1 row deleted audit_events: 4 rows deleted authz_store_registry: 1 row deleted identity_users: 1 row deleted adapter_registry: 6 rows deleted memory_erasures: 1 row deleted memory_facts: 2 rows deleted projects: 1 row deleted artifacts' bytes: 1 deleted OpenFGA store: 01M4WPFNJJ7YY6RXSJK1KB9W7D deleted No table holds a row of it now, and no bytes are under its prefix; its tenants row stays, marked purged. record: purged/1ccae8e9-3dc8-4db9-b393-ba3d44e7f86e.json written ``` - **It deletes the tenant for good:** every row of it, its artifacts' bytes and its OpenFGA store. Its tenants row stays, marked purged, with nothing left that signs in. - **`--dry-run`** says what would be deleted, and changes nothing. - **Refused (exit 2)** for a tenant that isn't closing or is still in its window, or one with artifacts or an OpenFGA store when `KINDGI_ARTIFACTS` or `KINDGI_OPENFGA_API_URL` isn't set as the server has it. **Exit 1** when rows or bytes of it are left, named: run it again. - **With a key for OpenFGA** ([Require a key for OpenFGA](https://docs.kindgi.com/v0.1.8/deploy/authorization.md#require-a-key-for-openfga)), purging a tenant with an OpenFGA store needs runtime 0.1.8.1 or later. An earlier one doesn't send the key, or the ID token behind Cloud Run IAM, so the store's delete is refused, and the purge exits `2` with `OpenFGA answered to deleting store `. By then the tenant's artifacts' bytes are deleted, but none of its rows. - **Exit 1, too, when work in flight at the mark wrote rows of it:** each is named (`written after its mark: …`) and deleted. One that needs a setting the run doesn't have is left, with why (`written after its mark, not deleted: …`): run it again with that setting, or report it. - **A day later,** run it again with `--dry-run`, and without it if it lists anything. On a tenant already purged, the purge deletes what's of it, and its mark and its record stay as they were. Nothing should write to a tenant a day after its mark, so report the tables it names. `kindgi-runtime check purged-tenants` lists every tenant marked purged that still holds rows, by table and count: rows only, not the bytes under its prefix. - **`--record`** (`local:` or `gcs:[/]`) keeps a record of each purge. After a [restore](https://docs.kindgi.com/v0.1.8/deploy/back-up-and-restore.md), and before the runtime serves, `kindgi-runtime tenants repurge --from ` purges again every tenant a record names. On one still marked purged, it deletes what's of it, and leaves its mark and its record as they were. ## After each upgrade: tenants repair A runtime serving many tenants sets nothing up for each tenant when it starts. After every upgrade that adds a step for each tenant, run this once: ```sh kindgi-runtime tenants repair --all ``` ```text tenant 6aa3383b-3cd9-4e02-b9e4-6234fb4663a2 (tenant-1791937260612-75bcb7): Default project kept; authorization store kept; 0 tuples added; adapters: 6 kept up to date, 0 removed; sign-in providers reconciled 1 tenant repaired. ``` It adds only what's missing: the Default project, the authorization store's tuples and current model, the shipped adapters, and the sign-in providers. It finishes a create whose setup didn't finish, if the create named a first admin. Run it again at will. `--tenant ` repairs one tenant. `--providers` redoes only sign-in's providers, after `KINDGI_PUBLIC_URL` changes. It exits 0 when every tenant is set up, and 1 when something is left for you, named: ```text tenant f309aaca-024a-4fb9-aff8-ef76c4c40e16 (tenant-a): Default project created; no authorization store, and none can be made without naming its first admin: left for the operator; 0 tuples added; adapters: 6 kept up to date, 0 removed; sign-in providers reconciled 1 tenant gone through; 1 left for the operator (named above). ``` ## Switch a one-tenant runtime to many The first start with many tenants keeps the tenant it served, with its data, keys and people, as the first of many. 1. **Run OpenFGA with a preshared key,** and point the runtime at it ([Require a key for OpenFGA](https://docs.kindgi.com/v0.1.8/deploy/authorization.md#require-a-key-for-openfga)). 2. **[Upgrade](https://docs.kindgi.com/v0.1.8/deploy/operate.md#upgrade-the-runtime) and start it still as one tenant,** `KINDGI_TENANT_ID` kept. If the tenant has no authorization store yet, because it ran without OpenFGA, that start sets one up. Going straight to many tenants would leave it without one, and `tenants repair` can't make one. The store this start sets up makes only the seed user an admin. For such a tenant, this step needs runtime 0.1.8.1 or later: with an older runtime, the people and service accounts the tenant added stay refused, because `tenants repair` doesn't put their access back. 3. **Start it with many tenants:** `KINDGI_TENANCY=multi` and the [settings above](https://docs.kindgi.com/v0.1.8/deploy/serve-many-tenants.md#the-settings-it-starts-only-with), with `KINDGI_TENANT_ID`, `KINDGI_SEED_USER_ID` and the rest that name one tenant removed ([What changes](https://docs.kindgi.com/v0.1.8/deploy/serve-many-tenants.md#what-changes)). If the tenant has its own identity provider, list its domains in `KINDGI_AUTH_VERIFIED_DOMAINS`. If the emailed sign-in link is on, set the Turnstile keys: the start stops without them. `KINDGI_API_TOKEN` is now the operator's and acts in no tenant, so whoever used it as the tenant's admin needs another way in first. A tenant key minted before the switch keeps working, and so does the tenant's sign-in. 4. **Run `kindgi-runtime tenants repair --all`** once. If you switched a tenant that ran without OpenFGA with a runtime older than 0.1.8.1, upgrade to 0.1.8.1 or later and run it again: it puts back the access of the people and service accounts the tenant added. 5. **Set the tenant's image repository:** `PUT /v1/system/tenants/{tenantId}/image-prefix` with `{"prefix": "/"}`. 6. **Deploy each pack again.** Its deployments from before have no routes, and the banner says how many: ```text ⚠ Per-deployment routing: 1 deployment registered before it has no routes; deploy it again. The same image is enough. ``` Its `endpoint` is then `pending` until you record its pack service. ## Next - The system routes in full: [system tenants](https://docs.kindgi.com/v0.1.8/reference/api/operations/tags/system-tenants.md), [system keys](https://docs.kindgi.com/v0.1.8/reference/api/operations/tags/system-keys.md) and [system deployments](https://docs.kindgi.com/v0.1.8/reference/api/operations/tags/system-deployments.md). - [Back up and restore](https://docs.kindgi.com/v0.1.8/deploy/back-up-and-restore.md), then `tenants repurge`.