Serve many tenants

View as Markdown

A runtime serves one tenant by default, KINDGI_TENANT_ID's. With KINDGI_TENANCY=multi it serves many. Each tenant is created through the system API, and each request's tenant comes from its key or session. This page is for the operator: whoever runs the runtime, its database and the services around it, for tenants who don't trust one another. The runtime's license applies as on any deployment (Licensing).

Commands that take the operator's token read it from $KINDGI_API_TOKEN, and the runtime's address from $KINDGI_URL.

What changes

No tenant is set on the server. KINDGI_TENANT_ID is refused: "KINDGI_TENANT_ID (or --tenant, or the config file's tenant) is refused with KINDGI_TENANCY=multi: a runtime serving many tenants takes each request's tenant from its key or session. Unset it."
Nothing else that names one tenant is taken either: KINDGI_SEED_USER_ID, KINDGI_PACK_DIR, KINDGI_DEV_CONSOLE_LOGIN, the dev echo provider and the config file's mcp.endpoints are each refused, by name: "KINDGI_TENANCY=multi serves many tenants, so it refuses what names one: seedUserId (KINDGI_SEED_USER_ID)."
The deployment's own token is the operator's. KINDGI_API_TOKEN has the full system capability, kindgi:system. It acts in no tenant and reaches /v1/system/* only.
A tenant's own identity provider is offered only on the domains you list in KINDGI_AUTH_VERIFIED_DOMAINS (Which domains find a workspace's identity provider). A runtime serving one tenant routes that tenant's domains with nothing to set. On a domain not listed, its people reach the provider only through its own link, and GitHub and the emailed link aren't refused to them (Who gets in).
The emailed sign-in link needs a captcha. Without the Turnstile keys (Sign-in links by email), the start stops: "The emailed sign-in link on a runtime that serves several tenants needs a captcha: set KINDGI_AUTH_TURNSTILE_SECRET (or KINDGI_AUTH_TURNSTILE_SECRET_PATH) and KINDGI_AUTH_TURNSTILE_SITE_KEY."
Each tenant is created through POST /v1/system/tenants, with its setup: its Default project and its authorization store.
The start sets nothing up per tenant. After an upgrade that adds a step for each tenant, run tenants repair.
Background work keeps up every tenant that's set up, and starts new work (schedules, deliveries, notices) only in the active ones.

The banner says so:

  Tenants: many (KINDGI_TENANCY=multi), each created through POST /v1/system/tenants
  Token:   kgi_bt_… (provided), the operator's: kindgi:system, in no tenant, /v1/system/* only
  …
  Background work: upkeep in every tenant set up; new work (schedules, deliveries, notices) in the active ones
  …
  Tenant host access: isolated (stdio MCP endpoints refused; tenant-chosen hosts can't reach the metadata server, this host or private networks; providers can't sign in with this server's own credentials; KINDGI_TENANT_HOST_ACCESS)
  OpenFGA calls carry its preshared key (KINDGI_OPENFGA_PRESHARED_KEY)
  Pack services: one per deployment, at the endpoints recorded through the system API (1 allowed host pattern)

Excerpt

The settings it starts only with

A runtime serving many tenants starts only with every setting that keeps them apart, and the two that let you allow and count them:

Setting	What it does
`KINDGI_TENANT_HOST_ACCESS=isolated`	What a tenant registers reaches public hosts only: not the metadata server, this host or private networks, and never with the server's own cloud credentials. The server then refuses `KINDGI_DEV`, `KINDGI_WEBHOOK_PRIVATE_NETWORKS=allow` and `KINDGI_AUTH_PRIVATE_IDP_ORIGINS`.
`KINDGI_PACK_ROUTING=per-deployment`	Each call goes to the pack service of the deployment that shipped the tool or check, never to one pack service every tenant shares. It needs `KINDGI_SECRETS_BACKEND=postgres`, because each endpoint's token is sealed under the secrets' key. `KINDGI_PACK_SERVICE_URL`, `KINDGI_PACK_SERVICE_TOKEN` and `KINDGI_PACK_SERVICE_AUTH` are refused.
`KINDGI_PACK_ENDPOINT_HOSTS`	The hosts a deployment's pack service may be on, comma-separated: a host name, `.` and a domain, or `-`, a tail and a domain.
`KINDGI_OPENFGA_API_URL` and `KINDGI_OPENFGA_PRESHARED_KEY`	The authorization store, and the key it requires of its callers, so only the server reads and writes tenants' grants (Require a key for OpenFGA).
`KINDGI_PRODUCTION_GATE=on`	A tenant puts a project in production only once you allow it (Limits and production).
`KINDGI_USAGE_METERING=on`	Each run is counted when it's created and when it ends, so tenants' limits read what each one did.

A start that lacks any of them is refused, naming each, and exits 2:

KINDGI_TENANCY=multi serves tenants who don't trust one another, so it starts only with the hosted profile's settings: KINDGI_PACK_ENDPOINT_HOSTS must name the hosts a deployment's endpoint may be on (it's empty, so every endpoint would be refused).

Environment variables has each setting in full.

At every start, the server checks that OpenFGA takes its key. If OpenFGA refuses the server's call, the start stops, exits 1 and names the settings to fix. If OpenFGA doesn't answer, the start goes on and logs a warning: "OpenFGA at isn't answering (…). The server is up; requests that need authorization fail until it answers."

The operator's keys

Keep the deployment's own token for yourself. Give each service that acts for you, such as sign-up, the service that runs pack services, or billing, its own system key (kgi_sy_…), with the least scope it needs:

Scope	What it reaches
`system:tenants`	Creating, listing, reading and closing tenants
`system:tenants:create`	Creating tenants only: each with a first admin, production blocked, no image repository, not a test tenant
`system:tenants:image-prefix`	Setting a tenant's image repository
`system:tenants:production`	A tenant's production switch
`system:tenants:limits`	A tenant's limits
`system:tenants:contacts`	A tenant's admins' contact addresses, for service notices
`system:suspend`	Suspending and restoring a tenant, a project or a pack
`system:deployments`	The deployments and their pack services' endpoints
`system:license`	Reading the deployment's license, for its renewal
`kindgi:system`	All of them: a full key

curl -sS -X POST "$KINDGI_URL/v1/system/keys" \
  -H "Authorization: Bearer $KINDGI_API_TOKEN" \
  -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"scopes": ["system:deployments"], "label": "pack services", "expiresAt": "2027-10-01T00:00:00Z"}'

The token comes back once, in the answer's token.
A system key acts in no tenant. Every tenant route refuses it (system-key-on-tenant-route). GET /v1/system/whoami tells a service which key it holds.
Only the full capability mints, lists and revokes keys. A key can't outlive the key minting it. A full key minted by another system key needs an expiresAt, so only the deployment's own token mints one that never expires.
Revoking a key revokes every key minted from it, however deep: POST /v1/system/keys/{keyId}/revoke.
Every request a system key makes is recorded, and a request to the system routes that changes something is recorded before it acts. When that record can't be written, nothing is done (access-record-unavailable).

Create a tenant

curl -sS -X POST "$KINDGI_URL/v1/system/tenants" \
  -H "Authorization: Bearer $KINDGI_API_TOKEN" \
  -H 'Content-Type: application/json' \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{
    "name": "Acme Store",
    "slug": "acme-store",
    "imagePrefix": "registry.acme.com/tenants/acme-store",
    "firstAdmin": { "email": "pat@acme-store.example", "name": "Pat" }
  }'

With firstAdmin, that person becomes the tenant's admin and member by email, and signs in with the runtime's own sign-in (Turn on sign-in). No key comes back: once signed in, they make the tenant's keys (API keys).
Without firstAdmin, the answer carries the tenant's first admin key, once. Store it, because nothing shows it again, and without it nothing gives the tenant another admin key. Only the full capability creates a tenant that way.
Production starts blocked. "productionAllowed": true allows it from the start, and takes system:tenants:production.

imagePrefix is the repository every image the tenant deploys must sit in, <host>/<path>. Without one, the tenant can't register a deployment. You can set it later, with PUT /v1/system/tenants/{tenantId}/image-prefix:

{
  "error": {
    "code": "image-prefix-unset",
    "message": "This tenant has no image repository set, so it can't register a deployment yet: its operator sets one.",
    "requestId": "req-f5000e71-d1ec-4ae8-984b-1778ece7b597"
  }
}

Send an Idempotency-Key with every create. If the setup didn't finish, the tenant waits, frozen, and the same request finishes it. A repeat with the first request's key gets the first answer again, except an admin key, which is never sent twice. Without the key, a lost answer can't be told apart from another tenant's slug (tenant-slug-taken).
A setup left pending shows at every start: the log names each tenant (the first 10; it counts the rest, and tenants repair --all names them all) and says to retry the same create, or to run kindgi-runtime tenants repair --tenant <id> if its create named a first admin.

A pack service for each deployment

Each deployment runs in its own pack service, which you run: an orchestrator, or a script, with a system:deployments key. The runtime records where each service is, and checks it. It never creates or deletes one.

List the deployments that need a service: GET /v1/system/deployments?endpoint=missing, oldest first across tenants. A deployment needs one while it's routed: it's the newest with routes for some tool version or pack check, it has no endpoint or a cleared one, and nothing about it is closing or suspended. Each entry has the tenant's id, the deployment's id, its pack (and project, if it has one), and its image: the digest-pinned imageRef, its digest, the artifact version and when it went live. It has nothing else of the tenant.
Run the image as a service whose host begins with the deployment's own service name, packServiceName(tenantId, deploymentId) from @kindgi/api (kp- and 20 characters), then - or .. Give it a token of its own, its KINDGI_PACK_SERVICE_TOKEN: 16 to 512 printable characters, no spaces.
Record its endpoint:
Terminal window
```
curl -sS -X PUT "$KINDGI_URL/v1/system/tenants/$TENANT_ID/deployments/$DEPLOYMENT_ID/endpoint" \
  -H "Authorization: Bearer $KINDGI_SYSTEM_KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"url\": \"https://$SERVICE_HOST\", \"auth\": \"token\", \"token\": \"$PACK_TOKEN\"}"
```
auth is token, or google-id-token to add an ID token for the URL (Cloud Run's invoker check). The runtime records the endpoint only once every check passes:
- the URL is https://<host> (endpoint-url-invalid);
- its host is in KINDGI_PACK_ENDPOINT_HOSTS (endpoint-host-not-allowed);
- its host names the deployment's service (endpoint-name-mismatch);
- the image is in the tenant's repository (image-not-allowed);
- the service answers /v1/info as this deployment, with the kindgi-tenant-id header (endpoint-unreachable, endpoint-info-mismatch).
The deployment is then ready. A refusal is also recorded as the deployment's problem, and the tenant sees its code.
While you can't make a service, say why: PUT …/deployments/{deploymentId}/endpoint/problem with a code: tenant-not-provisioned, capacity-unavailable, service-start-failed or image-not-allowed. The tenant sees the code and when, so a deploy that waits for its pack service says why at once.
Retire what should go: GET /v1/system/deployments?endpoint=retire lists deployments with an endpoint whose service should go, each with its reason. Clear the endpoint first, with DELETE …/endpoint, then delete the service. A superseded deployment is one whose every tool version and check newer deployments ship.

A call to a deployment's tool never goes anywhere else. With no endpoint yet it fails with pack-endpoint-pending, and after a clear with pack-endpoint-cleared. A tenant sees its deployment's endpoint: pending (with the problem, if one is recorded), ready or cleared.

Limits and production

Limits: GET and PUT /v1/system/tenants/{tenantId}/limits (system:tenants:limits). PUT replaces them all: a limit left out isn't set, and isn't enforced. Read them, change one, and put them all back. The tenant reads its own on GET /v1/tenant.
Production: PUT /v1/system/tenants/{tenantId}/production-allowed with {"allowed": true} (system:tenants:production). Until then, putting a project in production is refused with 403 production-not-allowed, and projects already in production stay so. KINDGI_PRODUCTION_NEXT_URL names where a refused tenant goes to be allowed (Put a project in production). The projectsInProduction limit caps how many of its projects are in production at once.

Suspend and restore

Suspending takes system:suspend, which no other scope includes, and can record a reason. A second suspend keeps the first one's time and reason, and restoring clears both. A closing tenant can't be suspended (409 tenant-closing).

A tenant: PUT /v1/system/tenants/{tenantId}/suspended with {"suspended": true, "reason": "…"}. It takes hold within 5 seconds. Its own keys and sessions still read and export it. Nothing runs (its running runs are cancelled), and any other write is refused (409 tenant-suspended), but for deletes, signing in and out, revoking, and erasing or searching memory. Its schedules and triggers are skipped, not changed, and a webhook whose signature checks is answered 202 and skipped. At the restore, nothing that came due meanwhile runs late: a schedule fires next at its first time after the restore.
A project: PUT /v1/system/tenants/{tenantId}/projects/{projectId}/suspended. Nothing new runs in it (409 project-suspended), and no pack service's endpoint is recorded for it. Its running runs are cancelled, its schedules and webhooks are skipped, and its deployments list to retire. The rest of the tenant is untouched.
A pack: PUT /v1/system/tenants/{tenantId}/deployments/{deploymentId}/suspended. Its calls fail with pack-suspended, never going to an older deployment, and it lists to retire. Restored, it serves again from the endpoint recorded for it. If its endpoint is pending, or was cleared meanwhile, it lists as needing a pack service.

Close and purge a tenant

POST /v1/system/tenants/{tenantId}/close starts the end of the tenant's contract, and there's no way back. The tenant is closing. For its 30-day export window, its own keys and sessions still read and export it, and its people can still sign in. Nothing runs (its running runs are cancelled), and writes are refused as for a suspended tenant, with 409 tenant-closing. Its deployments list to retire.

Once the window has passed, purge it. Its keys and sessions go with its rows. Until then, nothing of the tenant runs, and it writes only deletes, revokes, erasures, exports and sign-ins. One of those during the purge can leave a row: the purge exits 1 and names it, and running it again finishes it.

When nothing of the tenant is left, the purge marks it purged. Each server keeps a tenant's state for about 5 seconds, so within about 5 seconds of the mark nothing of it signs in or is served. The purge waits three times that, then deletes again: work of the tenant under way at the mark, such as a request that started before it, can still write a row.

kindgi-runtime tenants purge <tenant id> --confirm <tenant slug> --record <where>

tenants purge: tenant 1ccae8e9-3dc8-4db9-b393-ba3d44e7f86e (acme-shop)
  artifacts: 1 row deleted
  audit_events: 4 rows deleted
  authz_store_registry: 1 row deleted
  identity_users: 1 row deleted
  adapter_registry: 6 rows deleted
  memory_erasures: 1 row deleted
  memory_facts: 2 rows deleted
  projects: 1 row deleted
  artifacts' bytes: 1 deleted
  OpenFGA store: 01M4WPFNJJ7YY6RXSJK1KB9W7D deleted
  No table holds a row of it now, and no bytes are under its prefix; its tenants row stays, marked purged.
  record: purged/1ccae8e9-3dc8-4db9-b393-ba3d44e7f86e.json written

It deletes the tenant for good: every row of it, its artifacts' bytes and its OpenFGA store. Its tenants row stays, marked purged, with nothing left that signs in.
--dry-run says what would be deleted, and changes nothing.
Refused (exit 2) for a tenant that isn't closing or is still in its window, or one with artifacts or an OpenFGA store when KINDGI_ARTIFACTS or KINDGI_OPENFGA_API_URL isn't set as the server has it. Exit 1 when rows or bytes of it are left, named: run it again.
With a key for OpenFGA (Require a key for OpenFGA), purging a tenant with an OpenFGA store needs runtime 0.1.8.1 or later. An earlier one doesn't send the key, or the ID token behind Cloud Run IAM, so the store's delete is refused, and the purge exits 2 with OpenFGA answered <status> to deleting store <id>. By then the tenant's artifacts' bytes are deleted, but none of its rows.
Exit 1, too, when work in flight at the mark wrote rows of it: each is named (written after its mark: …) and deleted. One that needs a setting the run doesn't have is left, with why (written after its mark, not deleted: …): run it again with that setting, or report it.
A day later, run it again with --dry-run, and without it if it lists anything. On a tenant already purged, the purge deletes what's of it, and its mark and its record stay as they were. Nothing should write to a tenant a day after its mark, so report the tables it names. kindgi-runtime check purged-tenants lists every tenant marked purged that still holds rows, by table and count: rows only, not the bytes under its prefix.
--record (local:<dir> or gcs:<bucket>[/<prefix>]) keeps a record of each purge. After a restore, and before the runtime serves, kindgi-runtime tenants repurge --from <where> purges again every tenant a record names. On one still marked purged, it deletes what's of it, and leaves its mark and its record as they were.

After each upgrade: tenants repair

A runtime serving many tenants sets nothing up for each tenant when it starts. After every upgrade that adds a step for each tenant, run this once:

kindgi-runtime tenants repair --all

tenant 6aa3383b-3cd9-4e02-b9e4-6234fb4663a2 (tenant-1791937260612-75bcb7): Default project kept; authorization store kept; 0 tuples added; adapters: 6 kept up to date, 0 removed; sign-in providers reconciled
1 tenant repaired.

It adds only what's missing: the Default project, the authorization store's tuples and current model, the shipped adapters, and the sign-in providers. It finishes a create whose setup didn't finish, if the create named a first admin. Run it again at will. --tenant <id> repairs one tenant. --providers redoes only sign-in's providers, after KINDGI_PUBLIC_URL changes. It exits 0 when every tenant is set up, and 1 when something is left for you, named:

tenant f309aaca-024a-4fb9-aff8-ef76c4c40e16 (tenant-a): Default project created; no authorization store, and none can be made without naming its first admin: left for the operator; 0 tuples added; adapters: 6 kept up to date, 0 removed; sign-in providers reconciled
1 tenant gone through; 1 left for the operator (named above).

Switch a one-tenant runtime to many

The first start with many tenants keeps the tenant it served, with its data, keys and people, as the first of many.

Run OpenFGA with a preshared key, and point the runtime at it (Require a key for OpenFGA).
Upgrade and start it still as one tenant, KINDGI_TENANT_ID kept. If the tenant has no authorization store yet, because it ran without OpenFGA, that start sets one up. Going straight to many tenants would leave it without one, and tenants repair can't make one. The store this start sets up makes only the seed user an admin. For such a tenant, this step needs runtime 0.1.8.1 or later: with an older runtime, the people and service accounts the tenant added stay refused, because tenants repair doesn't put their access back.
Start it with many tenants: KINDGI_TENANCY=multi and the settings above, with KINDGI_TENANT_ID, KINDGI_SEED_USER_ID and the rest that name one tenant removed (What changes). If the tenant has its own identity provider, list its domains in KINDGI_AUTH_VERIFIED_DOMAINS. If the emailed sign-in link is on, set the Turnstile keys: the start stops without them. KINDGI_API_TOKEN is now the operator's and acts in no tenant, so whoever used it as the tenant's admin needs another way in first. A tenant key minted before the switch keeps working, and so does the tenant's sign-in.
Run kindgi-runtime tenants repair --all once. If you switched a tenant that ran without OpenFGA with a runtime older than 0.1.8.1, upgrade to 0.1.8.1 or later and run it again: it puts back the access of the people and service accounts the tenant added.
Set the tenant's image repository: PUT /v1/system/tenants/{tenantId}/image-prefix with {"prefix": "<host>/<path>"}.
Deploy each pack again. Its deployments from before have no routes, and the banner says how many:
```
  ⚠ Per-deployment routing: 1 deployment registered before it has no routes; deploy it again. The same image is enough.
```
Its endpoint is then pending until you record its pack service.

The system routes in full: system tenants, system keys and system deployments.
Back up and restore, then tenants repurge.