Secrets
How the stacks get their keys through Sealed Secrets. Every key the seal command writes, the Secret it lands in and who reads it, the files to back up, and how to rotate a key or restore on a new cluster.
The keys of a stack live in two git-ignored files on the machine that runs the seal command: infra/stack/secrets/<stack>.env and infra/stack/secrets/<stack>.restate.pem. The seal command, infra/cmd/seal, encrypts them with the public certificate of the Sealed Secrets controller and writes the result as SealedSecrets to infra/stack/secrets/<stack>/, which is committed. Only the controller, inside the cluster, can turn them back into Secrets.
flowchart TB env["dev.env / prod.env<br/>+ .restate.pem<br/>(git-ignored)"] --> seal["infra/cmd/seal"] cert["controller certificate"] -.->|"kubeseal fetches"| seal seal --> yaml["infra/stack/secrets/#lt;stack#gt;/*.yaml<br/>SealedSecrets, committed"] yaml -->|"git push"| argo["Argo CD<br/>stack-dev, stack-prod"] argo --> ss["SealedSecret resources"] ss --> ctrl["sealed-secrets-controller<br/>kube-system"] ctrl -->|"decrypts"| sec["Secrets"] sec --> pods["server, rustfs, ai-gateway,<br/>Restate, Jobs"]
The Sealed Secrets model
The sealed-secrets Application (infra/platform/sealed-secrets.yaml, chart 2.22.0) runs the controller in kube-system as sealed-secrets-controller, the name kubeseal looks for. On its first start the controller creates a key pair and keeps it in a Secret labelled sealedsecrets.bitnami.com/sealed-secrets-key. kubeseal encrypts with the public half; the private half never leaves the cluster.
The seal command calls kubeseal without --scope, so each SealedSecret uses the default strict scope: it decrypts only under its own name and namespace. A SealedSecret copied to the other stack does not open there.
The chart includes every file in infra/stack/secrets/<stack>/. When the directory is empty, the render fails with infra/stack/secrets/<stack>/ holds no sealed secrets: run go -C infra run ./cmd/seal <stack>, commit and push, and the stack’s Application shows that error until the files arrive.
What the seal command does
Run it from the umbrella root:
go -C infra run ./cmd/seal dev|prod [-dry-run]-C infra makes infra/ the working directory, so the default paths are relative to it. -dir sets the secrets directory (default stack/secrets), -example the template (default stack/secrets/example.env) and -image the server image that runs /envgen (default ghcr.io/shopnexus/server:main). Flags may come before or after the stack name. An extra argument makes the command print its usage and exit with status 2.
-
Create the .env file
On the first run
/envgencopiesinfra/stack/secrets/example.envto<stack>.envwith mode 600. On later runs it adds the variables thatexample.envhas and<stack>.envlacks, with their example values. -
Generate the missing keys
For each generated key that has no value in
<stack>.env, it makes one. The JWT, console and Restate keys come from/envgenin the server image (-image), which runs in Docker. The rest are random bytes from Go’scrypto/rand, hex-encoded. Both fill in an emptyKEY=line where it is and append a missing one. -
Check the operator's values
It stops with
seal: .../<stack>.env: set OPENROUTER_API_KEYwhile that value is empty. For prod it stops whileFCM_CREDENTIALSis empty, withprod needs FCM_CREDENTIALS (APP_ENV=prod refuses to only log pushes). -
Seal four Secrets
It builds each Secret itself, pipes it to
kubeseal --format yaml, and writesserver-secrets.yaml,restate-secrets.yaml,litellm-secrets.yamlandlitellm-db.yamltoinfra/stack/secrets/<stack>/.kubesealfetches the certificate from the cluster that kubectl points at. The command ends withsealed into .../infra/stack/secrets/<stack>; commit and push.
A second run finds every key already set and seals the same values. kubeseal encrypts with a fresh random session key each time, so the sealed files change anyway.
Keys
| Key | Made by | Secret | Read by |
|---|---|---|---|
JWT_KEY_ID | /envgen: k and the UTC time, e.g. k20261104120000 | server-secrets | server |
JWT_PRIVATE_KEY | /envgen: base64 of a 32-byte Ed25519 seed | server-secrets | server, which signs access tokens with it |
CONSOLE_KEY | /envgen: base64 of 32 random bytes | server-secrets | server, which derives the TOTP sealing key and the sign-in lock key from it |
RESTATE_IDENTITY_KEY | /envgen: the public key, publickeyv1_… | server-secrets | server, which rejects Restate requests not signed by the matching private key |
<stack>.restate.pem | /envgen: the Ed25519 private key | restate-secrets as private.pem | Restate (requestSigningPrivateKey) |
OBJECT_STORE_ACCESS_KEY | 12 random bytes, hex | server-secrets; restate-secrets as AWS_ACCESS_KEY_ID | server, RustFS (RUSTFS_ACCESS_KEY), Restate snapshots |
OBJECT_STORE_SECRET_KEY | 24 random bytes, hex | server-secrets; restate-secrets as AWS_SECRET_ACCESS_KEY | server, RustFS (RUSTFS_SECRET_KEY), Restate snapshots |
LLM_API_KEY | sk- and 24 random bytes, hex | server-secrets | server; the ai-gateway-keys Job registers it as the key server-llm |
STT_API_KEY | sk- and 24 random bytes, hex | server-secrets | server; the ai-gateway-keys Job registers it as the key server-stt |
EMBED_INDEX_API_KEY | sk- and 24 random bytes, hex | server-secrets | server, when it indexes listings; the ai-gateway-keys Job registers it as the key server-embed-index |
EMBED_SEARCH_API_KEY | sk- and 24 random bytes, hex | server-secrets | server, when it embeds a query; the ai-gateway-keys Job registers it as the key server-embed-search |
LITELLM_MASTER_KEY | sk- and 24 random bytes, hex | litellm-secrets | ai-gateway, the ai-gateway-keys Job |
LITELLM_DB_PASSWORD | 24 random bytes, hex | litellm-db as password, with username: litellm | CloudNativePG (the managed role litellm), ai-gateway |
OPENROUTER_API_KEY | you: the stack’s OpenRouter key, with a daily credit limit set on OpenRouter | litellm-secrets | ai-gateway, for every alias on OpenRouter |
FCM_CREDENTIALS | you: a Firebase service account’s JSON in base64 | server-secrets | server, to send pushes |
FCM_CREDENTIALS may stay empty on dev, where the server logs pushes instead of sending them. APP_ENV=prod refuses it empty, so the seal command for prod does too. OPENROUTER_API_KEY is required on both stacks.
server-secrets, litellm-secrets and litellm-db land in shopnexus-<stack>. restate-secrets lands in restate-<stack>, the namespace of the RestateCluster. The server also gets every key of server-secrets in the migrate, objectstore-init and objectstore-init-snapshots Jobs, which run the server image with the same environment.
CloudNativePG creates one more Secret, postgres-app, whose uri is the server’s DATABASE_URL. JWT_PREVIOUS_KEY_ID and JWT_PREVIOUS_PUBLIC_KEY are empty strings in infra/stack/env/server.env, which the chart puts in the server-env ConfigMap and the root docker-compose.yml reads with env_file.
Files to back up
| File | Holds | Without it |
|---|---|---|
infra/stack/secrets/dev.env, prod.env | every key in the table above, in plain text | a reseal generates new keys. A new CONSOLE_KEY makes every stored TOTP secret unreadable; a new JWT key rejects access tokens issued before it, for up to their 15-minute life |
infra/stack/secrets/dev.restate.pem, prod.restate.pem | Restate’s private signing key | /envgen stops, and the seal command prints its message after envgen in <image>: <stack>.env has RESTATE_IDENTITY_KEY but <stack>.restate.pem is missing: restore <stack>.restate.pem from the backup |
| the controller’s key | the private key that opens every SealedSecret in git | the committed SealedSecrets cannot be opened on a new cluster; you reseal from the .env and .pem files |
Export the controller’s key with:
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.yamlKeep the five files in a password manager or another store that is not this machine.
While a cluster still runs, the decrypted values are also in its Secrets, for example kubectl -n shopnexus-prod get secret server-secrets -o yaml and kubectl -n restate-prod get secret restate-secrets -o yaml. That is a way to rebuild a lost .env before the cluster goes.
File permissions
/envgen creates <stack>.env and <stack>.restate.pem with mode 600, running in Docker as your own uid and gid, so the files belong to you. The seal command then sets <stack>.env to mode 600 at every run, which also repairs an older file left at 644, and writes it back with that mode. Other users of the host cannot read either file. The chart’s .helmignore leaves secrets/*.env and secrets/*.pem out of the chart, and the umbrella’s .gitignore ignores them; only example.env is tracked.
Dry run
-dry-run prints the four plain Secrets to stdout, as JSON, instead of sealing them. It needs no cluster and no kubeseal, so it shows what the seal command would seal before the controller exists.
It still needs Docker, creates <stack>.env, generates the missing keys and stops on an empty OPENROUTER_API_KEY, like a real run. The output holds every key in plain text; do not paste it into a ticket or a chat.
Rotating a key
-
Remove the line
Delete the whole line of the key from
<stack>.env, including the name. An emptyKEY=line also works: the seal command writes the new value into it. For the JWT key, delete bothJWT_KEY_IDandJWT_PRIVATE_KEY:/envgenwrites the pair when either is missing. -
Reseal and push
Terminalgo -C infra run ./cmd/seal prod git add infra/stack/secrets/prod git commit -m "infra: rotate <key> in prod" git push -
Restart the Restate pods if needed
Argo CD applies the new SealedSecrets, the controller updates the Secrets, and the workloads in the table below restart on their own or by hand, as it says.
| Workload | On a reseal |
|---|---|
server (RestateDeployment) | a new version starts, because the pod template carries checksum/secrets, a hash of every file in infra/stack/secrets/<stack>/ |
| RustFS (StatefulSet) | restarts, through the same checksum/secrets |
ai-gateway (Deployment, Application ai-<stack>) | keeps the old values until kubectl -n shopnexus-<stack> rollout restart deploy/ai-gateway; it carries no checksum of the stack’s secrets |
migrate, objectstore-init, objectstore-init-snapshots, ai-gateway-keys | run at every sync of their Application anyway |
Restate pods (RestateCluster) | keep the old values until they restart; the chart puts no checksum on them |
Each key has its own effect:
- A new
LLM_API_KEYmakesai-gateway-keysfail to update the old key and create the new one at the next sync ofai-<stack>. The old virtual key stays in the gateway’s database until you delete it there. - A new
LITELLM_DB_PASSWORDis set on thelitellmrole by CloudNativePG; restartai-gatewayby hand to use it. - A new pair of object store keys reaches RustFS and the server through the checksum, and reaches Restate’s snapshot writer only when the Restate pods restart. Rotating them on a RustFS volume that already has data has not been exercised.
- A new
CONSOLE_KEYcannot be rotated without loss. The server has no previous-key setting for it, and the TOTP secrets in Postgres are sealed under a key derived from it.
The Restate key
The private key in restate-secrets and the public key in server-secrets must match. A server pod rejects every request signed by any other key. During a rotation the two change at different times: the new server version starts with the new public key through the checksum, while the Restate pods sign with the old private key until they restart. Requests from Restate to the new version fail in that window, and the old server version, still draining, rejects the new key after Restate restarts.
The seal command refuses a .env that has RESTATE_IDENTITY_KEY while its .pem is missing. Without that check, a restore that forgot the .pem would generate a new pair and rotate the key by accident. To rotate on purpose, delete the RESTATE_IDENTITY_KEY line and the .pem file, run the seal command, push, and then restart the Restate pods one at a time:
kubectl -n restate-prod get pods -l app.kubernetes.io/name=restate
kubectl -n restate-prod delete pod <one pod>Deleting only the line keeps the old key: /envgen reuses a .pem that exists and writes its public key back. A deliberate rotation of the Restate key has not been exercised on a cluster.
The JWT key
The server verifies access tokens against JWT_KEY_ID and, while a rotation is in progress, also against JWT_PREVIOUS_KEY_ID with JWT_PREVIOUS_PUBLIC_KEY. The two previous-key variables are set together or not at all, and the previous ID must differ from the current one. Access tokens live 15 minutes, so the previous key has to stay for at least 15 minutes after the new key is live.
In the cluster, the chart fixes both previous-key variables to empty strings in server-env, and the seal command does not seal them. A rotation without loss therefore needs a change first: add JWT_PREVIOUS_KEY_ID and JWT_PREVIOUS_PUBLIC_KEY to the keys of server-secrets in infra/cmd/seal/main.go. The pod reads server-secrets after server-env, and Kubernetes gives the later envFrom source precedence, so the sealed values override the empty ones. To get the old public key from the old seed in <stack>.env:
seed=$(grep -m1 '^JWT_PRIVATE_KEY=' infra/stack/secrets/prod.env | cut -d= -f2-)
{ printf '302e020100300506032b657004220420' | xxd -r -p; base64 -d <<<"$seed"; } \
| openssl pkey -inform DER -pubout -outform DER | tail -c 32 | base64Write the old ID and that public key as JWT_PREVIOUS_KEY_ID and JWT_PREVIOUS_PUBLIC_KEY, delete the old JWT_KEY_ID and JWT_PRIVATE_KEY lines, reseal and push. After 15 minutes, delete the two previous-key lines and reseal again.
Restoring on a new cluster
The SealedSecrets in git open again once the new controller has the old key.
-
Create the cluster
Run
k3d cluster createwithinfra/cluster/k3d-direct.yamlorinfra/cluster/k3d-behind-proxy.yamlas in Bring up a cluster, and wait untilsealed-secretsis Synced and Healthy. -
Apply the old key
Strip the fields the old cluster set, then apply:
Terminalyq 'del(.items[].metadata.resourceVersion, .items[].metadata.uid, .items[].metadata.creationTimestamp)' \ sealed-secrets-key.yaml | kubectl apply -f - -
Restart the controller
The controller loads its keys at start.
Terminalkubectl -n kube-system rollout restart deployment sealed-secrets-controllerIt now opens SealedSecrets made with the old key and with the key it created on its first start.
-
Restore the plain files
Copy
dev.env,prod.env,dev.restate.pemandprod.restate.pemback toinfra/stack/secrets/and runchmod 600on them, so a later reseal keeps the same keys.
Without the controller’s key, reseal against the new controller.
-
Create the cluster
Create the cluster with
k3d cluster createand wait forsealed-secretsto be Healthy. -
Restore the plain files
Copy the four files back to
infra/stack/secrets/with mode 600. Restore each.pemwith its.env; the seal command stops if one is missing. -
Reseal and push
Terminalgo -C infra run ./cmd/seal dev go -C infra run ./cmd/seal prod git add infra/stack/secrets/dev infra/stack/secrets/prod git commit -m "infra: reseal for the new cluster" git pushThe seal command reuses every key in the
.envfiles, so the stacks get the same keys under new ciphertext.
After a reseal with the second tab, back up the new controller’s key, because the SealedSecrets in git are now sealed with it.
Key rules
<stack>.envand<stack>.restate.pemare never committed. Only the sealed YAML underinfra/stack/secrets/<stack>/is.<stack>.env,<stack>.restate.pemand the controller’s key are backed up off the host after every change.CONSOLE_KEYis permanent for the life of a stack’s database.- A
.pemis restored with its.env. The Restate key is rotated only on purpose, by deleting both the line and the file. - A JWT key is rotated with the old key in
JWT_PREVIOUS_KEY_IDandJWT_PREVIOUS_PUBLIC_KEYfor at least 15 minutes. - A key is changed in
<stack>.envand resealed. A Secret edited in the cluster is overwritten at the next reseal.
Troubleshooting
kubeseal: error: cannot fetch certificate
kubeseal reaches the controller through the current kubectl context. Check kubectl config current-context names the k3d cluster and that kubectl -n kube-system get deploy sealed-secrets-controller is ready.
A SealedSecret does not become a Secret
Read its status and the controller’s log:
kubectl -n shopnexus-prod get sealedsecret server-secrets -o jsonpath='{.status.conditions}'
kubectl -n kube-system logs deploy/sealed-secrets-controller --tail=50no key could decrypt secret means the files were sealed for another cluster’s controller. Restore that controller’s key, or reseal against this one.
The server logs a missing or malformed variable
The server names every variable it is missing at start. Check that the key is among the keys of server-secrets in infra/cmd/seal/main.go and has a value in <stack>.env, then reseal. RESTATE_IDENTITY_KEY must start with publickeyv1_.