Bring up a cluster
From a bare machine to the dev and prod stacks running on k3d, in the order the files in infra/ expect.
This tutorial takes a fork of the umbrella repo to one k3d cluster named shopnexus that runs two stacks: dev in shopnexus-dev and prod in shopnexus-prod. You run three commands: k3d cluster create with a config from infra/cluster/ creates the cluster, and k3s installs Argo CD in it; go -C infra run ./cmd/seal writes the secrets; and Hurl runs infra/smoke.hurl to check the result. Argo CD installs and syncs everything else from your fork.
Before you start
- A Linux machine with a public IP address and enough free memory for two stacks. The author’s bring-up plan stops to reconsider when less than 6 GB is available.
- Docker. The seal command also runs the server image with Docker to generate keys.
- k3d. The cluster was built with k3d v5.9.0; the author’s server runs v5.8.3. Both configs in
infra/cluster/,k3d-direct.yamlandk3d-behind-proxy.yaml, pin the node imagerancher/k3s:v1.37.1-k3s1. - kubectl (tested with v1.37.2), Helm (v4.3.1) and kubeseal (0.41.2).
- Go, to run the seal command in
infra/cmd/seal.infra/go.modasks for Go 1.27. - Hurl, to run the smoke test in
infra/smoke.hurl. - yq (v4.54.1) and the Helm plugin helm-unittest (v1.2.1), which only
infra/check.shneeds, to check the chart before you push a change. Install the plugin withhelm plugin install https://github.com/helm-unittest/helm-unittest --version v1.2.1 --verify=false. - A domain whose DNS you control, and a GitHub account to hold the fork.
Bootstrap order
flowchart TD up["k3d cluster create<br/>--config infra/cluster/k3d-#lt;mode#gt;.yaml"] --> k3d["k3d cluster shopnexus<br/>+ Traefik edge config"] k3d --> argo["Argo CD<br/>(Helm chart 10.11.7)"] argo --> root["Application root"] root --> w0["Wave 0: platform<br/>cnpg, sealed-secrets,<br/>restate-operator, image-updater"] w0 --> w1["Wave 1: stack-dev, stack-prod,<br/>ImageUpdater shopnexus"] w1 --> w2["Wave 2: ai-dev, ai-prod"] w1 -->|"ComparisonError:<br/>no sealed secrets"| seal["cmd/seal dev, cmd/seal prod<br/>commit and push"] seal --> sm1["Stack wave -1<br/>Namespace restate-#lt;stack#gt;"] sm1 --> s0["Stack wave 0<br/>postgres, rustfs, SealedSecrets"] s0 --> s1["Stack wave 1<br/>migrate, objectstore-init,<br/>RestateCluster"] s1 --> s2["Stack wave 2<br/>server, website, docs,<br/>console, Ingress"] s2 --> a0["ai-#lt;stack#gt; wave 0<br/>ai-gateway, ai-models"] a0 --> a1["ai-#lt;stack#gt; wave 1<br/>ai-gateway-keys"]
The root Application syncs infra/platform and infra/argocd from your fork. Its wave 0 installs the four platform operators, its wave 1 creates one Application per stack, and its wave 2 creates ai-dev and ai-prod. Each stack Application renders the Helm chart in infra/stack and applies it in its own waves. ai-<stack> renders the chart in chart/ of the ai repo into the same namespace once the stack Application is Healthy, so the stack’s Postgres already holds the litellm database.
Steps
-
Fork the repo and point it at your fork
Fork
shopnexus/shopnexuson GitHub and clone the fork onto the machine. Argo CD reads the fork without credentials, so the fork must be public.Set
repoURLto your fork’s clone URL in two files:infra/argocd/root.yaml, which has it twice, once forinfra/platformand once forinfra/argocd;infra/argocd/stacks.yaml, once instack-devand once instack-prod.
infra/argocd/ai.yamlreads theairepo,https://github.com/shopnexus/ai.git, once inai-devand once inai-prod. Point it at your fork ofaiif you change the hub.Set
domainininfra/stack/values-dev.yamlandinfra/stack/values-prod.yaml. The upstream values aredev.shopnexus.id.vnandshopnexus.id.vn; keep the same shape, with dev one label below prod.Each stack’s
ai-gatewaysends every model call to the model its alias names ingateway.aliasesof theairepo’schart/values.yaml, most of them on OpenRouter, so you need an OpenRouter key for each stack. The daily budget of each key is ininfra/argocd/ai.yaml. If you build your own images, change theghcr.io/shopnexus/*names ininfra/stack/values.yaml,infra/argocd/stacks.yamland theaichart’simages, and make the GHCR packages public.Commit and push to
main.You should see your clone URL in every
repoURLline:Terminalgrep -n repoURL infra/argocd/*.yaml -
Create the DNS records
Create four A records pointing at the machine’s public IP. With
example.comas the prod domain:Record Covers @example.com, the prod website*api.,media.,docs.andconsole.example.com*.devapi.dev.,media.dev.,docs.dev.andconsole.dev.example.comdevdev.example.com, the dev websiteThe
*record stops coveringdev.example.comonce*.devexists, so the dev website needs its owndevrecord (Edge and network explains why).You should see the machine’s IP for both the dev website and a host below it:
Terminaldig +short dev.example.com dig +short api.dev.example.com -
Create the cluster with k3d
Pick the edge mode.
directlets Traefik inside the cluster take ports 80 and 443 and get certificates from Let’s Encrypt.behind-proxyis for a machine where another proxy already holds those ports; Traefik then listens on127.0.0.1:18080and the proxy terminates TLS. A fresh machine usesdirect. The author’s server usesbehind-proxybehind a shared Caddy.ACME_EMAILis required; k3d fills${ACME_EMAIL}in the config from the environment. Traefik gets its certificates from Let’s Encrypt’s production server.TerminalACME_EMAIL=you@example.com k3d cluster create --config infra/cluster/k3d-direct.yamlTo try the setup against Let’s Encrypt’s staging server, whose rate limits are higher and whose certificates browsers do not trust, add
caServer: https://acme-staging-v02.api.letsencrypt.org/directoryunderacmeininfra/cluster/k3d-direct.yamlbefore you create the cluster. k3d copies the Traefik config into the cluster when it creates it, so a later edit to the file does not reach a running cluster.Traefik answers the TLS-ALPN-01 challenge on port 443, so port 443 must be open from the internet. Port 80 redirects to HTTPS with a 301. Edge and network has the port mapping and the Traefik settings of this mode.
Terminalk3d cluster create --config infra/cluster/k3d-behind-proxy.yamlThe proxy on the host terminates TLS and forwards plain HTTP to
127.0.0.1:18080. Traefik trusts theX-Forwarded-*headers that arrive from private addresses. The Caddy block on the author’s server lists every host of both stacks:/etc/caddy/conf.d/shopnexus.caddyshopnexus.id.vn, api.shopnexus.id.vn, console.shopnexus.id.vn, media.shopnexus.id.vn, docs.shopnexus.id.vn, dev.shopnexus.id.vn, api.dev.shopnexus.id.vn, console.dev.shopnexus.id.vn, media.dev.shopnexus.id.vn, docs.dev.shopnexus.id.vn { request_body { max_size 50MB } reverse_proxy 127.0.0.1:18080 }Replace the names with your domain, then run
caddy validate --adapter caddyfile --config <file>and reload Caddy. The proxy must pass theHostheader through unchanged, whichreverse_proxydoes by default: a presigned upload tomedia.is signed for that host. Plain HTTP is redirected by Caddy with a 308.The cluster has one server node and three agents, with the port mapping of the edge mode.
agent-0carries the labelrole=stateful, andagent-1andagent-2carryrole=app. The config’sfiles:list puts three manifests into the k3s manifests directory on the server node, and k3s applies them at its first start:- A
HelmChartConfigthat reconfigures the Traefik k3s ships. It is written inline in the k3d config of each mode. infra/cluster/argocd.yaml, a k3sHelmChartthat installs theargo-cdchart, version 10.11.7, into the namespaceargocd. Its values add a health check for Applications, without whichrootwould not wait for the platform to be healthy before starting the stacks.infra/argocd/root.yaml. k3s retries it until Argo CD’s CRDs exist. From then on Argo CD manages that file too.
When the server node restarts, k3s applies the copy of
root.yamlit got at cluster creation again. If the file in git has changed since, Argo CD’s self-heal puts the git version back within minutes.You should see
rootin the list once Argo CD is up:Terminalkubectl -n argocd get applications - A
-
Read what Argo CD shows
Terminalkubectl -n argocd get applicationsAfter a few minutes the four platform Applications
cnpg,sealed-secrets,restate-operatorandimage-updaterare Synced and Healthy.stack-devandstack-prodshow Unknown with a ComparisonError:infra/stack/secrets/dev/ holds no sealed secrets: run go -C infra run ./cmd/seal dev, commit and pushThe error is expected. The chart in
infra/stack/templates/secrets.yamlrefuses to render whileinfra/stack/secrets/<stack>/holds no SealedSecrets, because the stack cannot start without them. The secrets cannot exist yet:kubesealencrypts with the certificate of the Sealed Secrets controller, and that controller came up a minute ago in wave 0. Both stack Applications retry, with a backoff that grows from 30 seconds to 5 minutes, until the sealed files arrive in git.For the web UI, forward a port and read the initial admin password:
Terminalkubectl -n argocd port-forward svc/argocd-server 8080:443 kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d -
Seal the secrets
Run the seal command once per stack, from the umbrella root. It needs Docker, Go, and a kubectl context on this cluster, which k3d set as the current one.
Terminalgo -C infra run ./cmd/seal devThe first run copies
infra/stack/secrets/example.envtoinfra/stack/secrets/dev.envwith mode 600 and generates every key the stack needs, filling an emptyKEY=line in place or adding a line: the JWT and console keys and the Restate signing key (from the server image’s/envgen), the RustFS keys, the gateway’s master key and database password, and the virtual keys of the server and the console agent. The Restate private key goes todev.restate.pembeside it. Then it stops:seal: .../infra/stack/secrets/dev.env: set OPENROUTER_API_KEYOpen
dev.envand fill the values the script cannot make:Variable dev prod OPENROUTER_API_KEYrequired required FCM_CREDENTIALSoptional; empty logs push notifications instead of sending them required; APP_ENV=prodrefuses it empty, and so does the seal command for prodOPENROUTER_API_KEYis the stack’s OpenRouter key, whichai-gatewaysends for every alias on OpenRouter; give it a daily credit limit on OpenRouter.FCM_CREDENTIALSis a Firebase service account’s JSON, encoded in base64.Run the command again for dev, then twice for prod:
Terminalgo -C infra run ./cmd/seal dev go -C infra run ./cmd/seal prod # creates prod.env and stops go -C infra run ./cmd/seal prod # after filling prod.envA second run keeps the keys already in
<stack>.env, so it seals the same values. Each successful run writes four SealedSecrets toinfra/stack/secrets/<stack>/:server-secrets.yaml,restate-secrets.yaml,litellm-secrets.yamlandlitellm-db.yaml. Commit and push them:Terminalgit add infra/stack/secrets/dev infra/stack/secrets/prod git commit -m "infra: sealed secrets for dev and prod" git pushThe
.envand.pemfiles are git-ignored and stay on the machine.You should see
sealed into .../infra/stack/secrets/<stack>; commit and pushafter each successful run. -
Watch the sync waves
Argo CD picks up the push on its next poll of the repo. To have it look now:
Terminalkubectl -n argocd annotate application stack-dev argocd.argoproj.io/refresh=normal kubectl -n argocd annotate application stack-prod argocd.argoproj.io/refresh=normalEach stack then syncs in waves -1 to 2, and its
ai-<stack>in waves 0 and 1 after it, as in the diagram above. Argo CD waits for one wave to be healthy before it starts the next. Postgres, Restate and RustFS lists every resource of each wave.Terminalkubectl -n argocd get applications -w kubectl -n shopnexus-prod get pods -o wide kubectl -n restate-prod get pods -o wideYou should see
stack-dev,stack-prod,ai-devandai-prodSynced and Healthy. Dev runs one of everything. Prod runs three server pods onagent-1andagent-2, two Postgres pods on different nodes and three Restate pods on three different nodes. -
Run the smoke test
infra/smoke.hurlrequests every public host of one stack. Add--insecureif the certificates come from Let’s Encrypt staging.Terminalhurl --test --continue-on-error --variable domain=dev.example.com infra/smoke.hurl hurl --test --continue-on-error --variable domain=example.com infra/smoke.hurlIt checks
https://<domain>/,https://api.<domain>/healthzand/readyz,https://docs.<domain>/,https://console.<domain>/andhttps://media.<domain>/healthfor a 200, andhttp://api.<domain>/healthzfor a redirect to HTTPS (301 from Traefik, 308 from Caddy).--continue-on-errorruns all seven requests even after one fails, so the report lists every failing host.You should see the file reported as a success and an exit status of 0.
-
Open the console
The console at
console.<domain>needs a member account, and a new cluster has none. The API and the console create moderators only; the first admin comes from theconsolecommand in a server pod:Terminalkubectl -n shopnexus-prod exec -it $(kubectl -n shopnexus-prod get pod -l app.kubernetes.io/name=server -o name | head -1) -- \ /console create-admin --email you@example.com --name "Your Name"It prints
created admin you@example.com (<id>)and a temporary password. Openhttps://console.example.comand sign in with it. The console asks for a new password and then for a TOTP authenticator. Run the same command inshopnexus-devfor the dev console.You should see the console’s home page. Runbook has the other
consolecommands.
Tear down
k3d cluster delete shopnexusThis deletes every node, volume and database in the cluster, including the Sealed Secrets controller’s key. The SealedSecrets in git can only be opened on a new cluster if you restore that key from the backup before sealing again; Secrets has the procedure. Without the key, run the seal command against the new cluster, which reuses the keys in <stack>.env, and commit the new SealedSecrets.