Cluster overview
The k3d cluster in depth. Its nodes, the dev and prod stacks, every component with its replica count, the Argo CD app-of-apps and the layout of infra/.
One k3d cluster named shopnexus runs two stacks of ShopNexus, dev and prod. Each stack lives in its own namespace and has its own database, Restate cluster, object store and AI hub. Argo CD keeps both stacks in step with the umbrella repo, and the design is in docs/superpowers/specs/2026-11-04-infra-k3d-stacks-design.md.
Nodes
Both k3d configs, infra/cluster/k3d-direct.yaml and infra/cluster/k3d-behind-proxy.yaml, create one server node and three agents from the image rancher/k3s:v1.37.1-k3s1. Each agent stands in for a machine and carries a role label that the chart’s workloads prefer.
| Node | Label | Prefers |
|---|---|---|
server-0 | control plane | Traefik (both edge configs pin it here with a nodeSelector) |
agent-0 | role=stateful | RustFS |
agent-1 | role=app | server, website, docs, console, ai-gateway, ai-models |
agent-2 | role=app | the same as agent-1 |
flowchart TB
subgraph s0["server-0 (control plane)"]
traefik[Traefik]
end
subgraph a0["agent-0, role=stateful"]
rustfs[rustfs]
end
subgraph a1["agent-1, role=app"]
app1["server, website, docs,<br/>console, ai-gateway, ai-models"]
end
subgraph a2["agent-2, role=app"]
app-android["server, website, docs,<br/>console, ai-gateway, ai-models"]
end
spread["postgres and restate pods:<br/>one per node where possible"] -.-> a0
spread -.-> a1
spread -.-> a2
traefik --> app1
traefik --> app-android
traefik --> rustfsEvery placement rule is a preference. The node affinity on role and the pod anti-affinity on server, website, docs and console use preferredDuringSchedulingIgnoredDuringExecution. CloudNativePG uses podAntiAffinityType: preferred, and the RestateCluster spreads its pods with topologySpreadConstraints and whenUnsatisfiable: ScheduleAnyway. Pods therefore still schedule when an agent is stopped; the three prod Restate pods spread over the three agents while all three run.
k3d runs every node as a Docker container on one host, so the agents share one kernel and one disk. Stopping an agent with docker stop still behaves like losing a machine to Kubernetes, which is what the failover drills need. The project is a learning project, and k3d keeps the cost at one machine. No file outside infra/cluster/ names k3d, so a move to k3s on real hosts changes infra/cluster/ alone.
Stacks
| dev | prod | |
|---|---|---|
| Domain | dev.shopnexus.id.vn | shopnexus.id.vn |
| Namespace | shopnexus-dev | shopnexus-prod |
| Restate namespace | restate-dev | restate-prod |
| Argo CD Application | stack-dev | stack-prod |
| Values file | values-dev.yaml | values-prod.yaml |
APP_ENV | dev | prod |
| Images | :main, by digest | X.Y.Z of the umbrella’s release vX.Y.Z |
The RestateCluster of a stack is named restate-<stack>, and restate-operator puts it in a namespace of the same name. The chart creates that namespace in sync wave -1, so the sealed restate-secrets land in it before the cluster starts. NetworkPolicy denies all ingress in a stack namespace and opens only the paths the stack needs (Edge and network lists them). No policy opens a path between shopnexus-dev and shopnexus-prod.
Components
Each row is one resource that the chart in infra/stack renders into a stack namespace. The counts come from values.yaml (dev) and values-prod.yaml (prod).
| Component | Kind | Port | dev | prod |
|---|---|---|---|---|
server | RestateDeployment | 8080 HTTP, 9080 Restate | 1 | 3 |
website | Deployment | 3000 | 1 | 2 |
docs | Deployment | 80 | 1 | 1 |
console | Deployment | 80 | 1 | 2 |
postgres | CloudNativePG Cluster | 5432 | 1 instance, 5Gi | 2 instances, 10Gi |
restate-<stack> | RestateCluster | 8080 ingress, 9070 admin | 1 node, 2 GiB | 3 nodes, 4 GiB |
rustfs | StatefulSet | 9000 | 1, 10Gi | 1, 10Gi |
The Application ai-<stack> renders ai-gateway (port 4000) and ai-models (port 8000) into the same namespace from the ai repo’s chart, one replica each in both stacks.
Restate replicates each partition to min(2, replicas) nodes, which is 1 copy in dev and 2 in prod. Postgres runs the image paradedb/paradedb:0.26.4-pg18; ai-gateway keeps its own database litellm on the same CloudNativePG cluster. The server-api Service on port 8080 selects every server pod of any version, and restate-operator creates a Service per server version for port 9080.
Three Jobs run as Argo CD Sync hooks:
| Job | Wave | Does |
|---|---|---|
migrate | 1 | applies the server’s migrations (cmd/migrate) |
objectstore-init | 1 | creates the bucket shopnexus |
objectstore-init-snapshots | 1 | creates the bucket restate-snapshots |
Argo CD applies a stack in sync waves -1 to 2. The table of what each wave holds, and the stateful workloads themselves, are on Postgres, Restate and RustFS.
The app-of-apps
k3s installs Argo CD (chart argo-cd 10.11.7, namespace argocd) from infra/cluster/argocd.yaml and applies one Application, root, at its first start; both files come in through the k3d config’s files: list. From then on Argo CD manages everything else, root included. When the server node restarts, k3s applies the copy of root.yaml it got at cluster creation again, and Argo CD’s self-heal puts the git version back within minutes.
flowchart LR root["root<br/>infra/platform + infra/argocd"] root -->|wave 0| cnpg["cnpg<br/>cloudnative-pg 0.29.1<br/>ns cnpg-system"] root -->|wave 0| ss["sealed-secrets<br/>sealed-secrets 2.22.0<br/>ns kube-system"] root -->|wave 0| iuapp["image-updater<br/>argocd-image-updater 1.4.0<br/>ns argocd"] root -->|wave 0| ro["restate-operator<br/>restate-operator-helm 3.2.0<br/>ns restate-operator"] root -->|wave 1| dev["stack-dev<br/>infra/stack + values-dev.yaml"] root -->|wave 1| prod["stack-prod<br/>infra/stack + values-prod.yaml<br/>at vX.Y.Z"] root -->|wave 1| iu["ImageUpdater shopnexus"] root -->|wave 2| aidev["ai-dev<br/>repo ai, chart/"] root -->|wave 2| aiprod["ai-prod<br/>repo ai, chart/"] iu -. "digest of :main" .-> dev iu -. "digest of :main" .-> aidev dev --> devns[shopnexus-dev] prod --> prodns[shopnexus-prod] aidev --> devns aiprod --> prodns
root has two sources in the umbrella repo: infra/platform, with one Application per platform chart, and infra/argocd, with root.yaml itself, stack-dev, stack-prod, the ImageUpdater, and ai-dev and ai-prod in ai.yaml. The platform charts come from their upstream Helm repositories and are not vendored. restate-operator’s chart is read from GHCR as an OCI repository through the Secret repo-restatedev-oci. ai-dev and ai-prod render the chart in chart/ of the ai repo (shopnexus/ai) into the stack namespaces, ai-dev at main and ai-prod at the ai commit of the release, after the stacks; AI hub describes them.
Argo CD stopped judging the health of an Application in version 1.8. The values in infra/cluster/argocd.yaml add a health check for argoproj.io/Application, so root waits for the platform Applications in wave 0 to be Healthy before it creates the stacks in wave 1. Every Application syncs automatically with prune and selfHeal and retries without limit. root ignores /spec/source/helm/parameters on the Applications, because Image Updater writes the image tags of stack-dev and ai-dev there and self-heal would otherwise put them back.
Repository layout
infra/
├── check.sh # lints the chart, runs its tests, checks Argo CD and the edge; CI runs it
├── smoke.hurl # hurl --test --variable domain=<domain>: requests every host of one stack
├── go.mod # Go module for cmd/
├── cmd/
│ └── seal/ # go -C infra run ./cmd/seal dev|prod [-dry-run]: seals a stack's secrets
│ ├── main.go
│ └── main_test.go # go -C infra test ./...; needs Docker
├── cluster/ # used once per machine; the only k3d-specific part
│ ├── k3d-direct.yaml # k3d config: Traefik on 80/443 with ACME, Argo CD, root
│ ├── k3d-behind-proxy.yaml # k3d config: Traefik behind a proxy on 127.0.0.1:18080, Argo CD, root
│ └── argocd.yaml # k3s HelmChart: Argo CD, with a health check for Applications
├── argocd/
│ ├── root.yaml # the app-of-apps
│ ├── stacks.yaml # stack-dev, stack-prod and the ImageUpdater
│ └── ai.yaml # ai-dev, ai-prod: the ai repo's chart, keys' budgets per stack
├── platform/ # one Application per upstream chart
│ ├── cnpg.yaml
│ ├── image-updater.yaml
│ ├── restate-operator.yaml
│ └── sealed-secrets.yaml
└── stack/ # Helm chart "shopnexus": one stack
├── Chart.yaml
├── values.yaml # defaults, which are the dev values
├── values-dev.yaml # stack, domain, appEnv, which vectors are on
├── values-prod.yaml # the same, plus replicas, storage and 0.1.0 tags
├── secrets/
│ ├── example.env # template that cmd/seal copies to <stack>.env
│ ├── dev/ # SealedSecrets of stack-dev, committed
│ └── prod/ # SealedSecrets of stack-prod, committed
├── templates/ # server, web, postgres, restate, rustfs,
│ # jobs, ingress, networkpolicy, secrets, validate
└── tests/ # helm-unittest suites, and probe SealedSecrets they render withThe seal command keeps the generated keys in infra/stack/secrets/<stack>.env and <stack>.restate.pem, which git ignores and which belong in a backup with the controller’s key (Secrets).
Design principles
Key rules
- One Helm chart,
infra/stack, renders both stacks.values-dev.yamlandvalues-prod.yamlhold every difference between them. - Every host derives from one
domainvalue:<domain>,api.<domain>,media.<domain>,docs.<domain>andconsole.<domain>. - Nothing specific to an environment lives outside
infra/. The one exception is the host proxy fileconf.d/shopnexus.caddyinlinux-server-setup, used inbehind-proxymode. - Only
infra/cluster/knows about k3d. - CI builds images and never connects to the cluster; Argo CD pulls every change from git.
A fork, following Bring up a cluster, changes repoURL in infra/argocd/root.yaml and infra/argocd/stacks.yaml, and domain in the two values files. Images under another owner than ghcr.io/shopnexus also need new names in values.yaml and stacks.yaml. The website image sets API_URL=http://server-api:8080/v1, an in-cluster Service name that resolves in whichever stack namespace the website runs in, so one image serves both stacks.
Later work
The spec leaves backups, observability, real machines and the root docker-compose.yml out of scope.
- Nothing backs up Postgres, RustFS or Restate. The Sealed Secrets controller’s key and the files the seal command writes are saved by hand.
- The chart sets
OTEL_EXPORTER_OTLP_ENDPOINTto an empty string, so the server sends nothing over OTLP and writes its JSON logs to stdout only. The cluster runs no collector. - The agents are containers on one host and share its disk, so a failed disk takes every node with it. A move to k3s on separate hosts changes
infra/cluster/. - The root
docker-compose.ymlnow has three stacks, one per profile, each with its own network, Postgres, Restate, RustFS and volumes.devruns the server withgo runand the web apps’ dev servers with Compose Watch, behind Caddy on port 56481.prodbuilds every component from its Dockerfile, behind Caddy on port 56480.testruns the server’s Go tests.infra/check.shfails when the Postgres, Restate and RustFS image tags there differ frominfra/stack/values.yaml. - The spec estimated the two stacks at 6 to 8 GB of memory before photo search. Photo search adds
ai-models: 6Gi requested in prod, with SigLIP 2 (4.0 GiB held idle, measured 2026-11-10), and 2Gi in dev, which loads no SigLIP 2 since 2026-11-17.ai-gatewayholds about 0.6 GiB per stack. Text embeddings come from OpenRouter, so no stack runs TEI. On 2026-11-17 TEI (5.5 GiB) andai(4.1 GiB) filled the 19 GiB dev machine together with its other projects. The fallbacks are to scale dev to zero when unused or to drop the cluster to two agents.