AI hub
The ai repo as the AI hub of each stack. ai-gateway (LiteLLM) with its aliases, keys and daily budgets, ai-models for speech to text, photo embeddings and video frames, OpenRouter upstream, and how to change a model, a budget or an alias.
Search asks a language model what a query means, a listing read turns a seller’s video or photo into a draft, and photo search compares embeddings of text and photos. In the cluster no client calls a model provider itself. Each stack runs the AI hub from the ai repo (shopnexus/ai): ai-gateway, a LiteLLM proxy with an OpenAI-compatible API, and ai-models, the Go service with the models ShopNexus hosts itself. The design is in docs/superpowers/specs/2026-11-17-ai-hub-design.md.
The clients are the server and the console agent. Each names a purpose, an alias such as search, and holds a virtual key per purpose with a daily budget. Each stack has its own gateway, keys and budgets, so a loop in dev cannot use up the prod allowance.
| Alias | Upstream | Model | Used for |
|---|---|---|---|
search | OpenRouter | anthropic/claude-haiku-5.5 | reading a search query |
listing-read | OpenRouter | anthropic/claude-haiku-5.5 | reading a seller’s video frames and words, one photo, or a Chợ Tốt listing into a draft |
scam-check | OpenRouter | anthropic/claude-haiku-5.5 | checking a chat window for scams |
console-agent | OpenRouter | anthropic/claude-sonnet-5.5 | the console agent |
embed-text | OpenRouter, DeepInfra only | baai/bge-m3, 1024 dimensions | text vectors of listings and queries |
embed-image | ai-models | siglip2-so400m-patch16-384, 1152 dimensions | photo vectors, and short English texts for SigLIP’s text tower |
stt | ai-models | gipformer-1.5 | speech to text in a video read |
Each alias is an entry of gateway.aliases in the ai repo’s chart/values.yaml. Changing the model behind one needs no new server image.
Deployment
The ai repo holds a Helm chart in chart/. The umbrella deploys it once per stack with two Argo CD Applications in infra/argocd/ai.yaml, ai-dev into shopnexus-dev and ai-prod into shopnexus-prod. Each has one source, the ai repo’s chart/, ai-dev at main and ai-prod at the ai commit of the umbrella’s release, and sets the stack’s own values inline in spec.source.helm.valuesObject. root creates them in wave 2, after the stack Applications in wave 1, whose Postgres holds the gateway’s database and whose sealed secrets hold the keys.
flowchart TB
app["Application ai-dev<br/>repo shopnexus/ai, chart/, main<br/>+ valuesObject in infra/argocd/ai.yaml"]
app -->|wave 0| gw["Deployment and Service ai-gateway<br/>ghcr.io/berriai/litellm-database:v1.108.0<br/>ConfigMap ai-gateway-config"]
app -->|wave 0| models["Deployment and Service ai-models<br/>ghcr.io/shopnexus/ai"]
app -->|"wave 1, Sync hook"| job["Job ai-gateway-keys<br/>curlimages/curl"]
gw --> db[("postgres-rw<br/>database litellm")]| Values | Where | Holds |
|---|---|---|
| the hub’s choices | chart/values.yaml in the ai repo | the aliases and their models, the images, the keys and the aliases each may call, the settings of ai-models |
| the stack’s | valuesObject of ai-<stack> in infra/argocd/ai.yaml | stack, every key’s dailyUnits and tpmLimit or rpmLimit, models.embedModels, models.memoryRequest and models.memoryLimit |
The chart refuses to render without the stack’s values. Image Updater moves the ai-models image on ai-dev by the digest of :main. On ai-prod the umbrella’s Release workflow writes the ai commit and images.models.tag (Releases). A change to the chart or to chart/values.yaml reaches dev as soon as it is on the ai repo’s main, and prod with the next release.
Request path
flowchart LR
server["server pod<br/>LLM_MODEL=search"] -->|"POST /v1/chat/completions<br/>Bearer LLM_API_KEY"| gw["ai-gateway:4000<br/>alias search"]
gw -->|"checks key, budget, tpm"| db[("Postgres<br/>database litellm")]
gw -->|"openai/anthropic/claude-haiku-5.5<br/>Bearer OPENROUTER_API_KEY"| or["OpenRouter<br/>openrouter.ai/api/v1"]flowchart LR server["server pod<br/>module assist"] -->|"POST /v1/video/frames<br/>no key"| models["ai-models:8000<br/>ffmpeg, frames, Gipformer"] server -->|"POST /v1/audio/transcriptions, model stt<br/>Bearer STT_API_KEY"| gw["ai-gateway:4000<br/>aliases stt, listing-read"] server -->|"POST /v1/chat/completions, model listing-read<br/>Bearer ASSIST_LLM_API_KEY"| gw gw -->|"openai/gipformer-1.5<br/>no key"| models gw -->|"openai/anthropic/claude-haiku-5.5<br/>Bearer OPENROUTER_API_KEY"| or["OpenRouter"]
flowchart LR
server["server pod<br/>EMBED_TEXT_MODEL=embed-text<br/>EMBED_IMAGE_MODEL=embed-image"] -->|"POST /v1/embeddings<br/>Bearer EMBED_INDEX_API_KEY or EMBED_SEARCH_API_KEY"| gw["ai-gateway:4000<br/>aliases embed-text, embed-image"]
gw -->|"checks key, budget, rpm"| db[("Postgres<br/>database litellm")]
gw -->|"openai/baai/bge-m3, provider DeepInfra<br/>Bearer OPENROUTER_API_KEY"| or["OpenRouter"]
gw -->|"openai/siglip2-so400m-patch16-384<br/>no key"| models["ai-models:8000<br/>SigLIP 2"]The server’s ConfigMap server-env (infra/stack/templates/_server-env.tpl) points every model setting at the hub:
LLM_BASE_URL: http://ai-gateway:4000/v1
LLM_MODEL: search
ASSIST_LLM_BASE_URL: http://ai-gateway:4000/v1
ASSIST_LLM_MODEL: listing-read
TRUST_LLM_BASE_URL: http://ai-gateway:4000/v1
TRUST_LLM_MODEL: scam-check
STT_BASE_URL: http://ai-gateway:4000/v1
STT_MODEL: stt
AI_FRAMES_BASE_URL: http://ai-models:8000/v1
EMBED_TEXT_BASE_URL: http://ai-gateway:4000/v1
EMBED_IMAGE_BASE_URL: http://ai-gateway:4000/v1
EMBED_TEXT_MODEL: embed-text
EMBED_IMAGE_MODEL: embed-image
EMBED_TEXT_VERSION: bge-m3@5617a9f61b028005a4858fdac845db406aefb181
EMBED_IMAGE_VERSION: siglip2-so400m-patch16-384@29f74e673c298fd07482d8e20138179282001eb7-fp32The console agent gets LITELLM_BASE_URL=http://ai-gateway:4000/v1 and AGENT_MODEL_ID=console-agent. The server calls ai-models’ /v1/video/frames itself, because LiteLLM routes only OpenAI’s API. The Service names ai-gateway and ai-models resolve inside the stack’s own namespace, so the dev server reaches dev’s hub and the prod server reaches prod’s.
Aliases and the gateway config
The aliases in chart/values.yaml, search and embed-text shown:
gateway:
secret: litellm-secrets
database: { host: postgres-rw, name: litellm, user: litellm, passwordSecret: litellm-db }
aliases:
search:
model: openai/anthropic/claude-haiku-5.5
apiBase: https://openrouter.ai/api/v1
apiKeySecret: OPENROUTER_API_KEY
price: token
embed-text:
model: openai/baai/bge-m3
apiBase: https://openrouter.ai/api/v1
apiKeySecret: OPENROUTER_API_KEY
mode: embedding
price: token
extraBody: { provider: { order: [DeepInfra], allow_fallbacks: false } }The chart renders one model_list entry per alias into the ConfigMap ai-gateway-config. The search entry and the shared settings come out as:
model_list:
- model_name: search
litellm_params:
model: openai/anthropic/claude-haiku-5.5
api_base: https://openrouter.ai/api/v1
api_key: os.environ/OPENROUTER_API_KEY
model_info:
input_cost_per_token: 0.000001
output_cost_per_token: 0.000001
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
store_prompts_in_spend_logs: false
maximum_spend_logs_retention_period: 30d
litellm_settings:
drop_params: trueThe openai/ prefix tells LiteLLM to speak the OpenAI API to api_base. An alias with apiKeySecret sends that key of litellm-secrets; one without it sends api_key: none, for ai-models, which has no authentication. extraBody goes to the upstream as extra_body. drop_params: true makes LiteLLM drop request fields the upstream does not accept. The stack’s NetworkPolicies leave egress open, so ai-gateway reaches OpenRouter.
store_prompts_in_spend_logs: false keeps request bodies, the photos of photo search among them, out of LiteLLM’s spend logs, and the spend logs are kept 30 days. infra/probes/litellm-embed/run.sh measured this on LiteLLM v1.108.0 with a stub upstream on 2026-11-10: data URLs of 706 kB and 1.72 MB reached the upstream unchanged, three calls cost 0.001728 (576 tokens each, priced from the upstream’s usage), and none of the three spend log rows held an input.
OpenRouter
Every alias but embed-image and stt goes to OpenRouter with the stack’s OPENROUTER_API_KEY, one key per stack. Give that key a daily credit limit on OpenRouter; it is the last stop behind the gateway’s own budgets. Measured through LiteLLM v1.108.0 on 2026-11-17:
- LiteLLM passes OpenRouter’s
usage(tokens andcost) through for chat and embeddings, so budgets count the tokens OpenRouter reports. - An upstream 402, OpenRouter’s answer when the key’s credit is spent, reaches the client as HTTP 402 with code
"402". - OpenRouter serves
baai/bge-m3from two providers. DeepInfra (fp32) gave the vectors of the TEI the stacks ran before at a cosine of 0.999999 on every reference text under 2048 tokens (ai/embed/testdata/bge-m3-reference.json); Parasail gave 0.996.embed-textpins DeepInfra withallow_fallbacks: false, so stored vectors and new ones stay in one space. LiteLLM forwards the pin: a Parasail pin through it gave Parasail’s vectors. - 16 inputs in one
embed-textrequest took 2.3 s.
Budget model
LiteLLM budgets count money, and each alias’s price sets what that money stands for:
price | Cost | 1 unit of budget is |
|---|---|---|
token | 0.000001 per token, input and output | 1,000,000 tokens |
second | 0.000001 per second of audio | 1,000,000 seconds |
usd | the model’s list price, inputCostPerToken and outputCostPerToken | 1 USD |
console-agent is priced at 2 and 10 USD per 1M tokens in and out. A key’s dailyUnits is in millionths of its aliases’ unit, and the chart divides it by 1,000,000 for max_budget. The chart refuses a key over aliases priced in different units.
| Key | Secret | Aliases | dev dailyUnits | dev limit | prod dailyUnits | prod limit |
|---|---|---|---|---|---|---|
server-llm | server-secrets LLM_API_KEY | search | 200,000 tokens | 20,000 tpm | 2,000,000 tokens | 100,000 tpm |
server-stt | server-secrets STT_API_KEY | stt | 3,600 seconds | 10 rpm | 36,000 seconds | 60 rpm |
server-assist | server-secrets ASSIST_LLM_API_KEY | listing-read | 1,200,000 tokens | 60,000 tpm | 12,000,000 tokens | 300,000 tpm |
server-trust | server-secrets TRUST_LLM_API_KEY | scam-check | 600,000 tokens | 30,000 tpm | 6,000,000 tokens | 150,000 tpm |
server-embed-index | server-secrets EMBED_INDEX_API_KEY | embed-text, embed-image | 10,000,000 tokens | 120 rpm | 50,000,000 tokens | 600 rpm |
server-embed-search | server-secrets EMBED_SEARCH_API_KEY | embed-text, embed-image | 1,000,000 tokens | 60 rpm | 10,000,000 tokens | 600 rpm |
agent | agent-secrets AGENT_LLM_API_KEY | console-agent | 20 USD | 100,000 tpm | 20 USD | 200,000 tpm |
chart/values.yaml sets the agent’s 20 USD a day for every stack; infra/argocd/ai.yaml sets the rest. LiteLLM resets each key’s spend every day, with budget_duration: 1d.
The server indexes listings with server-embed-index and embeds queries with server-embed-search. A backfill of every listing spends the index key, and photo search keeps its own budget. ai-models counts 576 tokens a photo, its 24 × 24 patches, and a text its token count; LiteLLM prices from that usage. A listing of ten photos costs about 5,800 tokens, so dev’s index budget covers about 1,700 listings a day.
A video read sends listing-read 8 frames at 768 px, the list of subcategories and the transcript, about 5,500 tokens, so dev’s budget covers about 200 reads a day and prod’s about 2,000. A photo read sends one photo, about 1,500 tokens; each account may start 30 video reads, 60 photo reads and 30 link reads a day. A scam check sends the prompt and 20 short messages, about 1,300 tokens, so dev covers about 460 checks a day and prod about 4,600.
A test on a local k3d cluster set a 60-token budget on the search key: once it was spent, calls failed with HTTP 422 and Budget has been exceeded, and a call for an alias outside the key’s models got a 403. A test against LiteLLM v1.108.0 sent 5.55 s of audio, and the spend of server-stt rose by 0.00000555. LiteLLM asks ai-models for verbose_json when the caller names no response_format, and prices every speech call from the duration field that json and verbose_json answers carry. A text answer has no duration and costs 0, so the server must not ask for text.
The key Job
The keys are created by the Job ai-gateway-keys, an Argo CD Sync hook in wave 1 of ai-<stack>, after ai-gateway (wave 0) is ready. chart/templates/gateway-keys.yaml renders it from gateway.keys. Its pod runs curlimages/curl and does this for each key:
flowchart TD start["Sync hook ai-gateway-keys"] --> update["POST /key/update<br/>key = the key's value from its Secret"] update -->|200| ok["log: key server-llm: HTTP 200"] update -->|not 200| generate["POST /key/generate<br/>same key value"] generate -->|200| ok generate -->|not 200| fail["log status and message<br/>exit 1, retry up to 10 times"]
Both calls send the same body: the key’s value from the Secret its secret names, key_alias (the key’s name), models, max_budget, budget_duration: 1d, and tpm_limit or rpm_limit. They authenticate with the master key from litellm-secrets. A successful sync logs one key <name>: HTTP 200 line per key, seven in all.
The key values are fixed in advance: the seal command generates them and seals them into server-secrets and agent-secrets, and the Job writes the same values into LiteLLM. Running the Job twice gives the same result. After the gateway’s database is lost, the update fails, the generate call recreates the key with the value the client already holds, and the client does not restart. A key renamed in gateway.keys keeps its value, so the next sync’s /key/update finds it by its value and changes its alias; on v1.108.0 on 2026-11-10 a renamed key kept its spend.
The response of LiteLLM contains the key, so the Job logs only the HTTP status. On a failure it also prints the message field of the answer.
kubectl -n shopnexus-dev logs job/ai-gateway-keysargocd.argoproj.io/hook-delete-policy: BeforeHookCreation keeps the last run’s Job until the next sync replaces it.
The gateway’s database
LiteLLM keeps keys and spend in Postgres, in a database of its own on the stack’s CNPG cluster postgres. The stack chart makes it, since Postgres belongs to the stack:
postgres.yamlininfra/stackdeclares a managed rolelitellmwith its password from the Secretlitellm-db, and a CNPGDatabasenamedlitellmowned by that role (stack wave 1).ai-gatewaybuildsDATABASE_URL=postgresql://litellm:$(DB_PASSWORD)@postgres-rw:5432/litellmfromgateway.database.- The image is
ghcr.io/berriai/litellm-database:v1.108.0, the LiteLLM build that uses a database.
The server’s data lives in the database app of the same cluster. The litellm role owns only litellm.
Secrets
go -C infra run ./cmd/seal <stack> generates every value below but OPENROUTER_API_KEY into the git-ignored infra/stack/secrets/<stack>.env. You write OPENROUTER_API_KEY yourself, and the seal command stops with set OPENROUTER_API_KEY until you do. The Secret names litellm-secrets and litellm-db are the same as before the hub.
Value in <stack>.env | Sealed into | Read by |
|---|---|---|
LITELLM_MASTER_KEY | litellm-secrets | ai-gateway, the ai-gateway-keys Job, you in the admin UI |
OPENROUTER_API_KEY | litellm-secrets | ai-gateway, toward OpenRouter |
LITELLM_DB_PASSWORD | litellm-db (basic-auth), key password | CNPG for the role, ai-gateway for DATABASE_URL |
LLM_API_KEY, STT_API_KEY, ASSIST_LLM_API_KEY, TRUST_LLM_API_KEY, EMBED_INDEX_API_KEY, EMBED_SEARCH_API_KEY | server-secrets | the server; the ai-gateway-keys Job |
AGENT_LLM_API_KEY | agent-secrets | the console agent; the ai-gateway-keys Job |
ai-gateway is in another Application than the stack’s secrets and does not carry their checksum. After a reseal of litellm-secrets or litellm-db, restart it by hand:
kubectl -n shopnexus-dev rollout restart deploy/ai-gatewayA key the chart names but its Secret lacks keeps the ai-gateway-keys pod from starting, and the sync of ai-<stack> stops at wave 1: reseal and push the secrets before the ai commit that adds the key reaches main. Secrets covers how the seal command and the controller seal them.
When the budget runs out
The gateway refuses a spent key with HTTP 422 and Budget has been exceeded. When the stack’s OpenRouter credit is spent, it passes OpenRouter’s 402 through. The server’s LLM client treats a 402 like a 429, as a passing failure, and its embed client retries it. The console agent’s isBudgetError takes a 402 as a spent budget, as it takes LiteLLM’s own budget_exceeded.
The server’s search does not fail when the model does. Search in server/internal/discovery/service/search.go gives the model 4 seconds. On any error, a refusal for budget or rate limit included, it logs the model did not read the query at warn level with the error, and searches with the raw words alone.
sequenceDiagram participant App as app participant S as server participant G as ai-gateway App->>S: GET /v1/search?q=... S->>S: cache lookup (7 days) S->>G: chat/completions, model search G-->>S: 422 Budget has been exceeded, or 402 S->>S: log "the model did not read the query" S->>S: search the raw words as a phrase S-->>App: 200, listings, no chips
The person still gets a page of listings, matched on the words they typed. The page has no chips, and no boosts for category, price, place or freshness that the model would have read out of the query. A query the model read in the last 7 days comes from the server’s cache and keeps its understanding. The server’s error for a non-200 answer is model answered <status>, so a spent budget logs model answered 422 and spent credit model answered 402.
Calls over the tokens-per-minute limit fail too, and search falls back to the raw words until the minute has passed. Search and discovery lists the boosts the model can set.
ai-models
ai-models is a Deployment and a Service on port 8000 with no Ingress and no authentication; from-stack admits the stack’s pods. The gateway’s aliases stt and embed-image call it, and the server calls its frames route directly. Its settings are models.* in chart/values.yaml.
Speech to text
ai-models serves OpenAI’s POST /v1/audio/transcriptions with Gipformer 1.5, a Vietnamese model with 68.6M parameters, on CPU. It takes any audio or video file ffmpeg reads, up to 100 MB and 300 s of audio (models.maxUploadMB, models.maxAudioSeconds). Silero VAD cuts the speech at silences into pieces of at most 20 s, and the answer is lower-case text without punctuation. Two requests decode at once (models.maxConcurrent), and the others wait.
The server calls stt when it reads a seller’s video (module assist). It asks for response_format: json, so LiteLLM prices the call from duration.
To send stt to another provider, set gateway.aliases.stt.model to that provider’s LiteLLM model string, for example groq/whisper-large-v3, and gateway.aliases.stt.apiBase to its base URL; add the provider’s key to litellm-secrets and name it in gateway.aliases.stt.apiKeySecret. The server keeps asking for stt.
ai-models logs one line per request, without the transcript:
kubectl -n shopnexus-dev logs deploy/ai-modelsVideo frames
POST /v1/video/frames is not an OpenAI route, so the server calls it at AI_FRAMES_BASE_URL with no key, and the gateway counts nothing for it. It takes a video of 3 to 35 s (models.framesMaxSeconds), decodes 2 frames a second at most 1600 px on the long side, drops frames whose variance of the Laplacian is under models.framesMinSharpness and frames that repeat another, and returns at most 8 JPEGs spread over the video. Two requests cut frames at once (models.framesMaxConcurrent), apart from the two that decode speech. On 2026-11-10, two frame requests of a 30 s 1080p video and two transcriptions at once raised the pod to 2.42 GiB.
Photo embeddings
ai-models serves POST /v1/embeddings with SigLIP 2 (siglip2-so400m-patch16-384, fp32 ONNX, 1152 dimensions) when models.embedModels names it. An input that starts with data:image/ is a photo; any other input is text for SigLIP’s text tower. The weights are 4.5 GB. With models.embedModels empty, ai-models loads no SigLIP 2, and /v1/embeddings answers 404 model_not_found.
| Stack | models.embedModels | Memory request, limit | Measured |
|---|---|---|---|
| dev | empty | 2Gi, 3Gi | speech and frames alone peaked at 2.42 GiB on 2026-11-10 |
| prod | siglip2-so400m-patch16-384=/models/siglip2-so400m-patch16-384 | 6Gi, 8Gi | 4.0 GiB idle and 5.3 GiB on 16 photos, 2026-11-10 |
Dev loads no SigLIP 2 because the dev machine has no room for it (2026-11-17). Its stack values set server.embed.image: false, so the server writes and queries no photo vectors there. infra/check.sh fails when a stack’s photo vectors are on without SigLIP 2 in its ai-models, or off with it.
Vectors and their versions
The server stores EMBED_TEXT_VERSION and EMBED_IMAGE_VERSION with every vector. Both are in infra/stack/env/server.env, which the cluster’s chart and the root docker-compose.yml both read. A new model needs a new version in the same change: the server then treats the stored vectors as missing, and go run ./cmd/search-reindex -every 2s in the server embeds every listing again, one listing every 2 s.
EMBED_TEXT_VERSION stayed bge-m3@5617a9f61b028005a4858fdac845db406aefb181 when embed-text moved from TEI to OpenRouter, since DeepInfra’s vectors match. A text over 2048 tokens gave a cosine of 0.90, because TEI cut inputs at 2048 tokens and OpenRouter does not. The server cuts listing text at 2,000 characters (EmbeddingTextMax), under 2048 tokens.
Each kind of vector can be off. server.embed.text and server.embed.image in a stack’s values turn it on or off, and an off kind gets empty EMBED_*_MODEL, EMBED_*_BASE_URL and EMBED_*_VERSION. The server then writes no vectors of that kind and leaves its list out of search; BM25 always runs. A photo search with photo vectors off answers 503 search_unavailable.
Local stacks
The root docker-compose.yml runs ai-models-dev on 127.0.0.1:56491 and ai-models-prod on 127.0.0.1:56490, with no gateway and no TEI. The server calls ai-models-<stack> directly for photo vectors, speech and frames, and names the model siglip2-so400m-patch16-384; ai-models takes no key and ignores the one sent. For text vectors it reads EMBED_TEXT_BASE_URL, EMBED_TEXT_MODEL, EMBED_INDEX_API_KEY and EMBED_SEARCH_API_KEY from server/.env: OpenRouter’s baai/bge-m3 with your OpenRouter key, or an empty EMBED_TEXT_MODEL, which turns text vectors off. infra/check.sh fails when the AI_* settings of the compose file differ from models.* in the ai chart.
Smoke test
infra/smoke-models.sh <stack> port-forwards svc/ai-gateway and calls every alias with the server’s keys: search, listing-read with a photo, scam-check, stt with a silent clip, embed-text (1024 dimensions), and embed-image with a photo and with a text (1152 dimensions). It skips the embed-image calls when the stack’s EMBED_IMAGE_MODEL is empty. Then it checks that server-llm is refused embed-text, and server-assist and server-trust are refused search, each with a 403. It needs hurl and kubectl.
Change the model or the provider
Clients always ask for the same aliases. To change what answers search:
-
Edit the values
In the
airepo’schart/values.yaml, setgateway.aliases.search.modelto LiteLLM’s model string, provider prefix included (openai/anthropic/claude-haiku-5.5), andgateway.aliases.search.apiBaseto its OpenAI-compatible base URL. To change one stack only, set them undergateway.aliases.searchin thevaluesObjectofai-<stack>ininfra/argocd/ai.yamlinstead. The chart refuses to render if either is empty. -
Add the provider's key
For a provider other than OpenRouter, add its key to
infra/stack/secrets/<stack>.envand tolitellm-secretsin the seal command, name it inapiKeySecret, reseal both stacks and push, and restartai-gateway. -
Run the tests and push
Run
helm unittest chartin theairepo, commit and push to itsmain. Argo CD syncsai-dev;ai-prodgets the change with the next release. Thechecksum/configannotation onai-gateway’s pod changes with the config, so the gateway restarts on the new model. -
Check a search
Search for something on the stack and read the server log. A line
the model did not read the querymeans the new model failed.
The server asks for a strict json_schema response format. A model that ignores it returns text the server cannot decode, and search falls back to the raw words on every query. To change embed-text or embed-image, see Vectors and their versions: the vectors’ version changes with the model.
Change a budget
-
Edit the stack's values
In
infra/argocd/ai.yaml, under thevaluesObjectofai-devorai-prod, change a key’sdailyUnitsor itstpmLimitorrpmLimitundergateway.keys. The agent’s daily budget is in theairepo’schart/values.yaml. -
Check and push
Run
infra/check.sh, which renders theaichart with each stack’s values, then commit and push the umbrella.rootupdates the Application,ai-<stack>syncs, and theai-gateway-keysJob runs/key/updateon every key with the new limits. -
Check the Job
kubectl -n shopnexus-dev logs job/ai-gateway-keysprints onekey <name>: HTTP 200line per key.
Add another model alias
An alias is an entry of gateway.aliases, and a key reaches only the aliases in its models:
- Add the entry to
gateway.aliasesin theairepo’schart/values.yaml, withmodel,apiBase,price(token,secondorusd), andapiKeySecret,mode,extraBody,inputCostPerTokenandoutputCostPerTokenwhen it needs them. - Add its name to the
modelsof the key that calls it. For a new caller, add an entry togateway.keyswith itssecret(nameandkeyof the client’s Secret) andmodels, set itsdailyUnitsand atpmLimitorrpmLimitfor each stack ininfra/argocd/ai.yaml, and add the key to the seal command’s generated keys and to the client’s Secret. - Run
helm unittest chartin theairepo andinfra/check.shin the umbrella before you push. The chart refuses a key that names a missing alias, a key over aliases priced in different units, a key with neither limit, and an alias without a model.
A new alias is used only by a caller that asks for it.
Admin UI and spend
The stack’s dashboard host litellm.<domain>, behind the console’s sign-in, routes to the Service ai-gateway. From a shell, use a port-forward and the master key:
kubectl -n shopnexus-dev get secret litellm-secrets -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d; echo
kubectl -n shopnexus-dev port-forward svc/ai-gateway 4000:4000Open http://localhost:4000/ui and sign in with user admin and the master key as the password. The UI lists the keys with aliases server-llm, server-stt, server-assist, server-trust, server-embed-index, server-embed-search and agent, each with its spend, budget and reset time. Use shopnexus-prod for prod.
To read the spend from a shell while the port-forward runs:
master=$(kubectl -n shopnexus-dev get secret litellm-secrets -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d)
key=$(kubectl -n shopnexus-dev get secret server-secrets -o jsonpath='{.data.LLM_API_KEY}' | base64 -d)
curl -s "http://localhost:4000/key/info?key=$key" -H "Authorization: Bearer $master"info.spend is in budget units: a spend of 0.05 on server-llm is 50,000 tokens today, and on agent 0.05 USD. info.max_budget is the day’s limit and info.budget_reset_at the next reset.
Key rules
- Clients call only aliases of
gateway.aliases; the model behind each is set in theairepo’schart/values.yaml. - Budgets are set per stack in
infra/argocd/ai.yaml, in tokens, seconds or millionths of a dollar. - Indexing and search embed with separate keys,
server-embed-indexandserver-embed-search. - The key values come from the seal command; the
ai-gateway-keysJob only writes them into LiteLLM. - After a reseal of
litellm-secrets,ai-gatewayrestarts only by hand. embed-textstays pinned to DeepInfra whileEMBED_TEXT_VERSIONnames TEI’s bge-m3.