Skip to content
ShopNexus Docs
On this page Deployment 1/20

Search asks a language model what a query means, a listing read turns a seller’s video or photo into a draft, and photo search compares embeddings of text and photos. In the cluster no client calls a model provider itself. Each stack runs the AI hub from the ai repo (shopnexus/ai): ai-gateway, a LiteLLM proxy with an OpenAI-compatible API, and ai-models, the Go service with the models ShopNexus hosts itself. The design is in docs/superpowers/specs/2026-11-17-ai-hub-design.md.

The clients are the server and the console agent. Each names a purpose, an alias such as search, and holds a virtual key per purpose with a daily budget. Each stack has its own gateway, keys and budgets, so a loop in dev cannot use up the prod allowance.

AliasUpstreamModelUsed for
searchOpenRouteranthropic/claude-haiku-5.5reading a search query
listing-readOpenRouteranthropic/claude-haiku-5.5reading a seller’s video frames and words, one photo, or a Chợ Tốt listing into a draft
scam-checkOpenRouteranthropic/claude-haiku-5.5checking a chat window for scams
console-agentOpenRouteranthropic/claude-sonnet-5.5the console agent
embed-textOpenRouter, DeepInfra onlybaai/bge-m3, 1024 dimensionstext vectors of listings and queries
embed-imageai-modelssiglip2-so400m-patch16-384, 1152 dimensionsphoto vectors, and short English texts for SigLIP’s text tower
sttai-modelsgipformer-1.5speech to text in a video read

Each alias is an entry of gateway.aliases in the ai repo’s chart/values.yaml. Changing the model behind one needs no new server image.

Deployment

The ai repo holds a Helm chart in chart/. The umbrella deploys it once per stack with two Argo CD Applications in infra/argocd/ai.yaml, ai-dev into shopnexus-dev and ai-prod into shopnexus-prod. Each has one source, the ai repo’s chart/, ai-dev at main and ai-prod at the ai commit of the umbrella’s release, and sets the stack’s own values inline in spec.source.helm.valuesObject. root creates them in wave 2, after the stack Applications in wave 1, whose Postgres holds the gateway’s database and whose sealed secrets hold the keys.

flowchart TB
  app["Application ai-dev<br/>repo shopnexus/ai, chart/, main<br/>+ valuesObject in infra/argocd/ai.yaml"]
  app -->|wave 0| gw["Deployment and Service ai-gateway<br/>ghcr.io/berriai/litellm-database:v1.108.0<br/>ConfigMap ai-gateway-config"]
  app -->|wave 0| models["Deployment and Service ai-models<br/>ghcr.io/shopnexus/ai"]
  app -->|"wave 1, Sync hook"| job["Job ai-gateway-keys<br/>curlimages/curl"]
  gw --> db[("postgres-rw<br/>database litellm")]
What ai-dev renders into shopnexus-dev
ValuesWhereHolds
the hub’s choiceschart/values.yaml in the ai repothe aliases and their models, the images, the keys and the aliases each may call, the settings of ai-models
the stack’svaluesObject of ai-<stack> in infra/argocd/ai.yamlstack, every key’s dailyUnits and tpmLimit or rpmLimit, models.embedModels, models.memoryRequest and models.memoryLimit

The chart refuses to render without the stack’s values. Image Updater moves the ai-models image on ai-dev by the digest of :main. On ai-prod the umbrella’s Release workflow writes the ai commit and images.models.tag (Releases). A change to the chart or to chart/values.yaml reaches dev as soon as it is on the ai repo’s main, and prod with the next release.

Request path

flowchart LR
  server["server pod<br/>LLM_MODEL=search"] -->|"POST /v1/chat/completions<br/>Bearer LLM_API_KEY"| gw["ai-gateway:4000<br/>alias search"]
  gw -->|"checks key, budget, tpm"| db[("Postgres<br/>database litellm")]
  gw -->|"openai/anthropic/claude-haiku-5.5<br/>Bearer OPENROUTER_API_KEY"| or["OpenRouter<br/>openrouter.ai/api/v1"]
One search query, from the server to the model
flowchart LR
  server["server pod<br/>module assist"] -->|"POST /v1/video/frames<br/>no key"| models["ai-models:8000<br/>ffmpeg, frames, Gipformer"]
  server -->|"POST /v1/audio/transcriptions, model stt<br/>Bearer STT_API_KEY"| gw["ai-gateway:4000<br/>aliases stt, listing-read"]
  server -->|"POST /v1/chat/completions, model listing-read<br/>Bearer ASSIST_LLM_API_KEY"| gw
  gw -->|"openai/gipformer-1.5<br/>no key"| models
  gw -->|"openai/anthropic/claude-haiku-5.5<br/>Bearer OPENROUTER_API_KEY"| or["OpenRouter"]
A video read, from the server to ai-models and the model
flowchart LR
  server["server pod<br/>EMBED_TEXT_MODEL=embed-text<br/>EMBED_IMAGE_MODEL=embed-image"] -->|"POST /v1/embeddings<br/>Bearer EMBED_INDEX_API_KEY or EMBED_SEARCH_API_KEY"| gw["ai-gateway:4000<br/>aliases embed-text, embed-image"]
  gw -->|"checks key, budget, rpm"| db[("Postgres<br/>database litellm")]
  gw -->|"openai/baai/bge-m3, provider DeepInfra<br/>Bearer OPENROUTER_API_KEY"| or["OpenRouter"]
  gw -->|"openai/siglip2-so400m-patch16-384<br/>no key"| models["ai-models:8000<br/>SigLIP 2"]
Embeddings, from the server to OpenRouter and ai-models

The server’s ConfigMap server-env (infra/stack/templates/_server-env.tpl) points every model setting at the hub:

LLM_BASE_URL: http://ai-gateway:4000/v1
LLM_MODEL: search
ASSIST_LLM_BASE_URL: http://ai-gateway:4000/v1
ASSIST_LLM_MODEL: listing-read
TRUST_LLM_BASE_URL: http://ai-gateway:4000/v1
TRUST_LLM_MODEL: scam-check
STT_BASE_URL: http://ai-gateway:4000/v1
STT_MODEL: stt
AI_FRAMES_BASE_URL: http://ai-models:8000/v1
EMBED_TEXT_BASE_URL: http://ai-gateway:4000/v1
EMBED_IMAGE_BASE_URL: http://ai-gateway:4000/v1
EMBED_TEXT_MODEL: embed-text
EMBED_IMAGE_MODEL: embed-image
EMBED_TEXT_VERSION: bge-m3@5617a9f61b028005a4858fdac845db406aefb181
EMBED_IMAGE_VERSION: siglip2-so400m-patch16-384@29f74e673c298fd07482d8e20138179282001eb7-fp32

The console agent gets LITELLM_BASE_URL=http://ai-gateway:4000/v1 and AGENT_MODEL_ID=console-agent. The server calls ai-models’ /v1/video/frames itself, because LiteLLM routes only OpenAI’s API. The Service names ai-gateway and ai-models resolve inside the stack’s own namespace, so the dev server reaches dev’s hub and the prod server reaches prod’s.

Aliases and the gateway config

The aliases in chart/values.yaml, search and embed-text shown:

chart/values.yaml in the ai repo
gateway:
  secret: litellm-secrets
  database: { host: postgres-rw, name: litellm, user: litellm, passwordSecret: litellm-db }
  aliases:
    search:
      model: openai/anthropic/claude-haiku-5.5
      apiBase: https://openrouter.ai/api/v1
      apiKeySecret: OPENROUTER_API_KEY
      price: token
    embed-text:
      model: openai/baai/bge-m3
      apiBase: https://openrouter.ai/api/v1
      apiKeySecret: OPENROUTER_API_KEY
      mode: embedding
      price: token
      extraBody: { provider: { order: [DeepInfra], allow_fallbacks: false } }

The chart renders one model_list entry per alias into the ConfigMap ai-gateway-config. The search entry and the shared settings come out as:

model_list:
  - model_name: search
    litellm_params:
      model: openai/anthropic/claude-haiku-5.5
      api_base: https://openrouter.ai/api/v1
      api_key: os.environ/OPENROUTER_API_KEY
    model_info:
      input_cost_per_token: 0.000001
      output_cost_per_token: 0.000001
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_prompts_in_spend_logs: false
  maximum_spend_logs_retention_period: 30d
litellm_settings:
  drop_params: true

The openai/ prefix tells LiteLLM to speak the OpenAI API to api_base. An alias with apiKeySecret sends that key of litellm-secrets; one without it sends api_key: none, for ai-models, which has no authentication. extraBody goes to the upstream as extra_body. drop_params: true makes LiteLLM drop request fields the upstream does not accept. The stack’s NetworkPolicies leave egress open, so ai-gateway reaches OpenRouter.

store_prompts_in_spend_logs: false keeps request bodies, the photos of photo search among them, out of LiteLLM’s spend logs, and the spend logs are kept 30 days. infra/probes/litellm-embed/run.sh measured this on LiteLLM v1.108.0 with a stub upstream on 2026-11-10: data URLs of 706 kB and 1.72 MB reached the upstream unchanged, three calls cost 0.001728 (576 tokens each, priced from the upstream’s usage), and none of the three spend log rows held an input.

OpenRouter

Every alias but embed-image and stt goes to OpenRouter with the stack’s OPENROUTER_API_KEY, one key per stack. Give that key a daily credit limit on OpenRouter; it is the last stop behind the gateway’s own budgets. Measured through LiteLLM v1.108.0 on 2026-11-17:

  • LiteLLM passes OpenRouter’s usage (tokens and cost) through for chat and embeddings, so budgets count the tokens OpenRouter reports.
  • An upstream 402, OpenRouter’s answer when the key’s credit is spent, reaches the client as HTTP 402 with code "402".
  • OpenRouter serves baai/bge-m3 from two providers. DeepInfra (fp32) gave the vectors of the TEI the stacks ran before at a cosine of 0.999999 on every reference text under 2048 tokens (ai/embed/testdata/bge-m3-reference.json); Parasail gave 0.996. embed-text pins DeepInfra with allow_fallbacks: false, so stored vectors and new ones stay in one space. LiteLLM forwards the pin: a Parasail pin through it gave Parasail’s vectors.
  • 16 inputs in one embed-text request took 2.3 s.

Budget model

LiteLLM budgets count money, and each alias’s price sets what that money stands for:

priceCost1 unit of budget is
token0.000001 per token, input and output1,000,000 tokens
second0.000001 per second of audio1,000,000 seconds
usdthe model’s list price, inputCostPerToken and outputCostPerToken1 USD

console-agent is priced at 2 and 10 USD per 1M tokens in and out. A key’s dailyUnits is in millionths of its aliases’ unit, and the chart divides it by 1,000,000 for max_budget. The chart refuses a key over aliases priced in different units.

KeySecretAliasesdev dailyUnitsdev limitprod dailyUnitsprod limit
server-llmserver-secrets LLM_API_KEYsearch200,000 tokens20,000 tpm2,000,000 tokens100,000 tpm
server-sttserver-secrets STT_API_KEYstt3,600 seconds10 rpm36,000 seconds60 rpm
server-assistserver-secrets ASSIST_LLM_API_KEYlisting-read1,200,000 tokens60,000 tpm12,000,000 tokens300,000 tpm
server-trustserver-secrets TRUST_LLM_API_KEYscam-check600,000 tokens30,000 tpm6,000,000 tokens150,000 tpm
server-embed-indexserver-secrets EMBED_INDEX_API_KEYembed-text, embed-image10,000,000 tokens120 rpm50,000,000 tokens600 rpm
server-embed-searchserver-secrets EMBED_SEARCH_API_KEYembed-text, embed-image1,000,000 tokens60 rpm10,000,000 tokens600 rpm
agentagent-secrets AGENT_LLM_API_KEYconsole-agent20 USD100,000 tpm20 USD200,000 tpm

chart/values.yaml sets the agent’s 20 USD a day for every stack; infra/argocd/ai.yaml sets the rest. LiteLLM resets each key’s spend every day, with budget_duration: 1d.

The server indexes listings with server-embed-index and embeds queries with server-embed-search. A backfill of every listing spends the index key, and photo search keeps its own budget. ai-models counts 576 tokens a photo, its 24 × 24 patches, and a text its token count; LiteLLM prices from that usage. A listing of ten photos costs about 5,800 tokens, so dev’s index budget covers about 1,700 listings a day.

A video read sends listing-read 8 frames at 768 px, the list of subcategories and the transcript, about 5,500 tokens, so dev’s budget covers about 200 reads a day and prod’s about 2,000. A photo read sends one photo, about 1,500 tokens; each account may start 30 video reads, 60 photo reads and 30 link reads a day. A scam check sends the prompt and 20 short messages, about 1,300 tokens, so dev covers about 460 checks a day and prod about 4,600.

A test on a local k3d cluster set a 60-token budget on the search key: once it was spent, calls failed with HTTP 422 and Budget has been exceeded, and a call for an alias outside the key’s models got a 403. A test against LiteLLM v1.108.0 sent 5.55 s of audio, and the spend of server-stt rose by 0.00000555. LiteLLM asks ai-models for verbose_json when the caller names no response_format, and prices every speech call from the duration field that json and verbose_json answers carry. A text answer has no duration and costs 0, so the server must not ask for text.

The key Job

The keys are created by the Job ai-gateway-keys, an Argo CD Sync hook in wave 1 of ai-<stack>, after ai-gateway (wave 0) is ready. chart/templates/gateway-keys.yaml renders it from gateway.keys. Its pod runs curlimages/curl and does this for each key:

flowchart TD
  start["Sync hook ai-gateway-keys"] --> update["POST /key/update<br/>key = the key's value from its Secret"]
  update -->|200| ok["log: key server-llm: HTTP 200"]
  update -->|not 200| generate["POST /key/generate<br/>same key value"]
  generate -->|200| ok
  generate -->|not 200| fail["log status and message<br/>exit 1, retry up to 10 times"]
ai-gateway-keys on every sync, for each key

Both calls send the same body: the key’s value from the Secret its secret names, key_alias (the key’s name), models, max_budget, budget_duration: 1d, and tpm_limit or rpm_limit. They authenticate with the master key from litellm-secrets. A successful sync logs one key <name>: HTTP 200 line per key, seven in all.

The key values are fixed in advance: the seal command generates them and seals them into server-secrets and agent-secrets, and the Job writes the same values into LiteLLM. Running the Job twice gives the same result. After the gateway’s database is lost, the update fails, the generate call recreates the key with the value the client already holds, and the client does not restart. A key renamed in gateway.keys keeps its value, so the next sync’s /key/update finds it by its value and changes its alias; on v1.108.0 on 2026-11-10 a renamed key kept its spend.

The response of LiteLLM contains the key, so the Job logs only the HTTP status. On a failure it also prints the message field of the answer.

kubectl -n shopnexus-dev logs job/ai-gateway-keys

argocd.argoproj.io/hook-delete-policy: BeforeHookCreation keeps the last run’s Job until the next sync replaces it.

The gateway’s database

LiteLLM keeps keys and spend in Postgres, in a database of its own on the stack’s CNPG cluster postgres. The stack chart makes it, since Postgres belongs to the stack:

  • postgres.yaml in infra/stack declares a managed role litellm with its password from the Secret litellm-db, and a CNPG Database named litellm owned by that role (stack wave 1).
  • ai-gateway builds DATABASE_URL=postgresql://litellm:$(DB_PASSWORD)@postgres-rw:5432/litellm from gateway.database.
  • The image is ghcr.io/berriai/litellm-database:v1.108.0, the LiteLLM build that uses a database.

The server’s data lives in the database app of the same cluster. The litellm role owns only litellm.

Secrets

go -C infra run ./cmd/seal <stack> generates every value below but OPENROUTER_API_KEY into the git-ignored infra/stack/secrets/<stack>.env. You write OPENROUTER_API_KEY yourself, and the seal command stops with set OPENROUTER_API_KEY until you do. The Secret names litellm-secrets and litellm-db are the same as before the hub.

Value in <stack>.envSealed intoRead by
LITELLM_MASTER_KEYlitellm-secretsai-gateway, the ai-gateway-keys Job, you in the admin UI
OPENROUTER_API_KEYlitellm-secretsai-gateway, toward OpenRouter
LITELLM_DB_PASSWORDlitellm-db (basic-auth), key passwordCNPG for the role, ai-gateway for DATABASE_URL
LLM_API_KEY, STT_API_KEY, ASSIST_LLM_API_KEY, TRUST_LLM_API_KEY, EMBED_INDEX_API_KEY, EMBED_SEARCH_API_KEYserver-secretsthe server; the ai-gateway-keys Job
AGENT_LLM_API_KEYagent-secretsthe console agent; the ai-gateway-keys Job

ai-gateway is in another Application than the stack’s secrets and does not carry their checksum. After a reseal of litellm-secrets or litellm-db, restart it by hand:

kubectl -n shopnexus-dev rollout restart deploy/ai-gateway

A key the chart names but its Secret lacks keeps the ai-gateway-keys pod from starting, and the sync of ai-<stack> stops at wave 1: reseal and push the secrets before the ai commit that adds the key reaches main. Secrets covers how the seal command and the controller seal them.

When the budget runs out

The gateway refuses a spent key with HTTP 422 and Budget has been exceeded. When the stack’s OpenRouter credit is spent, it passes OpenRouter’s 402 through. The server’s LLM client treats a 402 like a 429, as a passing failure, and its embed client retries it. The console agent’s isBudgetError takes a 402 as a spent budget, as it takes LiteLLM’s own budget_exceeded.

The server’s search does not fail when the model does. Search in server/internal/discovery/service/search.go gives the model 4 seconds. On any error, a refusal for budget or rate limit included, it logs the model did not read the query at warn level with the error, and searches with the raw words alone.

sequenceDiagram
  participant App as app
  participant S as server
  participant G as ai-gateway
  App->>S: GET /v1/search?q=...
  S->>S: cache lookup (7 days)
  S->>G: chat/completions, model search
  G-->>S: 422 Budget has been exceeded, or 402
  S->>S: log "the model did not read the query"
  S->>S: search the raw words as a phrase
  S-->>App: 200, listings, no chips
Search when ai-gateway refuses

The person still gets a page of listings, matched on the words they typed. The page has no chips, and no boosts for category, price, place or freshness that the model would have read out of the query. A query the model read in the last 7 days comes from the server’s cache and keeps its understanding. The server’s error for a non-200 answer is model answered <status>, so a spent budget logs model answered 422 and spent credit model answered 402.

Calls over the tokens-per-minute limit fail too, and search falls back to the raw words until the minute has passed. Search and discovery lists the boosts the model can set.

ai-models

ai-models is a Deployment and a Service on port 8000 with no Ingress and no authentication; from-stack admits the stack’s pods. The gateway’s aliases stt and embed-image call it, and the server calls its frames route directly. Its settings are models.* in chart/values.yaml.

Speech to text

ai-models serves OpenAI’s POST /v1/audio/transcriptions with Gipformer 1.5, a Vietnamese model with 68.6M parameters, on CPU. It takes any audio or video file ffmpeg reads, up to 100 MB and 300 s of audio (models.maxUploadMB, models.maxAudioSeconds). Silero VAD cuts the speech at silences into pieces of at most 20 s, and the answer is lower-case text without punctuation. Two requests decode at once (models.maxConcurrent), and the others wait.

The server calls stt when it reads a seller’s video (module assist). It asks for response_format: json, so LiteLLM prices the call from duration.

To send stt to another provider, set gateway.aliases.stt.model to that provider’s LiteLLM model string, for example groq/whisper-large-v3, and gateway.aliases.stt.apiBase to its base URL; add the provider’s key to litellm-secrets and name it in gateway.aliases.stt.apiKeySecret. The server keeps asking for stt.

ai-models logs one line per request, without the transcript:

kubectl -n shopnexus-dev logs deploy/ai-models

Video frames

POST /v1/video/frames is not an OpenAI route, so the server calls it at AI_FRAMES_BASE_URL with no key, and the gateway counts nothing for it. It takes a video of 3 to 35 s (models.framesMaxSeconds), decodes 2 frames a second at most 1600 px on the long side, drops frames whose variance of the Laplacian is under models.framesMinSharpness and frames that repeat another, and returns at most 8 JPEGs spread over the video. Two requests cut frames at once (models.framesMaxConcurrent), apart from the two that decode speech. On 2026-11-10, two frame requests of a 30 s 1080p video and two transcriptions at once raised the pod to 2.42 GiB.

Photo embeddings

ai-models serves POST /v1/embeddings with SigLIP 2 (siglip2-so400m-patch16-384, fp32 ONNX, 1152 dimensions) when models.embedModels names it. An input that starts with data:image/ is a photo; any other input is text for SigLIP’s text tower. The weights are 4.5 GB. With models.embedModels empty, ai-models loads no SigLIP 2, and /v1/embeddings answers 404 model_not_found.

Stackmodels.embedModelsMemory request, limitMeasured
devempty2Gi, 3Gispeech and frames alone peaked at 2.42 GiB on 2026-11-10
prodsiglip2-so400m-patch16-384=/models/siglip2-so400m-patch16-3846Gi, 8Gi4.0 GiB idle and 5.3 GiB on 16 photos, 2026-11-10

Dev loads no SigLIP 2 because the dev machine has no room for it (2026-11-17). Its stack values set server.embed.image: false, so the server writes and queries no photo vectors there. infra/check.sh fails when a stack’s photo vectors are on without SigLIP 2 in its ai-models, or off with it.

Vectors and their versions

The server stores EMBED_TEXT_VERSION and EMBED_IMAGE_VERSION with every vector. Both are in infra/stack/env/server.env, which the cluster’s chart and the root docker-compose.yml both read. A new model needs a new version in the same change: the server then treats the stored vectors as missing, and go run ./cmd/search-reindex -every 2s in the server embeds every listing again, one listing every 2 s.

EMBED_TEXT_VERSION stayed bge-m3@5617a9f61b028005a4858fdac845db406aefb181 when embed-text moved from TEI to OpenRouter, since DeepInfra’s vectors match. A text over 2048 tokens gave a cosine of 0.90, because TEI cut inputs at 2048 tokens and OpenRouter does not. The server cuts listing text at 2,000 characters (EmbeddingTextMax), under 2048 tokens.

Each kind of vector can be off. server.embed.text and server.embed.image in a stack’s values turn it on or off, and an off kind gets empty EMBED_*_MODEL, EMBED_*_BASE_URL and EMBED_*_VERSION. The server then writes no vectors of that kind and leaves its list out of search; BM25 always runs. A photo search with photo vectors off answers 503 search_unavailable.

Local stacks

The root docker-compose.yml runs ai-models-dev on 127.0.0.1:56491 and ai-models-prod on 127.0.0.1:56490, with no gateway and no TEI. The server calls ai-models-<stack> directly for photo vectors, speech and frames, and names the model siglip2-so400m-patch16-384; ai-models takes no key and ignores the one sent. For text vectors it reads EMBED_TEXT_BASE_URL, EMBED_TEXT_MODEL, EMBED_INDEX_API_KEY and EMBED_SEARCH_API_KEY from server/.env: OpenRouter’s baai/bge-m3 with your OpenRouter key, or an empty EMBED_TEXT_MODEL, which turns text vectors off. infra/check.sh fails when the AI_* settings of the compose file differ from models.* in the ai chart.

Smoke test

infra/smoke-models.sh <stack> port-forwards svc/ai-gateway and calls every alias with the server’s keys: search, listing-read with a photo, scam-check, stt with a silent clip, embed-text (1024 dimensions), and embed-image with a photo and with a text (1152 dimensions). It skips the embed-image calls when the stack’s EMBED_IMAGE_MODEL is empty. Then it checks that server-llm is refused embed-text, and server-assist and server-trust are refused search, each with a 403. It needs hurl and kubectl.

Change the model or the provider

Clients always ask for the same aliases. To change what answers search:

  1. Edit the values

    In the ai repo’s chart/values.yaml, set gateway.aliases.search.model to LiteLLM’s model string, provider prefix included (openai/anthropic/claude-haiku-5.5), and gateway.aliases.search.apiBase to its OpenAI-compatible base URL. To change one stack only, set them under gateway.aliases.search in the valuesObject of ai-<stack> in infra/argocd/ai.yaml instead. The chart refuses to render if either is empty.

  2. Add the provider's key

    For a provider other than OpenRouter, add its key to infra/stack/secrets/<stack>.env and to litellm-secrets in the seal command, name it in apiKeySecret, reseal both stacks and push, and restart ai-gateway.

  3. Run the tests and push

    Run helm unittest chart in the ai repo, commit and push to its main. Argo CD syncs ai-dev; ai-prod gets the change with the next release. The checksum/config annotation on ai-gateway’s pod changes with the config, so the gateway restarts on the new model.

  4. Check a search

    Search for something on the stack and read the server log. A line the model did not read the query means the new model failed.

The server asks for a strict json_schema response format. A model that ignores it returns text the server cannot decode, and search falls back to the raw words on every query. To change embed-text or embed-image, see Vectors and their versions: the vectors’ version changes with the model.

Change a budget

  1. Edit the stack's values

    In infra/argocd/ai.yaml, under the valuesObject of ai-dev or ai-prod, change a key’s dailyUnits or its tpmLimit or rpmLimit under gateway.keys. The agent’s daily budget is in the ai repo’s chart/values.yaml.

  2. Check and push

    Run infra/check.sh, which renders the ai chart with each stack’s values, then commit and push the umbrella. root updates the Application, ai-<stack> syncs, and the ai-gateway-keys Job runs /key/update on every key with the new limits.

  3. Check the Job

    kubectl -n shopnexus-dev logs job/ai-gateway-keys prints one key <name>: HTTP 200 line per key.

Add another model alias

An alias is an entry of gateway.aliases, and a key reaches only the aliases in its models:

  1. Add the entry to gateway.aliases in the ai repo’s chart/values.yaml, with model, apiBase, price (token, second or usd), and apiKeySecret, mode, extraBody, inputCostPerToken and outputCostPerToken when it needs them.
  2. Add its name to the models of the key that calls it. For a new caller, add an entry to gateway.keys with its secret (name and key of the client’s Secret) and models, set its dailyUnits and a tpmLimit or rpmLimit for each stack in infra/argocd/ai.yaml, and add the key to the seal command’s generated keys and to the client’s Secret.
  3. Run helm unittest chart in the ai repo and infra/check.sh in the umbrella before you push. The chart refuses a key that names a missing alias, a key over aliases priced in different units, a key with neither limit, and an alias without a model.

A new alias is used only by a caller that asks for it.

Admin UI and spend

The stack’s dashboard host litellm.<domain>, behind the console’s sign-in, routes to the Service ai-gateway. From a shell, use a port-forward and the master key:

kubectl -n shopnexus-dev get secret litellm-secrets -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d; echo
kubectl -n shopnexus-dev port-forward svc/ai-gateway 4000:4000

Open http://localhost:4000/ui and sign in with user admin and the master key as the password. The UI lists the keys with aliases server-llm, server-stt, server-assist, server-trust, server-embed-index, server-embed-search and agent, each with its spend, budget and reset time. Use shopnexus-prod for prod.

To read the spend from a shell while the port-forward runs:

master=$(kubectl -n shopnexus-dev get secret litellm-secrets -o jsonpath='{.data.LITELLM_MASTER_KEY}' | base64 -d)
key=$(kubectl -n shopnexus-dev get secret server-secrets -o jsonpath='{.data.LLM_API_KEY}' | base64 -d)
curl -s "http://localhost:4000/key/info?key=$key" -H "Authorization: Bearer $master"

info.spend is in budget units: a spend of 0.05 on server-llm is 50,000 tokens today, and on agent 0.05 USD. info.max_budget is the day’s limit and info.budget_reset_at the next reset.

Key rules

  • Clients call only aliases of gateway.aliases; the model behind each is set in the ai repo’s chart/values.yaml.
  • Budgets are set per stack in infra/argocd/ai.yaml, in tokens, seconds or millionths of a dollar.
  • Indexing and search embed with separate keys, server-embed-index and server-embed-search.
  • The key values come from the seal command; the ai-gateway-keys Job only writes them into LiteLLM.
  • After a reseal of litellm-secrets, ai-gateway restarts only by hand.
  • embed-text stays pinned to DeepInfra while EMBED_TEXT_VERSION names TEI’s bge-m3.
ShopNexus Docs
EN VI