Litellm

LLM gateway: one OpenAI-compatible API for every provider, with virtual keys and spend limits

LiteLLM on Cubeship

LiteLLM is an open-source LLM gateway: one OpenAI-compatible API in front of OpenAI, Anthropic, Gemini, Bedrock, OpenRouter, Ollama and a hundred other providers, with virtual keys, budgets, rate limits and spend tracking per key, team and user.

This template installs the LiteLLM proxy on a Cubeship instance with the managed Postgres it keeps its models, keys and spend in.

What it creates

litellm — the LiteLLM proxy, from ghcr.io/berriai/litellm:v1.100.1. The API and the admin UI (at /ui) answer on the domain you choose, on port 4000 inside the instance.
litellm-db — a managed Postgres 18 database, attached to the app, so DATABASE_URL is set for you. The proxy creates its tables on first start.

What you are asked

Input	What to give
Where the proxy and its admin UI answer	A domain you control, pointed at your instance.
The master key, without its sk- prefix	Nothing — the instance generates it and shows it once. Keep a copy.
The key provider credentials are encrypted with	Nothing — the instance generates it and shows it once. Keep a copy, and never change it.
The username you sign in to the admin UI with	Anything; `admin` unless you change it.
The password you sign in to the admin UI with	Nothing — the instance generates it and shows it once.

The two keys start with `sk-`

LiteLLM expects its master key to start with sk-, and the instance generates letters and digits only, so the template sets LITELLM_MASTER_KEY to sk- followed by the generated value. When you use the master key, type the sk- in front of what the instance showed you. LITELLM_SALT_KEY is composed the same way.

The salt key must never change

LITELLM_SALT_KEY encrypts the provider API keys you add in the admin UI before they are written to the database. Change it, or lose it and set a new one, and every stored provider key becomes unreadable: each model has to be added again. Keep it wherever you keep the database's backups.

No model provider key is asked for. Add them in the admin UI instead: they are stored in the database, encrypted.

After installing

Open https://<your domain>/ui and sign in with the username and the generated password.
Under Models, add a model: choose the provider, the model name and its API key, and test the connection.
Under Virtual Keys, create a key for each app or person that will call the proxy, and choose which models it may use. Give that key out, never the master key.

Pointing other apps at it

Apps on the same instance reach the proxy at its internal address, without going through the domain:

http://cubeship-litellm-production-litellm:4000

with the suggested names — the address is on the app's page in the dashboard. Use a virtual key as the API key, and a model name exactly as you added it under Models.

OpenCode — add an OpenAI-compatible provider to opencode.json (in /root/.config/opencode), with "npm": "@ai-sdk/openai-compatible", options.baseURL set to the address above followed by /v1, and options.apiKey set to the virtual key; list the models under models.
OpenHands — under Settings, open the advanced options, set the model to litellm_proxy/<model name>, the base URL to the address above, and the API key to the virtual key.
OpenClaw — under Settings → Models, add an OpenAI-compatible provider with the address above followed by /v1 as its base URL and the virtual key as its API key.
Open WebUI — under Admin Settings → Connections, add an OpenAI API connection with the address above followed by /v1 as its URL and the virtual key as its key. Its model list is the models the key may use.
Your own code — any OpenAI SDK, with the base URL http://cubeship-litellm-production-litellm:4000/v1 inside the instance, or https://<your domain>/v1 from anywhere else.

It is on the internet

The domain serves the API and the admin UI. Every API route asks for a virtual key or the master key, and the UI asks for its username and password; only the health checks and a few public pages answer without one. The master key can create keys, read every key's spend and change every setting: keep it out of apps, and give each one a virtual key with a budget instead.

Resources

The app is limited to 1 CPU and 4 GiB of memory, what LiteLLM recommends for one proxy process. The proxy only forwards requests, so it needs no GPU and no more CPU for bigger models; raise limits in template.yaml if many requests arrive at once.

What this creates

litellm

ghcr.io/berriai/litellm:v1.100.1

litellm-db

Postgres 18

template.yaml

# yaml-language-server: $schema=https://cubeship.dev/schema/template/v1.json
version: 1
minCubeship: "0.6.0"
project: litellm

inputs:
  - key: domain
    type: domain
    label: Where the proxy and its admin UI answer
  - key: masterKey
    type: secret
    label: The master key, without its sk- prefix
    help: The key is sk- followed by this value. It is the proxy's admin credential; keep a copy.
    generate: 48
  - key: saltKey
    type: secret
    label: The key provider credentials are encrypted with, without its sk- prefix
    help: Keep a copy, and never change it. Changing it makes every provider key stored in the proxy unreadable.
    generate: 48
  - key: uiUsername
    type: text
    label: The username you sign in to the admin UI with
    default: admin
  - key: uiPassword
    type: secret
    label: The password you sign in to the admin UI with
    generate: 24

databases:
  - key: db
    name: litellm-db
    engine: postgres
    version: "18"
    database: litellm

apps:
  - key: proxy
    name: litellm
    # The published stable release; main-stable points at the same image.
    image: ghcr.io/berriai/litellm
    tag: "v1.100.1"
    port: 4000
    # Answers without a key, and does not wait on the database.
    health: /health/liveliness
    domains:
      - host: ${input.domain}
    attach:
      - database: db
    limits: { cpu: 1, memory: 4Gi }
    env:
      # LiteLLM expects both keys to start with sk-; generated secrets are
      # alphanumeric only.
      LITELLM_MASTER_KEY: sk-${input.masterKey}
      LITELLM_SALT_KEY: sk-${input.saltKey}
      # Models and their provider keys are added in the admin UI and kept in
      # the database, so no config file is needed.
      STORE_MODEL_IN_DB: "True"
      UI_USERNAME: ${input.uiUsername}
      UI_PASSWORD: ${input.uiPassword}