Self Host LLM Gateway
Deploy LLM Gateway on your own infrastructure with Docker, Docker Compose, or Kubernetes on AWS, GCP, and Azure.
LLM Gateway is a self-hostable platform that provides a unified API gateway for multiple LLM providers. Run it on your own infrastructure to keep full control over your data and avoid platform fees.
Pick the deployment path that matches where you're running it.
Single host
Which option should I choose?
- Trying it out or running a single low-traffic instance? Start with Docker or Docker Compose on one machine.
- Running in production? Deploy to Kubernetes with our Helm chart, and use a managed Postgres and Redis from your cloud.
- On a specific cloud? Follow the AWS, Google Cloud, or Azure guide for the exact managed services to provision.
- Unlocking Enterprise features? Install the signed token using the Enterprise licensing guide.
What you'll need
Every deployment is built from the same pieces:
- Stateless services — the gateway, API, UI, and a background worker. Scale these freely; they hold no data between requests.
- PostgreSQL — the source of truth for users, projects, keys, and usage records.
- Redis — response caching and the queue that feeds the worker.
- Provider API keys — the OpenAI, Anthropic, Google, and other credentials the gateway uses, injected as secrets.
In production, run PostgreSQL and Redis as managed services from your cloud provider so backups, failover, and patching are handled for you.
Database migrations
The API applies pending migrations at startup when RUN_MIGRATIONS=true. You can
also run them as a one-off job before rolling out new images:
node dist/migrate.js in the API image applies them, and
node dist/migrate.js --check is a read-only preflight that reports pending
migrations, index changes that need a manual step, and long-running
transactions holding locks on the tables they touch.
Each migration runs in its own transaction. A migration that cannot take a
lock within MIGRATIONS_LOCK_TIMEOUT_MS rolls back and retries with backoff
instead of queueing your traffic behind it; the error names the sessions
holding the lock.
| Variable | Default | Purpose |
|---|---|---|
MIGRATIONS_LOCK_TIMEOUT_MS | 5000 | Lock wait per statement before an attempt gives up |
MIGRATIONS_MAX_ATTEMPTS | 5 | Attempts per migration after lock timeouts or deadlocks |
MIGRATIONS_STATEMENT_TIMEOUT_MS | 900000 | Runtime cap per statement; 0 disables it |
MIGRATIONS_RUNNER_LOCK_WAIT_MS | 900000 | How long a replica waits for another replica's migration run |
MIGRATIONS_CHECK_MAX_TRANSACTION_AGE_S | 30 | Transaction age at which --check reports a lock holder |
MIGRATIONS_REQUIRE_PREBUILT_INDEXES | false | Refuse to deploy index changes on existing tables until they were built or dropped CONCURRENTLY |
With MIGRATIONS_REQUIRE_PREBUILT_INDEXES=true, a pending migration that
creates or drops an index on an existing table fails before any migration runs
and prints the exact CREATE INDEX CONCURRENTLY / DROP INDEX CONCURRENTLY
commands to run first (including dropping an INVALID index left by a failed
build). Run them with psql outside a transaction, then deploy again. Leave it
off for small databases, where building an index inside the migration is quick.
Product links
For self-hosted DevPass and Airside, set UI_URL and PLAYGROUND_URL to your Gateway dashboard and Lounge origins. When unset, these apps use https://llmgateway.io and https://lounge.llmgateway.io; with NODE_ENV=development, they use http://localhost:3002 and http://localhost:3003.
For frontends hosted separately from the backend, follow Split deployments to configure public and server-side URLs, shared session cookies, CORS, and client IP forwarding.
API explorer
Set GATEWAY_URL on the docs service to your public gateway URL. The API reference
uses that server in its examples and sends interactive requests through the docs
service's /api/proxy. Set GATEWAY_BACKEND_URL when the docs server should reach
the gateway over a separate internal URL; API keys and the configured client-IP
header are forwarded to it. Public and internal base paths may differ; the proxy
preserves the operation path and query.
Without GATEWAY_URL, production uses https://api.llmgateway.io and development
uses http://localhost:4001. Set it explicitly on isolated deployments to avoid
sending interactive requests to the hosted gateway.
Database sessions and health checks
The gateway, API, and worker set these limits on every Postgres connection, so
a query stuck behind a lock fails instead of holding a pool slot until the
blocker finishes. Each session's application_name is <service>/<hostname>,
which identifies it in pg_stat_activity.
| Variable | Default | Purpose |
|---|---|---|
DATABASE_LOCK_TIMEOUT_MS | 10000 | Lock wait before a statement fails |
DATABASE_IDLE_IN_TRANSACTION_TIMEOUT_MS | 60000 | Aborts a transaction left idle |
DATABASE_STATEMENT_TIMEOUT_MS | server default | Runtime cap per statement |
DATABASE_APPLICATION_NAME | <service>/<hostname> | Overrides the session's application_name |
METRICS_PORT | gateway 9090; API and worker off | Port serving Prometheus /metrics, including db_pool_* pool occupancy |
On the gateway and API, point readiness at / and liveness at /livez.
After boot, / returns 503 until one Postgres query and one Redis ping
succeed, so a release that cannot reach them never replaces healthy pods. From
then on neither endpoint touches Postgres or Redis, so a database or Redis
incident neither restarts healthy pods nor removes them from the load balancer.
For external monitoring, /health/deep checks Postgres and Redis and returns
503 when either is unreachable.
Admin dashboard access
Grant admin dashboard access with comma-separated email lists on the API service. Only verified email addresses count.
| Variable | Role | Access |
|---|---|---|
ADMIN_FULL_ACCESS_EMAILS | Admin | Full access |
ADMIN_SUPPORT_EMAILS | Support | Restricted reads, plus issuing DevPass refunds |
ADMIN_VIEWER_EMAILS | Viewer | Read-only, including financial information |
Support accounts do not see platform-wide revenue or profit: the dashboard home, Global Stats, LLM SDK, DevPass and Lounge plan KPIs, and provider credential spend are hidden, and the gateway's own margin and fee fields — including Airside carrier margins and routing adjustments — are removed from every response. A subscriber's plan margin stays visible, since it is the plan price minus usage support already sees. Request payloads in logs are shown unmodified.
Gateway during database outages
The gateway keeps a stale copy of the API keys, projects, organizations, and
provider settings it reads from Postgres in Redis (SWR_STALE_TTL_SECONDS,
default 4 hours) and serves it when a database read fails. After
DB_CIRCUIT_FAILURE_THRESHOLD (default 5) consecutive connection, lock, or
timeout errors, it stops querying Postgres for DB_CIRCUIT_OPEN_MS (default
5000): requests with a stale copy are served from it, and the rest fail
immediately instead of waiting for a database connection. One request per
interval probes Postgres and closes the circuit once it succeeds.
Global Stats time ranges
Global Stats defaults to UTC. Choose Display time zone to use your device's current zone or another named zone. The choice is saved for later visits. Chart labels and tooltips show bucket start times in that zone. Calendar ranges and daily totals retain UTC day boundaries.
Last 24 hours uses hourly aggregates for the 24 complete hours ending at the latest UTC hour boundary, with the exact window shown above the charts. It spans midnight and uses the same window for totals, breakdowns, and provider filters. The current incomplete hour is excluded. The worker refreshes hourly after a settling delay, so recent hours may still be processing. CSV timestamps remain in UTC. Apply database migrations and run the worker to populate hourly history.
Activity log searches
Dashboard log searches run with a statement timeout and a short lock timeout and are limited in concurrency, so a slow search cannot hold locks on the log table. Concurrency slots live in Redis; if Redis is unreachable, searches run unlimited.
| Variable | Default | Purpose |
|---|---|---|
LOG_QUERY_TIMEOUT_MS | 10000 | Cancels a log search after this long |
LOG_QUERY_LOCK_TIMEOUT_MS | 2000 | Fails a search that would wait on a lock |
LOG_QUERY_MAX_CONCURRENT_PER_USER | 3 | Concurrent searches per user |
LOG_QUERY_MAX_CONCURRENT_PER_ORG | 6 | Concurrent searches per organization |
Gateway load
The admin dashboard's Gateway Load → Requests view reports peak throughput from completed intervals: minutes for short model/provider views, and hours for longer ranges or tenant-filtered views. Daily charts retain hourly peaks, so switching from 7 to 30 days does not smooth away a spike. Peak timestamps identify the start of the interval; the current incomplete interval is excluded.
The All / Credits / BYOK filter applies to every value on the page:
requests, rates, error rates, error counts, duration, and time to first token,
on every grouping and tenant filter. It defaults to Credits, so
bring-your-own-key traffic is excluded unless you pick All or BYOK. After upgrading, the worker backfills the
per-mode error and latency split for organization, project, and API-key views
over the last 365 days (TENANT_STATS_BYOK_BACKFILL_DAYS); until it reaches
an hour, that hour counts its BYOK errors and latency as credits.
Switch to Tokens for platform, model, or provider throughput over 7 or 30 days. Choose total, input, or output tokens and a billing mode. Average TPM includes quiet minutes; average tokens/day normalizes the same window to 24 hours. Peak TPM is the busiest completed UTC minute, and peak tokens/day is the busiest complete UTC calendar day. Partial days appear faded in the daily chart and do not count toward daily peaks. The table ranks all models or providers by token volume, with ten rows per page. CSV export includes every row.
Token metrics use recorded upstream usage, excluding gateway response-cache hits. Minute history is retained for 30 days. Measurements follow worker refreshes; reload the page to update them. The Tokens view does not auto-refresh. These fixed UTC intervals do not reproduce a provider's rolling rate-limit or quota accounting. Organization, project, and API-key filters apply only to the Requests view.
Support chat diagnostics
Support chat errors can arrive inside an HTTP 200 event stream. In the API
logs, search for the conversation's conversationId, then follow requestId
for each visitor attempt and gatewayRequestId into the gateway logs.
Chat support request finished records whether the attempt completed, failed,
returned an empty answer, was rejected, was aborted, or stayed silent because
the conversation was escalated. Failure events include the stage and available
provider error code or HTTP status. Gateway and step events record timing,
selected models, token counts, and tool-call counts without recording chat
contents or raw SDK error payloads.
Sign-up email restrictions
With HOSTED=true, administrators can manage the custom email-domain blocklist in
Admin → Settings → Blocked sign-up email domains. Paste domains one per line
or separated by commas, then save. Entries also block subdomains; updates apply
to subsequent email sign-ups without a restart. Existing users can still sign in.
The custom list starts empty and has no built-in entries. Clearing and saving it removes custom domain blocks. The separate disposable-email and plus-address checks remain active in hosted mode.
Provider base URL overrides
A provider key may carry a base URL that replaces the
provider's published endpoint. Self-hosted deployments allow it; HOSTED=true
rejects it, so a provider is reached only at its published endpoint with the
key's own settings (such as an Azure resource) applied. Set
ALLOW_PROVIDER_BASE_URL_OVERRIDE to true or false to override either
default. Custom providers are unaffected — their
base URL defines them.
Where the override is disallowed, a key that already stores one is skipped by routing rather than redirected to the published endpoint, whose host the key's token may not be meant for. Another key for the provider, another provider, or (in hybrid mode) credits serve the request instead, and the gateway logs the key it skipped.
How is this guide?
Last updated on