LLM Gateway
Self Host

Self Host LLM Gateway

Deploy LLM Gateway on your own infrastructure with Docker, Docker Compose, or Kubernetes on AWS, GCP, and Azure.

LLM Gateway is a self-hostable platform that provides a unified API gateway for multiple LLM providers. Run it on your own infrastructure to keep full control over your data and avoid platform fees.

Pick the deployment path that matches where you're running it.

Which option should I choose?

  • Trying it out or running a single low-traffic instance? Start with Docker or Docker Compose on one machine.
  • Running in production? Deploy to Kubernetes with our Helm chart, and use a managed Postgres and Redis from your cloud.
  • On a specific cloud? Follow the AWS, Google Cloud, or Azure guide for the exact managed services to provision.
  • Unlocking Enterprise features? Install the signed token using the Enterprise licensing guide.

What you'll need

Every deployment is built from the same pieces:

  • Stateless services — the gateway, API, UI, and a background worker. Scale these freely; they hold no data between requests.
  • PostgreSQL — the source of truth for users, projects, keys, and usage records.
  • Redis — response caching and the queue that feeds the worker.
  • Provider API keys — the OpenAI, Anthropic, Google, and other credentials the gateway uses, injected as secrets.

In production, run PostgreSQL and Redis as managed services from your cloud provider so backups, failover, and patching are handled for you.

Database migrations

The API applies pending migrations at startup when RUN_MIGRATIONS=true. You can also run them as a one-off job before rolling out new images: node dist/migrate.js in the API image applies them, and node dist/migrate.js --check is a read-only preflight that reports pending migrations, index changes that need a manual step, and long-running transactions holding locks on the tables they touch.

Each migration runs in its own transaction. A migration that cannot take a lock within MIGRATIONS_LOCK_TIMEOUT_MS rolls back and retries with backoff instead of queueing your traffic behind it; the error names the sessions holding the lock.

VariableDefaultPurpose
MIGRATIONS_LOCK_TIMEOUT_MS5000Lock wait per statement before an attempt gives up
MIGRATIONS_MAX_ATTEMPTS5Attempts per migration after lock timeouts or deadlocks
MIGRATIONS_STATEMENT_TIMEOUT_MS900000Runtime cap per statement; 0 disables it
MIGRATIONS_RUNNER_LOCK_WAIT_MS900000How long a replica waits for another replica's migration run
MIGRATIONS_CHECK_MAX_TRANSACTION_AGE_S30Transaction age at which --check reports a lock holder
MIGRATIONS_REQUIRE_PREBUILT_INDEXESfalseRefuse to deploy index changes on existing tables until they were built or dropped CONCURRENTLY

With MIGRATIONS_REQUIRE_PREBUILT_INDEXES=true, a pending migration that creates or drops an index on an existing table fails before any migration runs and prints the exact CREATE INDEX CONCURRENTLY / DROP INDEX CONCURRENTLY commands to run first (including dropping an INVALID index left by a failed build). Run them with psql outside a transaction, then deploy again. Leave it off for small databases, where building an index inside the migration is quick.

For self-hosted DevPass and Airside, set UI_URL and PLAYGROUND_URL to your Gateway dashboard and Lounge origins. When unset, these apps use https://llmgateway.io and https://lounge.llmgateway.io; with NODE_ENV=development, they use http://localhost:3002 and http://localhost:3003.

For frontends hosted separately from the backend, follow Split deployments to configure public and server-side URLs, shared session cookies, CORS, and client IP forwarding.

API explorer

Set GATEWAY_URL on the docs service to your public gateway URL. The API reference uses that server in its examples and sends interactive requests through the docs service's /api/proxy. Set GATEWAY_BACKEND_URL when the docs server should reach the gateway over a separate internal URL; API keys and the configured client-IP header are forwarded to it. Public and internal base paths may differ; the proxy preserves the operation path and query.

Without GATEWAY_URL, production uses https://api.llmgateway.io and development uses http://localhost:4001. Set it explicitly on isolated deployments to avoid sending interactive requests to the hosted gateway.

Database sessions and health checks

The gateway, API, and worker set these limits on every Postgres connection, so a query stuck behind a lock fails instead of holding a pool slot until the blocker finishes. Each session's application_name is <service>/<hostname>, which identifies it in pg_stat_activity.

VariableDefaultPurpose
DATABASE_LOCK_TIMEOUT_MS10000Lock wait before a statement fails
DATABASE_IDLE_IN_TRANSACTION_TIMEOUT_MS60000Aborts a transaction left idle
DATABASE_STATEMENT_TIMEOUT_MSserver defaultRuntime cap per statement
DATABASE_APPLICATION_NAME<service>/<hostname>Overrides the session's application_name
METRICS_PORTgateway 9090; API and worker offPort serving Prometheus /metrics, including db_pool_* pool occupancy

On the gateway and API, point readiness at / and liveness at /livez. After boot, / returns 503 until one Postgres query and one Redis ping succeed, so a release that cannot reach them never replaces healthy pods. From then on neither endpoint touches Postgres or Redis, so a database or Redis incident neither restarts healthy pods nor removes them from the load balancer. For external monitoring, /health/deep checks Postgres and Redis and returns 503 when either is unreachable.

Admin dashboard access

Grant admin dashboard access with comma-separated email lists on the API service. Only verified email addresses count.

VariableRoleAccess
ADMIN_FULL_ACCESS_EMAILSAdminFull access
ADMIN_SUPPORT_EMAILSSupportRestricted reads, plus issuing DevPass refunds
ADMIN_VIEWER_EMAILSViewerRead-only, including financial information

Support accounts do not see platform-wide revenue or profit: the dashboard home, Global Stats, LLM SDK, DevPass and Lounge plan KPIs, and provider credential spend are hidden, and the gateway's own margin and fee fields — including Airside carrier margins and routing adjustments — are removed from every response. A subscriber's plan margin stays visible, since it is the plan price minus usage support already sees. Request payloads in logs are shown unmodified.

Gateway during database outages

The gateway keeps a stale copy of the API keys, projects, organizations, and provider settings it reads from Postgres in Redis (SWR_STALE_TTL_SECONDS, default 4 hours) and serves it when a database read fails. After DB_CIRCUIT_FAILURE_THRESHOLD (default 5) consecutive connection, lock, or timeout errors, it stops querying Postgres for DB_CIRCUIT_OPEN_MS (default 5000): requests with a stale copy are served from it, and the rest fail immediately instead of waiting for a database connection. One request per interval probes Postgres and closes the circuit once it succeeds.

Global Stats time ranges

Global Stats defaults to UTC. Choose Display time zone to use your device's current zone or another named zone. The choice is saved for later visits. Chart labels and tooltips show bucket start times in that zone. Calendar ranges and daily totals retain UTC day boundaries.

Last 24 hours uses hourly aggregates for the 24 complete hours ending at the latest UTC hour boundary, with the exact window shown above the charts. It spans midnight and uses the same window for totals, breakdowns, and provider filters. The current incomplete hour is excluded. The worker refreshes hourly after a settling delay, so recent hours may still be processing. CSV timestamps remain in UTC. Apply database migrations and run the worker to populate hourly history.

Activity log searches

Dashboard log searches run with a statement timeout and a short lock timeout and are limited in concurrency, so a slow search cannot hold locks on the log table. Concurrency slots live in Redis; if Redis is unreachable, searches run unlimited.

VariableDefaultPurpose
LOG_QUERY_TIMEOUT_MS10000Cancels a log search after this long
LOG_QUERY_LOCK_TIMEOUT_MS2000Fails a search that would wait on a lock
LOG_QUERY_MAX_CONCURRENT_PER_USER3Concurrent searches per user
LOG_QUERY_MAX_CONCURRENT_PER_ORG6Concurrent searches per organization

Gateway load

The admin dashboard's Gateway Load → Requests view reports peak throughput from completed intervals: minutes for short model/provider views, and hours for longer ranges or tenant-filtered views. Daily charts retain hourly peaks, so switching from 7 to 30 days does not smooth away a spike. Peak timestamps identify the start of the interval; the current incomplete interval is excluded.

The All / Credits / BYOK filter applies to every value on the page: requests, rates, error rates, error counts, duration, and time to first token, on every grouping and tenant filter. It defaults to Credits, so bring-your-own-key traffic is excluded unless you pick All or BYOK. After upgrading, the worker backfills the per-mode error and latency split for organization, project, and API-key views over the last 365 days (TENANT_STATS_BYOK_BACKFILL_DAYS); until it reaches an hour, that hour counts its BYOK errors and latency as credits.

Switch to Tokens for platform, model, or provider throughput over 7 or 30 days. Choose total, input, or output tokens and a billing mode. Average TPM includes quiet minutes; average tokens/day normalizes the same window to 24 hours. Peak TPM is the busiest completed UTC minute, and peak tokens/day is the busiest complete UTC calendar day. Partial days appear faded in the daily chart and do not count toward daily peaks. The table ranks all models or providers by token volume, with ten rows per page. CSV export includes every row.

Token metrics use recorded upstream usage, excluding gateway response-cache hits. Minute history is retained for 30 days. Measurements follow worker refreshes; reload the page to update them. The Tokens view does not auto-refresh. These fixed UTC intervals do not reproduce a provider's rolling rate-limit or quota accounting. Organization, project, and API-key filters apply only to the Requests view.

Support chat diagnostics

Support chat errors can arrive inside an HTTP 200 event stream. In the API logs, search for the conversation's conversationId, then follow requestId for each visitor attempt and gatewayRequestId into the gateway logs.

Chat support request finished records whether the attempt completed, failed, returned an empty answer, was rejected, was aborted, or stayed silent because the conversation was escalated. Failure events include the stage and available provider error code or HTTP status. Gateway and step events record timing, selected models, token counts, and tool-call counts without recording chat contents or raw SDK error payloads.

Sign-up email restrictions

With HOSTED=true, administrators can manage the custom email-domain blocklist in Admin → Settings → Blocked sign-up email domains. Paste domains one per line or separated by commas, then save. Entries also block subdomains; updates apply to subsequent email sign-ups without a restart. Existing users can still sign in.

The custom list starts empty and has no built-in entries. Clearing and saving it removes custom domain blocks. The separate disposable-email and plus-address checks remain active in hosted mode.

Provider base URL overrides

A provider key may carry a base URL that replaces the provider's published endpoint. Self-hosted deployments allow it; HOSTED=true rejects it, so a provider is reached only at its published endpoint with the key's own settings (such as an Azure resource) applied. Set ALLOW_PROVIDER_BASE_URL_OVERRIDE to true or false to override either default. Custom providers are unaffected — their base URL defines them.

Where the override is disallowed, a key that already stores one is skipped by routing rather than redirected to the published endpoint, whose host the key's token may not be meant for. Another key for the provider, another provider, or (in hybrid mode) credits serve the request instead, and the gateway logs the key it skipped.

How is this guide?

Last updated on

On this page

Ready for production?

Ship to production with SSO, audit logs, spend controls, and guardrails your security team will approve.

Explore Enterprise