AWS

This guide covers a production deployment of LLM Gateway on AWS using EKS for the application services and managed AWS services for the backing stores.

Architecture

What to configure

1. PostgreSQL — Amazon RDS

Create an RDS for PostgreSQL instance in a private subnet. Enable automated backups and Multi-AZ for high availability. Note the connection string for DATABASE_URL.

2. Redis — Amazon ElastiCache

Create an ElastiCache for Redis cluster in the same VPC. Use it for REDIS_URL. A single-node cluster is fine to start; enable replication for production.

3. Compute — Amazon EKS

Create an EKS cluster with a managed node group sized for your traffic. Install the AWS Load Balancer Controller to expose the gateway through an Application Load Balancer.

4. Networking

Run RDS and ElastiCache in private subnets and allow inbound traffic only from the EKS node security group. Expose only the gateway (and optionally the UI) to the internet through the load balancer.

5. Secrets — AWS Secrets Manager

Store AUTH_SECRET, GATEWAY_API_KEY_HASH_SECRET, and your provider API keys in Secrets Manager. Sync them into the cluster with the External Secrets Operator or the AWS Secrets and Configuration Provider (ASCP).

Deploy the Helm chart

With the backing services in place, deploy LLM Gateway with the Helm chart, pointing it at your RDS and ElastiCache endpoints:

helm install llmgateway oci://ghcr.io/theopenco/charts/llmgateway -f values.yaml

config:
  DATABASE_URL: "postgres://user:password@your-rds-endpoint:5432/llmgateway"
  REDIS_URL: "redis://your-elasticache-endpoint:6379"
  AUTH_SECRET: "from-secrets-manager"
  GATEWAY_API_KEY_HASH_SECRET: "from-secrets-manager"

See the Kubernetes guide for the full set of configurable values and how to scale the gateway.