Google Cloud

This guide covers a production deployment of LLM Gateway on Google Cloud using GKE for the application services and managed Google Cloud services for the backing stores.

Architecture

What to configure

1. PostgreSQL — Cloud SQL

Create a Cloud SQL for PostgreSQL instance with a private IP. Enable automated backups and high availability. Note the connection details for DATABASE_URL.

2. Redis — Memorystore

Create a Memorystore for Redis instance in the same VPC and use its endpoint for REDIS_URL. Enable a read replica for production.

3. Compute — GKE

Create a GKE cluster (Autopilot or Standard) sized for your traffic. GKE provisions an HTTP(S) load balancer automatically when you create an Ingress for the gateway.

4. Networking

Use a private VPC and connect Cloud SQL via Private Service Access and Memorystore via its private endpoint. Restrict access so only the GKE workloads can reach the database and cache, and expose only the gateway to the internet.

5. Secrets — Secret Manager

Store AUTH_SECRET, GATEWAY_API_KEY_HASH_SECRET, and your provider API keys in Secret Manager. Sync them into the cluster with the External Secrets Operator or the Secret Manager CSI driver.

Deploy the Helm chart

With the backing services in place, deploy LLM Gateway with the Helm chart, pointing it at your Cloud SQL and Memorystore endpoints:

helm install llmgateway oci://ghcr.io/theopenco/charts/llmgateway -f values.yaml

config:
  DATABASE_URL: "postgres://user:password@your-cloud-sql-ip:5432/llmgateway"
  REDIS_URL: "redis://your-memorystore-ip:6379"
  AUTH_SECRET: "from-secret-manager"
  GATEWAY_API_KEY_HASH_SECRET: "from-secret-manager"

See the Kubernetes guide for the full set of configurable values and how to scale the gateway.