Using the AI SDK

Generate text, stream responses, and call tools with the Vercel AI SDK and LLM Gateway

LLM Gateway ships a first-party provider for the Vercel AI SDK: @llmgateway/ai-sdk-provider. One provider instance and one API key reach every model in the catalog.

Install

pnpm add ai @llmgateway/ai-sdk-provider

Set your API key (create one from the dashboard):

export LLM_GATEWAY_API_KEY=llmgtwy_your_key_here

Generate text

import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { generateText } from "ai";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { text } = await generateText({
	model: llmgateway("openai/gpt-4o"),
	prompt: "Hello!",
});

Switching models is a one-line change — the same provider serves every model:

const { text } = await generateText({
	model: llmgateway("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Hello!",
});

Model ID formats

LLM Gateway supports two model ID formats:

Root model IDs (gpt-4o) — smart routing picks the best provider based on uptime, throughput, price, and latency
Provider-prefixed IDs (openai/gpt-4o) — routes to a specific provider with automatic failover if uptime drops below 90%

See the routing documentation for details and the models page for the full catalog.

Stream responses

import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { streamText } from "ai";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { textStream } = await streamText({
	model: llmgateway("anthropic/claude-3-5-sonnet-20241022"),
	prompt: "Write a poem about coding",
});

for await (const text of textStream) {
	process.stdout.write(text);
}

Next.js route handler

// app/api/chat/route.ts
import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { streamText } from "ai";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

export async function POST(req: Request) {
	const { messages } = await req.json();

	const result = await streamText({
		model: llmgateway("openai/gpt-4o"),
		messages,
	});

	return result.toDataStreamResponse();
}

Tool calling

import { createLLMGateway } from "@llmgateway/ai-sdk-provider";
import { generateText, tool } from "ai";
import { z } from "zod";

const llmgateway = createLLMGateway({
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { text, toolResults } = await generateText({
	model: llmgateway("openai/gpt-4o"),
	tools: {
		weather: tool({
			description: "Get the weather for a location",
			parameters: z.object({
				location: z.string(),
			}),
			execute: async ({ location }) => {
				return { temperature: 72, condition: "sunny" };
			},
		}),
	},
	prompt: "What's the weather in San Francisco?",
});

Without the provider package

If you prefer not to add a dependency, point @ai-sdk/openai at the gateway with a custom base URL:

import { createOpenAI } from "@ai-sdk/openai";
import { generateText } from "ai";

const llmgateway = createOpenAI({
	baseURL: "https://api.llmgateway.io/v1",
	apiKey: process.env.LLM_GATEWAY_API_KEY,
});

const { text } = await generateText({
	model: llmgateway("openai/gpt-4o"),
	prompt: "Hello!",
});

Every request made through the AI SDK shows up in your Activity and Usage & Metrics dashboards like any other gateway request — with per-request cost, tokens, and latency.