AI Gateway 是 Vercel 提供的代理服务，可将模型请求路由到各种 AI 提供商。它提供面向多个提供商的统一 API，并让你能够设置预算、监控用量、对请求进行负载均衡以及管理回退。你可以在其文档中了解更多。

如果你在 Colab 中打开这个 Notebook，可能需要先安装 LlamaIndex 🦙。

%pip install llama-index-llms-vercel-ai-gateway

!pip install llama-index

from llama_index.llms.vercel_ai_gateway import VercelAIGateway
from llama_index.core.llms import ChatMessage

llm = VercelAIGateway(
    model="anthropic/claude-4-sonnet",
    max_tokens=64000,
    context_window=200000,
    api_key="your-api-key",
    default_headers={
        "http-referer": "https://myapp.com/",  # Optional: Your app URL
        "x-title": "My App",  # Optional: Your app name
    },
)

print(llm.model)

使用 ChatMessage 列表调用 chat

你需要设置环境变量 VERCEL_AI_GATEWAY_API_KEY 或 VERCEL_OIDC_TOKEN，或者在类构造函数中设置 api_key

# import os
# os.environ['VERCEL_AI_GATEWAY_API_KEY'] = '<your-api-key>'

llm = VercelAIGateway(
    api_key="pBiuCWfswZCDxt8D50DSoBfU",
    max_tokens=64000,
    context_window=200000,
    model="anthropic/claude-4-sonnet",
)

message = ChatMessage(role="user", content="Tell me a joke")
resp = llm.chat([message])
print(resp)

流式

message = ChatMessage(role="user", content="Tell me a story in 250 words")
resp = llm.stream_chat([message])
for r in resp:
    print(r.delta, end="")

使用提示词调用 complete

resp = llm.complete("Tell me a joke")
print(resp)

resp = llm.stream_complete("Tell me a story in 250 words")
for r in resp:
    print(r.delta, end="")

模型配置

# This example uses Anthropic's Claude 4 Sonnet (models are specified as `provider/model`):
llm = VercelAIGateway(
    model="anthropic/claude-4-sonnet",
    api_key="pBiuCWfswZCDxt8D50DSoBfU",
)

resp = llm.complete("Write a story about a dragon who can code in Rust")
print(resp)