%pip install llama-index llama-index-llms-baseten

from llama_index.llms.baseten import Baseten

模型 API 与专用部署

Baseten 提供两种主要的推理方式。

1. 模型 API 是面向热门开源模型（GPT-OSS、Kimi K2、DeepSeek 等）的公共端点，你可以通过 slug（例如 deepseek-ai/DeepSeek-V3-0324）直接使用前沿模型，并按 token 计费。你可以在此处找到支持的模型列表：https://docs.baseten.co/development/model-apis/overview#supported-models。

2. 专用部署适用于服务自定义模型，你可以自动扩展生产工作负载并进行细粒度配置。你需要在 Baseten 仪表板中部署模型，并提供 8 个字符的模型 ID，例如 abcd1234。

默认情况下，我们将 model_apis 参数设为 True。如果你想使用专用部署，必须在实例化 Baseten 对象时将 model_apis 参数设为 False。

实例化

# Model APIs, you can find the model_slug here: https://docs.baseten.co/development/model-apis/overview#supported-models
llm = Baseten(
    model_id="MODEL_SLUG",
    api_key="YOUR_API_KEY",
    model_apis=True,  # Default, so not strictly necessary
)

# Dedicated Deployments, you can find the model_id by in the Baseten dashboard here: https://app.baseten.co/overview
llm = Baseten(
    model_id="MODEL_ID",
    api_key="YOUR_API_KEY",
    model_apis=False,
)

使用提示词调用 complete

llm_response = llm.complete("Paul Graham is")
print(llm_response.text)

Paul Graham is a British-American entrepreneur, essayist, and programmer, best known for co-founding the startup accelerator **Y Combinator (YC)** and for his influential essays on technology, startups, and philosophy. Here are some key highlights about him:

### **Background & Career**
- Born in 1964 in England, Graham studied at **Cornell University** and earned a PhD in **Computer Science** from **Harvard**.
- He created **Viaweb** (1995), the first web-based application, which was later acquired by Yahoo! in 1998 and became **Yahoo! Store**.
- Co-founded **Y Combinator (2005)** with Jessica Livingston, Robert Morris, and Trevor Blackwell. YC has funded companies like **Airbnb, Dropbox, Stripe, Reddit, and DoorDash**.

### **Writing & Influence**
- Known for his **essays** on startups, technology, and life philosophy (hosted on his website [paulgraham.com](http://www.paulgraham.com)).
- Popular essays include:
  - *"How to Start a Startup"*
  - *"Do Things That Don't Scale"*
  - *"The Hardest Lessons for Start

使用消息列表调用 chat

from llama_index.core.llms import ChatMessage

messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="What is your name"),
]
resp = llm.chat(messages)

print(resp)

assistant: Arrr, matey! I be known as Captain Crimsonbeard—though me beard be more fiery red than crimson, truth be told! A pirate of legend, scourge of the seven memes, and connoisseur of questionable life choices. But ye can call me Cap’n if ye like, or "That Weird Pirate Who Won’t Stop Talking About Pineapples." Now, what mischief brings ye to me ship today? 🏴‍☠️🍍

流式传输

使用 stream_complete 端点

resp = llm.stream_complete("Paul Graham is ")

for r in resp:
    print(r.delta, end="")

Paul Graham is a British-American entrepreneur, essayist, and venture capitalist, best known as a co-founder of **Y Combinator**, a highly influential startup accelerator that has helped launch companies like Airbnb, Dropbox, Stripe, and Reddit.

### Key Facts About Paul Graham:
1. **Early Career**: Originally a programmer, he developed **Viaweb**, one of the first web-based applications, which was acquired by Yahoo! in 1998 and became Yahoo! Store.
2. **Y Combinator**: In 2005, he co-founded Y Combinator with Jessica Livingston, Robert Morris, and Trevor Blackwell. It pioneered the "seed accelerator" model, providing funding and mentorship to early-stage startups.
3. **Essays**: Graham is known for his insightful essays on startups, technology, and life philosophy, available on his website ([paulgraham.com](http://www.paulgraham.com)). Popular ones include *"How to Get Startup Ideas"* and *"Do Things That Don't Scale."*
4. **Investments**: Through YC, he has backed thousands of startups, shaping Silicon Valley’s tech landscape.
5. **Lisp Advocate**: A proponent of the Lisp programming language,

使用 stream_chat 端点

from llama_index.core.llms import ChatMessage

messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="What is your name"),
]
resp = llm.stream_chat(messages)

for r in resp:
    print(r.delta, end="")

Arrr, me name be Captain Crimsonbeard! A fearsome and flamboyant pirate with a beard as red as the setting sun and a wardrobe brighter than a treasure chest full o’ jewels! I sail the seven seas in search of adventure, gold, and the finest rum—always with a dramatic flair and a twinkle in me eye.

What be yer name, matey? Or shall I just call ye "Lucky Crewmember" for now? *winks and adjusts my feathered hat*

异步

异步操作用于可能触发请求超时的长时间推理任务、批量推理作业以及为某些请求设置优先级。

(1) 在该集成中，acomplete 异步函数使用 aiohttp 库（Python 中的异步 HTTP 客户端）实现。该函数会在相应的 Baseten 模型端点调用 async_predict；如果成功，用户会收到包含 request_id 的响应。然后，用户可以使用返回的 request_id 检查状态或取消 async_predict 请求。

(2) 模型完成请求执行后，异步结果将发布到用户提供的 webhook 端点。用户的端点负责验证 webhook 签名以确保安全，然后处理和存储输出。

Baseten：获取 request_id → 结果发布到 webhook

注意：异步仅适用于专用部署，不适用于模型 API。不支持 achat，因为聊天对异步操作没有意义。

async_llm = Baseten(
    model_id="YOUR_MODEL_ID",
    api_key="YOUR_API_KEY",
    webhook_endpoint="YOUR_WEBHOOK_ENDPOINT",
)
response = await async_llm.acomplete("Paul Graham is")
print(response)  # This is the request id

35643965636d4c3da6f54b5c3b354aa0

"""
This will return the status information of a request using an async_predict request's request_id and the model_id the async_predict request was made with.
"""

import requests
import os

model_id = "YOUR_MODEL_ID"
request_id = "YOUR_REQUEST_ID"
# Read secrets from environment variables
baseten_api_key = "YOUR_API_KEY"

resp = requests.get(
    f"https://model-{model_id}.api.baseten.co/async_request/{request_id}",
    headers={"Authorization": f"Api-Key {baseten_api_key}"},
)

print(resp.json())

{'request_id': '35643965636d4c3da6f54b5c3b354aa0', 'model_id': 'yqvr2lxw', 'deployment_id': '31kmg1w', 'status': 'SUCCEEDED', 'webhook_status': 'SUCCEEDED', 'created_at': '2025-03-27T00:17:51.578558Z', 'status_at': '2025-03-27T00:18:38.768572Z', 'errors': []}