追踪 LLM 应用教程

在原型设计、beta 测试和生产各阶段，为一个 LLM 应用添加 LangSmith 可观测性。

在本教程中，你将使用检索增强生成（RAG）构建一个客户支持聊天机器人，并在从早期原型到生产的每个开发阶段添加 LangSmith 可观测性。

到最后，你将知道如何：

追踪单个 LLM 调用和完整的应用流水线。
收集并查询用户反馈。
记录元数据并将其用于过滤和 A/B 测试。
使用监控仪表板跟踪生产性能。

该应用会检索相关的文档片段并用它们来回答用户问题。检索器在本教程中是被模拟的；在真实应用中你会把它替换为一个向量搜索或类似的东西。

前置条件

开始之前，请确保你拥有：

一个 LangSmith 账户：在 smith.langchain.com 注册或登录。
一个 LangSmith API 密钥：遵循创建 API 密钥指南。
一个 OpenAI API 密钥：从 OpenAI 仪表板生成它。
LangSmith CLI（可选）：安装它以便从终端检查追踪。有关说明，参阅 LangSmith CLI。

安装所需的包：

PythonTypeScript

bash

pip install langsmith openai

bash

npm install langsmith openai
npm install -D typescript tsx

原型设计

从一开始就设置好可观测性让你能更快地迭代。你可以确切地看到正在发送给模型的内容、返回的内容，以及时间花在哪里，而无需添加 print 语句或运行调试器。

设置你的环境

在你的 shell 中设置以下环境变量：

bash

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

要把追踪发送到特定项目，使用 LANGSMITH_PROJECT 环境变量。如果未设置，LangSmith 会在追踪摄取时自动创建一个默认追踪项目。

NOTE

你可能会在其他地方看到这些变量被引用为 LANGCHAIN_*。两者都有效，但 LANGSMITH_TRACING 和 LANGSMITH_API_KEY 是推荐的名称。

追踪 LLM 调用

首先追踪你的 OpenAI 调用，即模型真正被调用的地方。这让你能立即看到你的应用发送的提示以及模型返回的响应。

用 wrap_openai（Python）或 wrapOpenAI（TypeScript）包裹 OpenAI 客户端。创建一个名为 app.py（或 app.ts）的文件，包含以下代码：

PythonTypeScript

python

from openai import OpenAI
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

typescript

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

async function supportBot(question: string): Promise<string> {
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
}

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

调用 support_bot("How many users can I have on the Starter plan?") 会产生一条 OpenAI 调用的追踪。

追踪整个流水线

追踪 LLM 调用很有用，但追踪整个流水线（包括检索）能给你应用行为的完整概览。把 @traceable（Python）或 traceable（TypeScript）加到主函数上：

PythonTypeScript

python

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

typescript

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable } from "langsmith/traceable";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

const supportBot = traceable(async function supportBot(question: string): Promise<string> {  
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});  

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

调用 support_bot("How many users can I have on the Starter plan?") 现在会产生一条完整 RAG 流水线的追踪。

LangSmith UI showing a trace with an outer application span and a nested LLM call span.

从终端检查你的追踪

如果你安装了 LangSmith CLI，无需打开 UI 就能列出你项目最近的追踪：

bash

langsmith trace list --project <your-project> --limit 5

要查看某条特定追踪的完整运行层级和输入/输出：

bash

langsmith trace get <trace-id> --full

Beta 测试

一旦你的应用在原型设计中表现良好，你就把它发布给一小群真实用户。在此阶段，你往往并不确切知道用户会如何与你的应用交互，所以需要更丰富的可观测性。你想理解的不只是应用做了什么，还有用户如何回应它。

收集反馈

把用户反馈链接到特定追踪，让你能识别哪些响应有帮助或没帮助。更新上一步的 app.py（或 app.ts），为每次调用添加一个运行 ID 并在之后附加一个分数：

PythonTypeScript

python

import os  

from openai import OpenAI
from langsmith import traceable, Client, uuid7  
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    run_id = str(uuid7())  
    support_bot(  
        "How many users can I have on the Starter plan?",  
        langsmith_extra={"run_id": run_id},  
    )  
    ls_client = Client()  
    # Feedback requires the UUID of the tracing project that owns the run  #
    project_name = os.environ.get("LANGSMITH_PROJECT", "default")  
    session_id = ls_client.create_project(project_name=project_name, upsert=True).id  
    ls_client.create_feedback(  
        run_id, key="user-score", score=1.0, session_id=session_id  
    )

typescript

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable, getCurrentRunTree } from "langsmith/traceable"; 
import { Client } from "langsmith"; 

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

let capturedRunId: string; 
let capturedProjectName: string; 

const supportBot = traceable(async function supportBot(question: string): Promise<string> {
    const runTree = getCurrentRunTree(); 
    capturedRunId = runTree.id; 
    capturedProjectName = runTree.project_name; 
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});

(async () => {
    await supportBot("How many users can I have on the Starter plan?"); 
    const lsClient = new Client(); 
    // Feedback requires the UUID of the tracing project that owns the run
    const { id: sessionId } = await lsClient.createProject({ 
        projectName: capturedProjectName, 
        upsert: true, 
    }); 
    await lsClient.createFeedback({ 
        runId: capturedRunId, 
        sessionId, 
        key: "user-score", 
        score: 1.0, 
    }); 
    await lsClient.flush(); 
})();

NOTE

在生产中，这两部分会位于分开的位置：带 run_id 的 support_bot 调用留在你的应用里，而 create_feedback 移到接收用户反馈的那个端点（例如一个 /feedback API 路由）。run_id 从一处传给另一处，以便反馈能被链接到正确的追踪。由于反馈还需要项目 UUID，请与 run_id 一起传入 session_id。

当你在 UI 中检查该运行时，反馈会出现在 Feedback 标签里。然后你可以使用 Runs 表格中的过滤控件按反馈分数过滤运行。

记录元数据

元数据让你能用一些对过滤和比较有价值的属性给运行打标签。例如，使用了哪个模型版本，或哪个用户发出了请求。

以下示例同时追踪检索器（带 run_type="retriever"）和主函数（带一个用于模型名的 metadata 属性）：

PythonTypeScript

python

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

@traceable(run_type="retriever")  
def retriever(query: str) -> list[str]:
    return docs

@traceable(metadata={"llm": "gpt-5.4-mini"})  
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    support_bot("How many users can I have on the Starter plan?")

typescript

import OpenAI from "openai";
import { wrapOpenAI } from "langsmith/wrappers";
import { traceable } from "langsmith/traceable";

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

const retriever = traceable(  
    function retriever(query: string): string[] {  
        return docs;  
    },  
    { run_type: "retriever" }  
);  

const supportBot = traceable(
    async function supportBot(question: string): Promise<string> {
        const context = await retriever(question);
        const systemMessage =
            "You are a helpful customer support agent. " +
            "Answer using only the information provided below:\n\n" +
            context.join("\n");
        const response = await client.chat.completions.create({
            model: "gpt-5.4-mini",
            messages: [
                { role: "system", content: systemMessage },
                { role: "user", content: question },
            ],
        });
        return response.choices[0].message?.content ?? "";
    },
    { metadata: { llm: "gpt-5.4-mini" } }  
);

(async () => {
    await supportBot("How many users can I have on the Starter plan?");
})();

两个元数据值都会出现在追踪上。你可以使用 Runs 表格中的过滤控件按元数据过滤运行。

生产

在有稳固的可观测性之后，你可以有信心地发布到生产。在生产中，你有多得多的流量，无法逐条检查每个追踪。LangSmith 提供监控工具帮助你理解聚合行为，并在出现问题时深入排查。

监控

在 UI 侧边栏中，选择 Monitoring，然后在左上角的下拉菜单中选择一个追踪项目。图表显示该项目随时间的关键指标，包括追踪计数、延迟、错误率、反馈分数和成本。有关可用指标和图表配置的更多信息，参阅仪表板。

LangSmith UI showing the monitoring page with the trace count chart and available tabs.

A/B 测试

NOTE

分组功能要求对给定元数据键至少有两个不同的值。

由于你一直在记录 llm 元数据属性，你可以按该属性对监控图表分组，以随时间比较模型性能。从 UI 侧边栏的 Monitoring，点击左上角的 Group by，从下拉菜单选择 Metadata，然后选择 llm。图表会更新以显示按该属性分组的结果。有关分组和自定义图表的更多信息，参阅仪表板。

下钻

当某个监控图表显示出意外情况时，点击一个数据点以冻结提示框，然后点击指标名（例如 Input）以跳转到该时间窗口的已过滤运行表格。有关搜索和过滤运行的更多信息，参阅过滤追踪。

LangSmith UI showing the monitoring page with a specific point on the Input Tokens chart highlighted.

结论

在本教程中，你为一个应用在其完整开发生命周期中添加了 LangSmith 可观测性。帮助你在原型设计期间快速迭代的同一套追踪设置，在生产中会持续提供价值。你将拥有对单个追踪和聚合性能趋势的可见性。

更多信息，参阅：

可观测性概念：术语和核心思想。
追踪集成：LangChain、LangGraph、Anthropic 及其他提供商。
自动化：在你的追踪上自动运行的规则和在线评估。

数据集

创建数据集

运行评估

评估技术

定义评估目标

打分方法

实验配置

多模态评估

分析实验结果

微调

教程

常见数据类型

评估器类型

UI

SDK

框架与集成

改进评估器

反馈

追踪设置

集成

手动埋点

配置与故障排除

项目与环境设置

高级追踪技术

数据与隐私

故障排除指南

查看与管理追踪

批量导出追踪数据

轨迹视图

数据类型参考

监控与告警

在线评估器

自动化

开发你的应用

设置依赖

持久化

能力

助手

运行

并发消息处理

如何构建

鉴权与访问控制

自定义鉴权教程

服务器定制

静态加密

替换内置后端

扩展 HTTP 服务器

请求头与日志

部署指南

参考

配置

参考

提示词

创建与管理提示词

连接模型

上下文中枢

教程

全栈 Web 应用

Agent Server API

控制平面 API

计费与用量

参考

按云服务商起步

用 Terraform 部署

AWS

GCP

Azure

安装实例

配置实例

连接外部服务

平台鉴权与访问控制

管理实例

自托管可观测性

混合部署

脚本

参考

组织