Braintrust

面向 CrewAI 的 Braintrust 集成，支持 OpenTelemetry 追踪与评估

Braintrust 集成

本指南演示如何借助 OpenTelemetry 将 Braintrust 与 CrewAI 集成，以实现全面的追踪与评估。完成本指南后，你将能够追踪 CrewAI 智能体、监控其性能，并使用 Braintrust 强大的可观测性平台评估其输出。

什么是 Braintrust？ Braintrust 是一个 AI 评估与可观测性平台，为 AI 应用提供全面的追踪、评估和监控，并内置实验跟踪与性能分析。

开始使用

我们将通过一个简单示例，演示如何使用 CrewAI，并通过 OpenTelemetry 将其与 Braintrust 集成，以实现全面的可观测性与评估。

第 1 步：安装依赖项

uv add braintrust[otel] crewai crewai-tools opentelemetry-instrumentation-openai opentelemetry-instrumentation-crewai python-dotenv

第 2 步：设置环境变量

设置 Braintrust API 密钥，并配置 OpenTelemetry 以便将追踪发送到 Braintrust。你需要 Braintrust API 密钥和 OpenAI API 密钥。

import os
from getpass import getpass

# Get your Braintrust credentials
BRAINTRUST_API_KEY = getpass("🔑 Enter your Braintrust API Key: ")

# Get API keys for services
OPENAI_API_KEY = getpass("🔑 Enter your OpenAI API key: ")

# Set environment variables
os.environ["BRAINTRUST_API_KEY"] = BRAINTRUST_API_KEY
os.environ["BRAINTRUST_PARENT"] = "project_name:crewai-demo"
os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY

第 3 步：使用 Braintrust 初始化 OpenTelemetry

初始化 Braintrust OpenTelemetry 插桩，开始捕获追踪并将其发送到 Braintrust。

import os
from typing import Any, Dict

from braintrust.otel import BraintrustSpanProcessor
from crewai import Agent, Crew, Task
from crewai.llm import LLM
from opentelemetry import trace
from opentelemetry.instrumentation.crewai import CrewAIInstrumentor
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.sdk.trace import TracerProvider

def setup_tracing() -> None:
    """Setup OpenTelemetry tracing with Braintrust."""
    current_provider = trace.get_tracer_provider()
    if isinstance(current_provider, TracerProvider):
        provider = current_provider
    else:
        provider = TracerProvider()
        trace.set_tracer_provider(provider)

    provider.add_span_processor(BraintrustSpanProcessor())
    CrewAIInstrumentor().instrument(tracer_provider=provider)
    OpenAIInstrumentor().instrument(tracer_provider=provider)

setup_tracing()

第 4 步：创建 CrewAI 应用

我们将创建一个 CrewAI 应用，让两个智能体协作研究并撰写一篇关于 AI 进展的博客文章，并启用全面的追踪。

from crewai import Agent, Crew, Process, Task
from crewai_tools import SerperDevTool

def create_crew() -> Crew:
    """Create a crew with multiple agents for comprehensive tracing."""
    llm = LLM(model="gpt-4o-mini")
    search_tool = SerperDevTool()

    # Define agents with specific roles
    researcher = Agent(
        role="Senior Research Analyst",
        goal="Uncover cutting-edge developments in AI and data science",
        backstory="""You work at a leading tech think tank.
        Your expertise lies in identifying emerging trends.
        You have a knack for dissecting complex data and presenting actionable insights.""",
        verbose=True,
        allow_delegation=False,
        llm=llm,
        tools=[search_tool],
    )

    writer = Agent(
        role="Tech Content Strategist",
        goal="Craft compelling content on tech advancements",
        backstory="""You are a renowned Content Strategist, known for your insightful and engaging articles.
        You transform complex concepts into compelling narratives.""",
        verbose=True,
        allow_delegation=True,
        llm=llm,
    )

    # Create tasks for your agents
    research_task = Task(
        description="""Conduct a comprehensive analysis of the latest advancements in {topic}.
        Identify key trends, breakthrough technologies, and potential industry impacts.""",
        expected_output="Full analysis report in bullet points",
        agent=researcher,
    )

    writing_task = Task(
        description="""Using the insights provided, develop an engaging blog
        post that highlights the most significant {topic} advancements.
        Your post should be informative yet accessible, catering to a tech-savvy audience.
        Make it sound cool, avoid complex words so it doesn't sound like AI.""",
        expected_output="Full blog post of at least 4 paragraphs",
        agent=writer,
        context=[research_task],
    )

    # Instantiate your crew with a sequential process
    crew = Crew(
        agents=[researcher, writer], 
        tasks=[research_task, writing_task], 
        verbose=True, 
        process=Process.sequential
    )

    return crew

def run_crew():
    """Run the crew and return results."""
    crew = create_crew()
    result = crew.kickoff(inputs={"topic": "AI developments"})
    return result

# Run your crew
if __name__ == "__main__":
    # Instrumentation is already initialized above in this module
    result = run_crew()
    print(result)

第 5 步：在 Braintrust 中查看追踪

运行你的团队后，你可以从不同视角在 Braintrust 中查看全面的追踪：

第 6 步：通过 SDK 进行评估（实验）

你也可以使用 Braintrust 的 Eval SDK 运行评估。这适用于比较版本或离线给输出打分。下面是一个 Python 示例，使用 Eval 类与我们上面创建的团队：

# eval_crew.py
from braintrust import Eval
from autoevals import Levenshtein

def evaluate_crew_task(input_data):
    """Task function that wraps our crew for evaluation."""
    crew = create_crew()
    result = crew.kickoff(inputs={"topic": input_data["topic"]})
    return str(result)

Eval(
    "AI Research Crew",  # Project name
    {
        "data": lambda: [
            {"topic": "artificial intelligence trends 2024"},
            {"topic": "machine learning breakthroughs"},
            {"topic": "AI ethics and governance"},
        ],
        "task": evaluate_crew_task,
        "scores": [Levenshtein],
    },
)

设置你的 API 密钥并运行：

export BRAINTRUST_API_KEY="YOUR_API_KEY"
braintrust eval eval_crew.py

更多详情请参阅 Braintrust Eval SDK 指南。

Braintrust 集成的关键特性

• 全面追踪：跟踪所有智能体交互、工具使用和 LLM 调用
• 性能监控：监控执行时间、token 用量和成功率
• 实验跟踪：比较不同的团队配置和模型
• 自动化评估：为团队输出设置自定义评估指标
• 错误跟踪：监控并调试团队执行过程中的失败
• 成本分析：跟踪 token 用量及相关成本

版本兼容性信息

• Python 3.8+
• CrewAI >= 0.86.0
• Braintrust >= 0.1.0
• OpenTelemetry SDK >= 1.31.0

参考

• Braintrust 文档 - Braintrust 平台概览
• Braintrust CrewAI 集成 - 官方 CrewAI 集成指南
• Braintrust Eval SDK - 通过 SDK 运行实验
• CrewAI 文档 - CrewAI 框架概览
• OpenTelemetry 文档 - OpenTelemetry 指南
• Braintrust GitHub - Braintrust SDK 的源代码