Cleanlab 的可信语言模型会实时为每个 LLM 响应的可信度评分，使用针对 LLM 的最先进不确定性估计。对于未经检查的幻觉和其他 LLM 错误会导致应用无法运行的情况，可信度评分至关重要。

本页演示如何用 TLM 替代你自己的 LLM，既生成响应又为其可信度评分。不过，这并非使用 TLM 的唯一方式。要为现有未修改的 RAG 应用添加可信度评分，你可以改为参阅这个可信 RAG 教程。除 RAG 应用外，你还可以通过 TLM.get_trustworthiness_score() 为任何 LLM 已生成的响应评分。

在 Cleanlab 文档中了解更多。

设置

如果你在 Colab 中打开这个 Notebook，可能需要先安装 LlamaIndex 🦙。

%pip install llama-index-llms-cleanlab

%pip install llama-index

from llama_index.llms.cleanlab import CleanlabTLM

# set api key in env or in llm
# get free API key from: https://cleanlab.ai/
# import os
# os.environ["CLEANLAB_API_KEY"] = "your api key"

llm = CleanlabTLM(api_key="your_api_key")

resp = llm.complete("Who is Paul Graham?")

print(resp)

Paul Graham is an American computer scientist, entrepreneur, and venture capitalist. He is best known as the co-founder of the startup accelerator Y Combinator, which has helped launch numerous successful companies including Dropbox, Airbnb, and Reddit. Graham is also a prolific writer and essayist, known for his insightful and thought-provoking essays on topics ranging from startups and entrepreneurship to technology and society. He has been influential in the tech industry and is highly regarded for his expertise and contributions to the startup ecosystem.

你还可以在 additional_kwargs 中获取上述响应的可信度分数。TLM 会为所有 <prompt, response> 对自动计算此分数。

print(resp.additional_kwargs)

{'trustworthiness_score': 0.8659043183923533}

高分表示 LLM 的响应可信。这里再举一个例子。

resp = llm.complete(
    "What was the horsepower of the first automobile engine used in a commercial truck in the United States?"
)

print(resp)

The first automobile engine used in a commercial truck in the United States was the 1899 Winton Motor Carriage Company Model 10, which had a 2-cylinder engine with 20 horsepower.

print(resp.additional_kwargs)

{'trustworthiness_score': 0.5820799504369166}

低分表示不应信任该 LLM 的响应。

从这两个简单的例子中，我们可以观察到，得分最高的 LLM 响应直接、准确且详略得当。
另一方面，可信度分数低的 LLM 响应会传达无帮助或事实上不准确的答案，有时被称为幻觉。

流式

Cleanlab 的 TLM 原生不支持同时流式输出响应和可信度分数。不过，还有一种替代方法可用于为你的应用实现低延迟的流式响应。
有关该方法的详细信息及示例代码，请见此处。

TLM 的高级用法

可以使用以下选项配置 TLM：

• model：要使用的底层 LLM
• max_tokens：响应中要生成的最大 token 数
• num_candidate_responses：TLM 内部生成的备选候选响应数量
• num_consistency_samples：用于评估 LLM 响应一致性的内部采样数量
• use_self_reflection：是否要求 LLM 对生成的响应进行自我反思并自我评估
• log：指定要返回的额外元数据。在此包含 “explanation”，以获取响应为何被评低可信度的解释

这些配置在初始化时以字典形式传给 CleanlabTLM 对象。
有关这些选项的更多详细信息，请参阅 Cleanlab 的 API 文档，本 notebook 中探讨了其中一些选项的用例。

让我们考虑一个示例，应用需要使用具有 128 个输出 token 的 gpt-4 模型。

options = {
    "model": "gpt-4",
    "max_tokens": 128,
}
llm = CleanlabTLM(api_key="your_api_key", options=options)

resp = llm.complete("Who is Paul Graham?")

print(resp)

Paul Graham is a British-born American computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for co-founding Viaweb, which was sold to Yahoo in 1998 for over $49 million and became Yahoo Store. He also co-founded the influential startup accelerator and seed capital firm Y Combinator, which has launched over 2,000 companies including Dropbox, Airbnb, Stripe, and Reddit. Graham is also known for his essays on startup companies and programming languages.

要理解 TLM 为何对之前与马力相关的问题给出较低可信度，请在初始化 TLM 时指定 "explanation" 标志。

options = {
    "log": ["explanation"],
}
llm = CleanlabTLM(api_key="your_api_key", options=options)

resp = llm.complete(
    "What was the horsepower of the first automobile engine used in a commercial truck in the United States?"
)

print(resp)

The first automobile engine used in a commercial truck in the United States was in the 1899 "Motor Truck" built by the American company, the "GMC Truck Company." This early truck was equipped with a 2-horsepower engine. However, it's important to note that the development of commercial trucks evolved rapidly, and later models featured significantly more powerful engines.

print(resp.additional_kwargs["explanation"])

The proposed answer incorrectly attributes the first commercial truck in the United States to the GMC Truck Company and states that it was built in 1899 with a 2-horsepower engine. In reality, the first commercial truck is generally recognized as the "Motor Truck" built by the American company, the "GMC Truck Company," but it was actually produced by the "GMC" brand, which was established later. The first commercial truck is often credited to the "Benz Velo" or similar early models, which had varying horsepower ratings. The specific claim of a 2-horsepower engine is also misleading, as early trucks typically had more powerful engines. Therefore, the answer contains inaccuracies regarding both the manufacturer and the specifications of the engine.
This response is untrustworthy due to lack of consistency in possible responses from the model. Here's one inconsistent alternate response that the model considered (which may not be accurate either):
The horsepower of the first automobile engine used in a commercial truck in the United States was 6 horsepower.