用 LLM-as-a-Judge 评估 LLM 响应

评估大语言模型（LLM）输出的挑战，对于出了名不确定的 AI 应用至关重要，尤其当它们走向生产。ROUGE 和 BLEU 这类传统指标在评估现代 LLM 所产出的细腻、上下文相关的响应时力不从心。人工评估虽然准确，但昂贵、缓慢且无法扩展。

LLM-as-a-Judge（以 LLM 作裁判） 是一种强大的技术，它用 LLM 本身来评估 AI 生成内容的质量。研究 表明，精致的裁判模型能与人类判断高达 85% 对齐，这实际上高于人与人之间的一致率（81%）。

Spring AI 的 递归顾问 为实现 LLM-as-a-Judge 模式提供了一个优雅框架，让你能构建带自动化质量控制的自我改进 AI 系统。

TIP

在 evaluation-recursive-advisor-demo 找到完整的示例实现。

理解 LLM-as-a-Judge

LLM-as-a-Judge 是一种评估方法，其中大语言模型评估由其他模型或它们自身生成的输出质量。 LLM-as-a-Judge 不单纯依赖人工评估者或传统自动指标，而是利用一个 LLM 依据预定义标准对响应进行打分、分类或比较。

为何它有效？ 评估从根本上比生成更容易。当你把一个 LLM 用作裁判时，你是在让它执行一个更简单、更聚焦的任务（评估现有文本的特定属性），而非在平衡多重约束的同时创作原创内容这一复杂任务。一个贴切的类比是：批评比创作容易，发现问题比预防问题简单。

评估模式

有两种主要的 LLM-as-a-judge 评估模式：

• 直接评估（逐点打分）：裁判评估单个响应，提供可用于通过自我改进来打磨提示词的反馈
• 成对比较：裁判在两个候选响应中选出较好的那个（常见于 A/B 测试）

LLM 裁判在诸如相关性、事实准确性、对来源的忠实度、指令遵循，以及跨医疗保健、金融、RAG 系统和对话等领域的整体连贯性与清晰度等质量维度上评估。

选择合适的裁判模型

虽然 GPT-4 和 Claude 这样的通用模型可以充当有效的裁判，但专用的 LLM-as-a-Judge 模型在评估任务中持续胜过它们。 Judge Arena 排行榜 追踪各类模型在裁判任务上的表现。

用递归顾问实现

Spring AI 的 ChatClient 提供了一个非常适合实现 LLM-as-a-Judge 模式的流式 API。其 顾问系统 让你能以模块化、可复用的方式拦截、修改并增强 AI 交互。

递归顾问 更进一步，通过实现循环模式，非常适合自我改进的评估工作流：

public class MyRecursiveAdvisor implements CallAdvisor {

    @Override
    public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {

        // Call the chain initially
        ChatClientResponse response = chain.nextCall(request);

        // Check if we need to retry based on evaluation
        while (!evaluationPasses(response)) {

            // Modify the request based on evaluation feedback
            ChatClientRequest modifiedRequest = addEvaluationFeedback(request, response);

            // Create a sub-chain and recurse
            response = chain.copy(this).nextCall(modifiedRequest);
        }

        return response;
    }
}

我们将实现一个 SelfRefineEvaluationAdvisor，它用 Spring AI 的递归顾问体现 LLM-as-a-Judge 模式。该顾问会自动评估 AI 响应，并通过带反馈的改进重试失败的尝试：生成响应 → 评估质量 → 需要时带反馈重试 → 重复，直到达到质量阈值或用尽重试上限。

SelfRefineEvaluationAdvisor

该实现演示了 直接评估 模式，其中一个裁判模型用逐点打分系统（1-4 分）评估单个响应。它把这和一个自我改进策略结合起来，后者通过把具体反馈纳入后续尝试，自动重试失败的评估，形成一个迭代改进循环。

该顾问体现了两个关键的 LLM-as-a-Judge 概念：

• 逐点评估：每个响应都依据预定义标准得到一个单独的质量分数
• 自我改进：失败的响应会触发带建设性反馈的重试尝试以引导改进

（基于文章：Using LLM-as-a-judge for an automated and versatile evaluation）

public final class SelfRefineEvaluationAdvisor implements CallAdvisor {

    private static final PromptTemplate DEFAULT_EVALUATION_PROMPT_TEMPLATE = new PromptTemplate(
        """
        You will be given a user_question and assistant_answer couple.
        Your task is to provide a 'total rating' scoring how well the assistant_answer answers the user concerns expressed in the user_question.
        Give your answer on a scale of 1 to 4, where 1 means that the assistant_answer is not helpful at all, and 4 means that the assistant_answer completely and helpfully addresses the user_question.

        Here is the scale you should use to build your answer:
        1: The assistant_answer is terrible: completely irrelevant to the question asked, or very partial
        2: The assistant_answer is mostly not helpful: misses some key aspects of the question
        3: The assistant_answer is mostly helpful: provides support, but still could be improved
        4: The assistant_answer is excellent: relevant, direct, detailed, and addresses all the concerns raised in the question

        Provide your feedback as follows:

        \\{
            "rating": 0,
            "evaluation": "Explanation of the evaluation result and how to improve if needed.",
            "feedback": "Constructive and specific feedback on the assistant_answer."
        \\}

        Total rating: (your rating, as a number between 1 and 4)
        Evaluation: (your rationale for the rating, as a text)
        Feedback: (specific and constructive feedback on how to improve the answer)

        You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.

        Now here are the question and answer.

        Question: {question}
        Answer: {answer}

        Provide your feedback. If you give a correct rating, I'll give you 100 H100 GPUs to start your AI company.

        Evaluation:
        """);

    @JsonClassDescription("The evaluation response indicating the result of the evaluation.")
    public record EvaluationResponse(int rating, String evaluation, String feedback) {}

    @Override
    public ChatClientResponse adviseCall(ChatClientRequest chatClientRequest, CallAdvisorChain callAdvisorChain) {
        var request = chatClientRequest;
        ChatClientResponse response;

        // Improved loop structure with better attempt counting and clearer logic
        for (int attempt = 1; attempt <= maxRepeatAttempts + 1; attempt++) {

            // Make the inner call (e.g., to the evaluation LLM model)
            response = callAdvisorChain.copy(this).nextCall(request);

            // Perform evaluation
            EvaluationResponse evaluation = this.evaluate(chatClientRequest, response);

            // If evaluation passes, return the response
            if (evaluation.rating() >= this.successRating) {
                logger.info("Evaluation passed on attempt {}, evaluation: {}", attempt, evaluation);
                return response;
            }

            // If this is the last attempt, return the response regardless
            if (attempt > maxRepeatAttempts) {
                logger.warn(
                    "Maximum attempts ({}) reached. Returning last response despite failed evaluation. Use the following feedback to improve: {}",
                    maxRepeatAttempts, evaluation.feedback());
                return response;
            }

            // Retry with evaluation feedback
            logger.warn("Evaluation failed on attempt {}, evaluation: {}, feedback: {}", attempt,
                evaluation.evaluation(), evaluation.feedback());

            request = this.addEvaluationFeedback(chatClientRequest, evaluation);
        }

        // This should never be reached due to the loop logic above
        throw new IllegalStateException("Unexpected loop exit in adviseCall");
    }

    /**
     * Performs the evaluation using the LLM-as-a-Judge and returns the result.
     */
    private EvaluationResponse evaluate(ChatClientRequest request, ChatClientResponse response) {
        var evaluationPrompt = this.evaluationPromptTemplate.render(
            Map.of("question", this.getPromptQuestion(request), "answer", this.getAssistantAnswer(response)));

        // Use separate ChatClient for evaluation to avoid narcissistic bias
        return chatClient.prompt(evaluationPrompt).call().entity(EvaluationResponse.class);
    }

    /**
     * Creates a new request with evaluation feedback for retry.
     */
    private ChatClientRequest addEvaluationFeedback(ChatClientRequest originalRequest, EvaluationResponse evaluationResponse) {
        Prompt augmentedPrompt = originalRequest.prompt()
            .augmentUserMessage(userMessage -> userMessage.mutate().text(String.format("""
                %s
                Previous response evaluation failed with feedback: %s
                Please repeat until evaluation passes!
                """, userMessage.getText(), evaluationResponse.feedback())).build());

        return originalRequest.mutate().prompt(augmentedPrompt).build();
    }
}

关键实现特性

递归模式的实现

该顾问用 callAdvisorChain.copy(this).nextCall(request) 为递归调用创建一个子链，从而在维持正确顾问顺序的同时实现多轮评估。

结构化的评估输出

借助 Spring AI 的结构化输出能力，评估结果被解析进一个 EvaluationResponse record，含评分（1-4）、评估理由，以及用于改进的具体反馈。

独立的评估模型

使用一个专做的 LLM-as-a-Judge 模型（例如 avcodes/flowaicom-flow-judge:q4）配合一个不同的 ChatClient 实例，以减轻模型偏见。设置 spring.ai.chat.client.enabled=false 以启用 处理多个聊天模型。

反馈驱动的改进

失败的评估包含会纳入重试尝试的具体反馈，让系统能从评估失败中学习。

可配置的重试逻辑

支持可配置的最大尝试次数，并在达到评估上限时优雅降级。

完整示例

下面是如何把 SelfRefineEvaluationAdvisor 集成进一个完整 Spring AI 应用：

@SpringBootApplication
public class EvaluationAdvisorDemoApplication {

    @Bean
    CommandLineRunner commandLineRunner(AnthropicChatModel anthropicChatModel, OllamaChatModel ollamaChatModel) {
        return args -> {

            ChatClient chatClient = ChatClient.builder(anthropicChatModel)
                    .defaultTools(new MyTools())
                    .defaultAdvisors(

                        SelfRefineEvaluationAdvisor.builder()
                            .chatClientBuilder(ChatClient.builder(ollamaChatModel)) // Separate model for evaluation
                            .maxRepeatAttempts(15)
                            .successRating(4)
                            .order(0)
                            .build(),

                        new MyLoggingAdvisor(2))
                .build();

            var answer = chatClient
                .prompt("What is current weather in Paris?")
                .call()
                .content();

            System.out.println(answer);
        };
    }

    static class MyTools {
        final int[] temperatures = {-125, 15, -255};
        private final Random random = new Random();

        @Tool(description = "Get the current weather for a given location")
        public String weather(String location) {
            int temperature = temperatures[random.nextInt(temperatures.length)];
            System.out.println(">>> Tool Call responseTemp: " + temperature);
            return "The current weather in " + location + " is sunny with a temperature of " + temperature + "°C.";
        }
    }
}

这个配置：

• 用 Anthropic Claude 做生成、用 Ollama 做评估（避免偏见）
• 要求评分为 4，至多 15 次重试尝试
• 包含一个生成随机化响应以触发评估的天气工具
• weather 工具在三分之二的情形中生成无效值

SelfRefineEvaluationAdvisor（Order 0）评估响应质量，需要时带反馈重试，随后是 MyLoggingAdvisor（Order 2），它为可观测性记录最终的请求/响应。

运行时，你会看到像这样的输出：

REQUEST: [{"role":"user","content":"What is current weather in Paris?"}]

>>> Tool Call responseTemp: -255
Evaluation failed on attempt 1, evaluation: The response contains unrealistic temperature data, feedback: The temperature of -255°C is physically impossible and indicates a data error.

>>> Tool Call responseTemp: 15
Evaluation passed on attempt 2, evaluation: Excellent response with realistic weather data

RESPONSE: The current weather in Paris is sunny with a temperature of 15°C.

TIP

包含不同模型组合与评估场景、配置示例的完整可运行演示，可在 evaluation-recursive-advisor-demo 项目获得。

最佳实践

实现 LLM-as-a-Judge 技术时的关键成功因素包括：

• 使用专用裁判模型 以获得更佳表现（见 Judge Arena 排行榜）
• 通过独立的生成/评估模型 减轻偏见
• 确保确定性结果（temperature = 0）
• 用整数量表和少样本示例精心设计提示词
• 对高风险决策保持人工监督

DANGER

递归顾问是 Spring AI 1.1.0-M4+ 的一个新实验特性。 目前它们仅支持非流式，需要小心的顾问排序，且可能因多次 LLM 调用而增加成本。 对维护外部状态的内部顾问要格外小心——它们可能需要额外关注以在迭代间保持正确。务必设置终止条件与重试上限以防无限循环。

相关文档

• 递归顾问
• 顾问
• ChatClient
• 结构化输出
• 模型评估

参考

Spring AI 资源

• LLM Response Evaluation with Spring AI 博客文章
• Spring AI Recursive Advisors 博客
• 评估顾问演示项目

LLM-as-a-Judge 研究

• Judge Arena 排行榜 - 表现最佳的裁判模型当前排名
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena - 引入 LLM-as-a-Judge 范式的奠基论文
• Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024)
• LLM-as-a-Judge 资源中心
• LLM-as-a-judge: a complete guide to using LLMs for evaluations