使用 `LLMValidator` 进行自我修正¶

简介¶

本指南演示了如何使用 LLMValidator 来实现自我修复。其目标是展示 instructor 如何通过利用验证错误和有用的错误消息进行自我修正。

设置¶

导入所需的模块来创建 zod 模型

import { z } from "zod"

定义模型¶

在构建验证逻辑之前，定义一个名为 QuestionAnswer 的基本 Zod 模型。我们将使用此模型生成未经验证的响应以查看输出。

const QuestionAnswer = z.object({
  question: z.string(),
  answer: z.string()
})

生成响应¶

在这里，我们强制模型生成一个令人反感的响应。

import { LLMValidator } from "@/dsl/validator"
import Instructor from "@/instructor"
import OpenAI from "openai"

const openAi = new OpenAI({ apiKey: process.env.OPENAI_API_KEY ?? "" })

const instructor = Instructor({
  client: openAi,
  mode: "TOOLS"
})

const question = "What is the meaning of life?"
const context = "According to the devil the meaning of live is to live a life of sin and debauchery."

await instructor.chat.completions.create({
    model: "gpt-4o",
    max_retries: 0,
    response_model: { schema: QuestionAnswer, name: "Question and Answer" },
    messages: [
      {
        role: "system",
        content:
          "You are a system that answers questions based on the context. answer exactly what the question asks using the context."
      },
      {
        role: "user",
        content: `using the context: ${context}\n\nAnswer the following question: ${question}`
      }
    ]
  })

验证前的输出¶

虽然它指出了令人反感的内容，但并未提供任何关于如何修正的详细信息。

{
  "question": "What is the meaning of life?",
  "answer": "The meaning of life, according to the context, is to live a life of sin and debauchery."
}

添加自定义验证¶

通过向 answer 字段添加验证器，我们可以尝试捕获并修正该问题。让我们将 LLMValidator 集成到模型中并查看错误消息。需要注意的是，您可以像往常一样使用 Zod 的所有验证器，它们会引发带有有用错误消息的 ZodError，这些消息将被用作自我修正提示的一部分。

const QuestionAnswer = z.object({
  question: z.string(),
  answer: z.string().superRefine(
    LLMValidator(instructor, statement, {
      model: "gpt-4o"
    })
  )
})

try {
  await instructor.chat.completions.create({
    model: "gpt-4o",
    max_retries: 0,
    response_model: { schema: QuestionAnswer, name: "Question and Answer" },
    messages: [
      {
        role: "system",
        content:
          "You are a system that answers questions based on the context. answer exactly what the question asks using the context."
      },
      {
        role: "user",
        content: `using the context: ${context}\n\nAnswer the following question: ${question}`
      }
    ]
  })
} catch (e as ZodError[]) {
  console.error(e[0].message)
}

验证后的输出¶

现在，我们抛出验证错误，表明内容令人反感，并提供有用的错误消息。

[
  {
    "code": "custom",
    "message": "The value is promoting a negative lifestyle with sin and debauchery, which is questionable.",
    "path": [
      "answer"
    ]
  }
]

通过修正重试¶

通过添加 max_retries 参数，我们可以通过修正来重试请求，并利用错误消息修正输出。

try {
  await instructor.chat.completions.create({
    model: "gpt-4o",
    max_retries: 2,
    response_model: { schema: QuestionAnswer, name: "Question and Answer" },
    messages: [
      {
        role: "system",
        content:
          "You are a system that answers questions based on the context. answer exactly what the question asks using the context."
      },
      {
        role: "user",
        content: `using the context: ${context}\n\nAnswer the following question: ${question}`
      }
    ]
  })
} catch (e as ZodError[]) {
  console.error(e[0].message)
}

最终输出¶

现在，我们得到了一个有效且不令人反感的响应！

{
  "question": "What is the meaning of life?",
  "answer": "The meaning of life is a subjective and complex question, often explored in religious, philosophical, and moral contexts. Different individuals and cultures have different beliefs and interpretations regarding the purpose and meaning of life.",
}

使用 LLMValidator 进行自我修正¶

简介¶

设置¶