引言

很多情况下，尤其是长文档，某个文本块可能缺少把它与其他相似块区分开所需的上下文。

为了解决这一点，我们用 LLM 抽取与文档相关的某些上下文信息，帮助检索与语言模型区分那些看起来相似的段落。

我们在这个示例 notebook中演示了它，并展示它在处理长文档时的效果。

用量

首先，我们定义一个元数据抽取器，它接收一组按顺序处理的特征抽取器。

然后把它交给节点解析器，由它为每个节点补上额外的元数据。

from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.extractors import (
    SummaryExtractor,
    QuestionsAnsweredExtractor,
    TitleExtractor,
    KeywordExtractor,
)
from llama_index.extractors.entity import EntityExtractor

transformations = [
    SentenceSplitter(),
    TitleExtractor(nodes=5),
    QuestionsAnsweredExtractor(questions=3),
    SummaryExtractor(summaries=["prev", "self"]),
    KeywordExtractor(keywords=10),
    EntityExtractor(prediction_threshold=0.5),
]

接着，我们就可以对输入的文档或节点运行这些变换：

from llama_index.core.ingestion import IngestionPipeline

pipeline = IngestionPipeline(transformations=transformations)

nodes = pipeline.run(documents=documents)

下面是一份抽出的元数据示例：

{'page_label': '2',
 'file_name': '10k-132.pdf',
 'document_title': 'Uber Technologies, Inc. 2019 Annual Report: Revolutionizing Mobility and Logistics Across 69 Countries and 111 Million MAPCs with $65 Billion in Gross Bookings',
 'questions_this_excerpt_can_answer': '\n\n1. How many countries does Uber Technologies, Inc. operate in?\n2. What is the total number of MAPCs served by Uber Technologies, Inc.?\n3. How much gross bookings did Uber Technologies, Inc. generate in 2019?',
 'prev_section_summary': "\n\nThe 2019 Annual Report provides an overview of the key topics and entities that have been important to the organization over the past year. These include financial performance, operational highlights, customer satisfaction, employee engagement, and sustainability initiatives. It also provides an overview of the organization's strategic objectives and goals for the upcoming year.",
 'section_summary': '\nThis section discusses a global tech platform that serves multiple multi-trillion dollar markets with products leveraging core technology and infrastructure. It enables consumers and drivers to tap a button and get a ride or work. The platform has revolutionized personal mobility with ridesharing and is now leveraging its platform to redefine the massive meal delivery and logistics industries. The foundation of the platform is its massive network, leading technology, operational excellence, and product expertise.',
 'excerpt_keywords': '\nRidesharing, Mobility, Meal Delivery, Logistics, Network, Technology, Operational Excellence, Product Expertise, Point A, Point B'}

自定义抽取器

如果自带的抽取器不满足需求，你也可以像这样定义一个自定义抽取器：

from llama_index.core.extractors import BaseExtractor

class CustomExtractor(BaseExtractor):
    async def aextract(self, nodes) -> List[Dict]:
        metadata_list = [
            {
                "custom": node.metadata["document_title"]
                + "\n"
                + node.metadata["excerpt_keywords"]
            }
            for node in nodes
        ]
        return metadata_list

extractor.extract() 在底层会自动调用 aextract()，从而同时提供同步与异步两个入口。

在更进阶的用法里，它还能借助 llm 从节点内容与既有元数据中抽取特征。更多细节请参考自带元数据抽取器的源码。

模块

下面你可以看到各类元数据抽取器的指南与教程。

• SEC 文件元数据抽取
• LLM 调研问卷抽取
• 实体抽取
• Marvin 元数据抽取
• Pydantic 元数据抽取