主题
你可以用我们的 Metadata Extractor 模块,让 LLM 自动完成元数据抽取。
我们的元数据抽取模块包含以下几种“特征抽取器”:
SummaryExtractor—— 自动为一组 Nodes 抽取摘要QuestionsAnsweredExtractor—— 抽取每个 Node 能够回答的一组问题TitleExtractor—— 根据每个 Node 的上下文抽取标题EntityExtractor—— 抽取每个 Node 内容中提到的实体(地名、人名、事物等)
然后可以把 Metadata Extractor 与节点解析器串联起来:
from llama_index.core.extractors import (
TitleExtractor,
QuestionsAnsweredExtractor,
)
from llama_index.core.node_parser import TokenTextSplitter
text_splitter = TokenTextSplitter(
separator=" ", chunk_size=512, chunk_overlap=128
)
title_extractor = TitleExtractor(nodes=5)
qa_extractor = QuestionsAnsweredExtractor(questions=3)
# assume documents are defined -> extract nodes
from llama_index.core.ingestion import IngestionPipeline
pipeline = IngestionPipeline(
transformations=[text_splitter, title_extractor, qa_extractor]
)
nodes = pipeline.run(
documents=documents,
in_place=True,
show_progress=True,
)1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
或者直接插入索引:
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(
documents, transformations=[text_splitter, title_extractor, qa_extractor]
)1
2
3
4
5
2
3
4
5