原有的 token 计数实现已弃用。

我们知道 token 计数对许多用户都很重要，因此写下这份指南，帮你尽量无痛地完成这次迁移。

此前，token 计数直接记录在 llm_predictor 与 embed_model 对象上，并可选地打印到控制台。该实现使用一个静态分词器（gpt-2）做 token 计数，而 last_token_usage 与 total_token_usage 两个属性并不总能被正确维护。

往后，token 计数已迁移为一个回调。使用 TokenCountingHandler 回调后，你在「如何统计 token」「统计结果的有效期」上有了更多选择，甚至可以为不同索引各建一个 token 计数器。

下面是把新的 TokenCountingHandler 与 OpenAI 模型搭配使用的最小示例：

import tiktoken
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
from llama_index.core import Settings

# you can set a tokenizer directly, or optionally let it default
# to the same tokenizer that was used previously for token counting
# NOTE: The tokenizer should be a function that takes in text and returns a list of tokens
token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode,
    verbose=False,  # set to true to see usage printed to the console
)

Settings.callback_manager = CallbackManager([token_counter])

document = SimpleDirectoryReader("./data").load_data()

# if verbose is turned on, you will see embedding token usage printed
index = VectorStoreIndex.from_documents(
    documents,
)

# otherwise, you can access the count directly
print(token_counter.total_embedding_token_count)

# reset the counts at your discretion!
token_counter.reset_counts()

# also track prompt, completion, and total LLM tokens, in addition to embeddings
response = index.as_query_engine().query("What did the author do growing up?")
print(
    "Embedding Tokens: ",
    token_counter.total_embedding_token_count,
    "\n",
    "LLM Prompt Tokens: ",
    token_counter.prompt_llm_token_count,
    "\n",
    "LLM Completion Tokens: ",
    token_counter.completion_llm_token_count,
    "\n",
    "Total LLM Token Count: ",
    token_counter.total_llm_token_count,
)