OpenVINO™ 是一个用于优化和部署 AI 推理的开源工具包。OpenVINO™ Runtime 可以在各种硬件设备上运行同一优化模型。加速你的深度学习性能，适用于语言 + LLM、计算机视觉、自动语音识别等用例。

OpenVINOGenAILLM 是 OpenVINO-GenAI API 的封装。可以通过由 LlamaIndex 封装的此实体在本地运行 OpenVINO 模型：

在下面一行中，我们安装本演示所需的软件包：

%pip install llama-index-llms-openvino-genai

%pip install optimum[openvino]

设置完成后，我们来试试：

如果你在 Colab 中打开这个 Notebook，可能需要先安装 LlamaIndex 🦙。

!pip install llama-index

from llama_index.llms.openvino_genai import OpenVINOGenAILLM

/home2/ethan/intel/llama_index/llama_test/lib/python3.10/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_path" in OpenVINOGenAILLM has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(

模型导出

可以使用 CLI 将模型导出为 OpenVINO IR 格式，并从本地文件夹加载模型。

!optimum-cli export openvino --model microsoft/Phi-3-mini-4k-instruct --task text-generation-with-past --weight-format int4 model_path

你可以从 Hugging Face 的 OpenVINO 模型中心下载优化后的 IR 模型。

import huggingface_hub as hf_hub

model_id = "OpenVINO/Phi-3-mini-4k-instruct-int4-ov"
model_path = "Phi-3-mini-4k-instruct-int4-ov"

hf_hub.snapshot_download(model_id, local_dir=model_path)

Fetching 17 files:   0%|          | 0/17 [00:00<?, ?it/s]

'/home2/ethan/intel/llama_index/docs/examples/llm/Phi-3-mini-4k-instruct-int4-ov'

模型加载

可以通过 OpenVINOGenAILLM 方法指定模型参数来加载模型。

如果你有 Intel GPU，可以指定 device="gpu" 在其上运行推理。

ov_llm = OpenVINOGenAILLM(
    model_path=model_path,
    device="CPU",
)

你可以通过 ov_llm.config 传入生成配置参数。支持的参数列在 openvino_genai.GenerationConfig。

ov_llm.config.max_new_tokens = 100

response = ov_llm.complete("What is the meaning of life?")
print(str(response))

# Answer
The meaning of life is a profound and complex question that has been debated by philosophers, theologians, scientists, and thinkers throughout history. Different cultures, religions, and individuals have their own interpretations and beliefs about what gives life purpose and significance.

From a philosophical standpoint, existentialists like Jean-Paul Sartre and Albert Camus have argued that life inherently has no meaning, and it is

流式

使用 stream_complete 端点

response = ov_llm.stream_complete("Who is Paul Graham?")
for r in response:
    print(r.delta, end="")

Paul Graham is a computer scientist and entrepreneur who is best known for founding the startup accelerator program Y Combinator. He is also the founder of the web development company Viaweb, which was acquired by PayPal for $497 million in 1raneworks.

What is Y Combinator?

Y Combinator is a startup accelerator program that provides funding, mentorship, and resources to early-stage start

使用 stream_chat 端点

from llama_index.core.llms import ChatMessage

messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="What is your name"),
]
resp = ov_llm.stream_chat(messages)

for r in resp:
    print(r.delta, end="")

I'm Phi, Microsoft's AI assistant. How can I assist you today?

更多信息请参阅：

• OpenVINO LLM 指南。

• OpenVINO 文档。

• OpenVINO 入门指南。

• 使用 LlamaIndex 的 RAG 示例。