Nomic 已发布 v1.5 🪆🪆🪆，它支持通过 matryoshka 学习生成可变尺寸的嵌入，上下文长度为 8192，嵌入维度介于 64 到 768 之间。

在本 Notebook 中，我们将探索在不同维度下使用 Nomic v1.5 embedding。

安装

%pip install -U llama-index llama-index-embeddings-nomic

设置 API 密钥

nomic_api_key = "<NOMIC API KEY>"

import nest_asyncio

nest_asyncio.apply()

from llama_index.embeddings.nomic import NomicEmbedding

维度为 128

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

128

embedding[:5]

[0.05569458, 0.057922363, -0.30126953, -0.09832764, 0.05947876]

维度为 256

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=256,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

256

embedding[:5]

[0.044708252, 0.04650879, -0.24182129, -0.07897949, 0.04776001]

维度为 768

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=768,
    model_name="nomic-embed-text-v1.5",
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

768

embedding[:5]

[0.027282715, 0.028381348, -0.14758301, -0.048187256, 0.029144287]

你仍然可以使用 v1 Nomic Embeddings

它具有固定的 768 个嵌入维度

embed_model = NomicEmbedding(
    api_key=nomic_api_key, model_name="nomic-embed-text-v1"
)

embedding = embed_model.get_text_embedding("Nomic Embeddings")

print(len(embedding))

768

embedding[:5]

[0.0059013367, 0.03744507, 0.0035305023, -0.047180176, 0.0154418945]

让我们使用 Nomic v1.5 Embedding 构建端到端 RAG 流水线。

我们将使用 OpenAI 完成生成步骤。

设置嵌入模型和 llm。

from llama_index.core import settings
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI

import os

os.environ["OPENAI_API_KEY"] = "<YOUR OPENAI API KEY>"

embed_model = NomicEmbedding(
    api_key=nomic_api_key,
    dimensionality=128,
    model_name="nomic-embed-text-v1.5",
)

llm = OpenAI(model="gpt-3.5-turbo")

settings.llm = llm
settings.embed_model = embed_model

下载数据

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

--2024-02-16 18:37:03--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8003::154, 2606:50c0:8000::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: 'data/paul_graham/paul_graham_essay.txt'

data/paul_graham/pa 100%[===================>]  73.28K  --.-KB/s    in 0.02s

2024-02-16 18:37:03 (3.87 MB/s) - 'data/paul_graham/paul_graham_essay.txt' saved [75042/75042]

加载数据

documents = SimpleDirectoryReader("./data/paul_graham").load_data()

创建索引

index = VectorStoreIndex.from_documents(documents)

查询引擎

query_engine = index.as_query_engine()

response = query_engine.query("what did author do growing up?")
print(response)

The author, growing up, worked on writing and programming. They wrote short stories and also tried writing programs on an IBM 1401 computer. Later, they got a microcomputer and started programming more extensively, writing simple games and a word processor.