[Question]: how to combine milvus with my custom node ingestion pipeline

[ZJL0111]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

nodes = meta_data2_nodes(documents)
storage_context = StorageContext.from_defaults()
storage_context.docstore.add_documents(nodes)
vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
keyword_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)

meta_data2_nodes is my custom metadata extractor function. now as my data amount is quite large, i want use milvus as my vector store backend, and here is my code

documents = SimpleDirectoryReader("./data/TEST_V1/Antibody").load_data()
nodes = meta_data2_nodes(documents)
# initialize storage context (by default it's in-memory)
storage_context = StorageContext.from_defaults()
storage_context.docstore.add_documents(nodes)
# 创建向量索引（使用 Milvus）
vector_index = VectorStoreIndex.from_vector_store(vector_store=milvus_store,
storage_context=storage_context)
# 创建关键词索引（使用默认存储）
keyword_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)

my question is without passing nodes to vector_index = VectorStoreIndex.from_vector_store(vector_store=milvus_store, storage_context=storage_context) , can i still get the correct vector index in milvus (nodes info, embedding info)

[dosubot]

Hey @ZJL0111! 👋 I'm here to help you out with any bugs, questions, or contributions you might have. Let's tackle this together!

Yes, you can get the correct vector index in Milvus without passing nodes to VectorStoreIndex.from_vector_store. You can create a VectorStoreIndex directly from a MilvusVectorStore without needing to pass nodes. Here's an example:

from llama_index.vector_stores.milvus import MilvusVectorStore
from llama_index.core import VectorStoreIndex

vector_store = MilvusVectorStore(
    overwrite=False,
    uri="your_milvus_uri",
    token="your_milvus_token",
    collection_name="your_collection_name",
)

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

This approach initializes the index using the existing data in Milvus, so you don't need to pass nodes explicitly ^[1].

To continue talking to Dosu, mention @dosu.

[ZJL0111]

Hey @ZJL0111! 👋 I'm here to help you out with any bugs, questions, or contributions you might have. Let's tackle this together!

Yes, you can get the correct vector index in Milvus without passing nodes to VectorStoreIndex.from_vector_store. You can create a VectorStoreIndex directly from a MilvusVectorStore without needing to pass nodes. Here's an example:

from llama_index.vector_stores.milvus import MilvusVectorStore
from llama_index.core import VectorStoreIndex

vector_store = MilvusVectorStore(
    overwrite=False,
    uri="your_milvus_uri",
    token="your_milvus_token",
    collection_name="your_collection_name",
)

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

This approach initializes the index using the existing data in Milvus, so you don't need to pass nodes explicitly [1].

To continue talking to Dosu, mention @dosu.

thanks! Dosu

[dosubot]

Hi, @ZJL0111. I'm Dosu, and I'm helping the LlamaIndex team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You sought advice on integrating Milvus as a vector store backend in your custom node ingestion pipeline.
• I confirmed that it's possible to create a VectorStoreIndex directly from a MilvusVectorStore without passing nodes.
• A code example was provided, which resolved your query, and you acknowledged the help.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the LlamaIndex repository by commenting on this issue.
• If there are no further updates, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!