Vector model - implementation preview

[lvca]

This is the first piece of the Vector Model for ArcadeDB. We extended the amazing Open Source HNSW library by Jelmer Kuperus (https://github.com/jelmerk/hnswlib) to be supported in ArcadeDB as an index.

HNSW stands for "Hierarchical Navigable Small World" and it's probably the best algorithm to index multi-dimensional data. For more information look at the paper "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs".

After trying to load some of the most popular datasets, we used the publicly available Word2Vec with 2M words, 300 dimenions.

To experiment this new feature, just run the Java class com.arcadedb.index.vector.FastTextDatabase that downloads the first time the dataset, and start indexing all the words in ArcadeDB using the new HNSW index.

The best way to load vectors in ArcadeDB is by using a RAM-based index to load the dataset quickly:

HnswVectorIndexRAM<String, float[], Word, Float> hnswIndex = HnswVectorIndexRAM.newBuilder(300, DistanceFunctions.FLOAT_INNER_PRODUCT, words.size())
          .withM(16).withEf(200).withEfConstruction(200).build();

And then make it persistent in ArcadeDB:

persistentIndex = hnswIndex.createPersistentIndex(database)//
.withVertexType("Word").withEdgeType("Proximity").withVectorPropertyName("vector").withIdProperty("name").create();

persistentIndex.save();

With the code above, you're told to store the HNSW index inside "Word" vertices by using the edge type "Proximity". The vector property will be stored in the property "vector", as a float[300]. The actual word as a string is saved in the property "name".

The types and properties mentioned above are automatically created if not already in the database.

After the first import, your data is in ArcadeDB, so you can just open your database and get the HNSW index as a normal index:

persistentIndex = (HnswVectorIndex) database.getSchema().getIndexByName("Word[name,vector]");

Note the index name is composed of the selected vertex type and the name and vector properties.

The Java test class pulls a random word from the database and looks for its neighbors.

List<SearchResult<Vertex, Float>> approximateResults = persistentIndex.findNeighbors(input, k);

Where input is the word and K is the maximum number of neighbors to retrieve, ordered by the highest proximity, namely the K closest words will be returned.

The result is an ordered list where you can get the word vertex and its proximity (as a Float).

[lvca]

Inspired by the Qdrant Benchmark page (https://qdrant.tech/benchmarks/) I downloaded the GloVe-100-Angular dataset (100 dimensions, 1.2M entries) and loaded it into ArcadeDB. Below are some numbers running on my Mac Book Pro 2019 (Intel CPU, 32GB RAM)

• The dataset was loaded and indexed in ArcadeDB in 9 minutes using only 8GB RAM.
• The RPS (request per second) is 435.58 by using M=16 and EF=128

I can't see what's the accuracy of the responses, I guess it should be measured against a distance calculated with all the vector embeddings.

[lvca]

Just integrated the vector index into the Importer. Now you can import a Word2Vec or GloVe text file with the IMPORT DATABASE:

import database file://glove-angular-100.txt
with distanceFunction = 'cosine', m = 16, ef = 128, efConstruction = 128, 
vertexType = 'Word', edgeType = 'Proximity', vectorProperty = 'vector', idProperty = 'name'" 

[codacy-production]

Coverage summary from Codacy

Merging #1148 (58d0f55) into main (5364986) - See PR on Codacy

Coverage variation	Diff coverage
Report missing for 53649861	24.16%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (5364986)	Report Missing	Report Missing	Report Missing
Head commit (58d0f55)	52743	32733	62.06%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#1148)	1759	425	24.16%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Footnotes

1. Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct. ↩