This is a homage project for the famous Word2Vec. This project is functionally identical to the original project but is newly written in C++23 along with more recent technologies, such as TBB. The original implementation is known for its fast training, so adding more parallel processing and redesigning may not be fruitful. However, it becomes easier to extend its functionality.

The main reason I wrote the project was to understand Word2Vec at the deepest level. I hope the implementation is beneficial to C++ developers who want to understand the internals of Word2Vec.

As the following code snippet demonstrates, the project consists of explicit building blocks, such as Dictionary and WordLayer. These building blocks are statically typed, so there is no runtime overhead during execution.

// 1. build vocabulary for Word2Vec
Dictionary dictionary(...);

// 2. Create a Word2Vec instance
Word2Vec word2vec(dictionary); 

//3. Specify a model and a train method with templates, 
//   such as Train<SkipGram, NegativeSampling>
WordLayer word_layer = 
  word2vec.Train<ContinuousBagOfWords, HierachicalSoftMax>(
                                                  .../*train file & opts. */);
  
// 4. word_layer has trained vectors.
word_layer.SaveVectors(...) 

The original implementation doesn’t use locks as well. However, it may compensate for file-reading performance in certain environments. A thread in the original Word2Vec implementation reads only a chunk of a training file. Though this is not wrong for training word vectors and it could be a minor issue, it may degrade IO performance when a file is on a sequential reading-oriented device, such as an HDD, which requires a head to move. The project uses a lock-free MPMC algorithm for file reading and guarantees that a given sentence is processed exclusively by a specific thread.

The buffer design for word vectors is intended to be read by multiple threads. It increases the chances of cache hits.

As the project uses the standard GNU option style, it can’t use the same options as the original version, such as a single-hyphen option. —train instead of -train.

In the original implementation, a training thread reads exclusively from a specific chunk of a training file. This project reads a training file sequentially multiple times, as specified by the given -iter option, but it distributes sentences to each training thread. It uses the Vyukov Ring Buffer algorithm.

The project inherits the way words are indexed: 0 is for </s>, and the others are descending,g sorted by their appearances in a vocabulary file, but the new line vector:0 is not trained.

There are two main differences from the original implementation. One is that the project uses a parallel model with TBB, and the other is that this implementation may intentionally produce fewer clusters than the given classes argument.