feat: Assistant API

[mudler]

OpenAI released the Assistant API. https://platform.openai.com/docs/assistants/overview

This thread is to collect thoughts on how to integrate this - it seems we have already all the needed pieces, but it's needed to stich things together to expose a similar API.

I didn't had a closer look at the API yet - not on my radar until #1126 is completed

For reference, the OpenAI API spec: https://github.com/openai/openai-openapi

[muka]

In the effort to contribute on this, I started collecting the requirements and functionalities offered by the Assistant API.

Assistant functionalities

• Assistant API
• Thread API
• Run API
• File upload API
• Tools support
• RAG support (involving also a vector db, embeddings, feature extraction and search)
• Code instruct support
• Automatic truncation (or other approaches) of long context to fit in a Thread

Links

• Docs
• Analysis of random guy on the internet
• Assistant API
• Thread API
• Messages API
• Runs API

[mudler]

First PR in that direction adding File API: #1703

[richiejp]

I have a few ideas for vector search in order of my personal preference:

1. Embedded vector store and search
2. Host one of the many vector DBs in a backend
3. Connect to an external store

All could exist at once. I like the first one for the use-case where someone has a limited number of documents and/or search volume. Also for being the default choice in LocalAI without incurring a lot of maintenance or bloat. It should be relatively simple because:

• Doing a brute force vector search in memory is reasonably fast for up to let's say 1M small vectors. Also it's an exact search whereas others are approximate. Both HNSW and brute force search implementations are included in the link
• Embeddings and document chunks could be saved to flat file and loaded into memory when needed
• Alternatively BadgerDB can be embedded which allows fast key iteration for comparing the vectors
  • Even large 4096 dimension embeddings can be stored as keys and the values can be document segments.
  • The keys can be prefixed with a file ID, so that only particular files are used in a query matching the OpenAI API
  • Should handle the use-case where events are being streamed into the database in real-time

So I went ahead and started an experiment external to LocalAI: https://github.com/richiejp/badger-cybertron-vector/blob/main/main.go

It's probably really slow due to copying the keys, lack of parallel execution and such, but it works. I expect these things can be optimized. The question is whether to go with BadgerDB or a pure in memory implementation?

BTW Cybertron is pretty cool, that could be a new backend.

[richiejp]

Perhaps instead of, or in addition to what I have done with the basic vector search. We could have a higher level API which is backed by https://github.com/bclavie/ragatouille as a starting point.

The reason is that it seems colBERT v2 is far superior to basic cosine similarity search, but it is difficult to unpack it and get it to work with some arbitrary vector database. It's possibly the wrong level of abstraction for LocalAI to be working at even internally.

Opinions on implementing Ragatouille or colBERT (https://github.com/stanford-futuredata/ColBERT) as a backend?