Looks cool. A couple of questions:
1. Does it support fine tuning with different losses? For example, where you don't need to provide negatives and it uses the other examples in the batch as negatives
2. Can you share inference speed info? I know that Colbert should be slow since it creates many embeddings per passage
Hi, there is a single loss right now, but I plan to add some Sentence Transformers losses. ColBERT is slow as a retriever, but is quite efficient as a Ranker on GPU (way faster than cross-encoder). I plan to release pre-trained checkpoints on HuggingFace with benchmarks using BEIRand inference speed info.
Do you mean it's faster when the embeddings are pre-computed or is it faster when the embeddings are computed on the fly as well.
Also, what's the recommended way to store the colbert embeddings as, because of the 2d nature of the embeddings it's not practical to store in a vector database.
It's because of the loss of the model. I ask the model to produce a higher similarity between the query and the positive document rather than between the query and the negative document. I'll add more losses soon so there are more choices
Anecdote: neural-cherche seems useful as I have analysts creating positive & negative feedback data (basically thumbs-up/down signals) that we will use to fine tune retrieval models.
Assuming not much effort is required to make this work for similar models? (i.e. BGE)
You could recommend content based on user query, tag content produced by the user, use colbert as part of a ChatBot to show evidences to the user questions
In the documentation there is an evaluation module with detailed informations. The idea is to gather relevant pairs of queries and documents that are not part of the training set. Then the idea is to measure, using various metrics, how your model can retrieve accurate documents.
