There are a lot of OSS vector search databases out there, we could probably list the main ones:
- Qdrant: https://github.com/qdrant/qdrant
- Weaviate: https://github.com/weaviate/weaviate
- Milvus: https://github.com/milvus-io/milvus
What else?
Another one most similar to qdrant we track internally is lancedb. They are clever by supporting an embedded architecture, so an architectural reason to prefer over most existing OSS DBs. In our survey 2 years ago, we predicted specialized vector DBs having regular OSS DBs be the elephant in the room, and missed embedded as a fundamentally different category: https://gradientflow.com/the-vector-database-index/ .
(Good luck to qdrant! I'm happy they waited before raising, hopefully this means they can operate more healthily than otherwise and easier to maintain the discipline to do that!)
And then there are traditional databases and search products that are integrating vector search capabilities as well: Postgres, Elasticsearch, Opensearch, Solr.
They each have their limitations of course but the 28M round suggests a moat that I'm not seeing that clearly in terms of tech. What's so special about qdrant relative to their competition?
At least they are Apache licensed for now. So, that's nice. But that also means e.g. Apache Lucene could borrow some code from them to beef up their vector search capabilities. Which would benefit Elasticsearch, Opensearch, and Solr which all depend on Lucene.
Which raises the question what the point is of QDrant long term and why investors are betting on this as opposed to other things.
It seems to me that the main challenge with vector search is inference cost (at index and query time), not storing the vectors. A secondary concern is the vector comparisons at query time. A good way to cut down on that is to reduce the overall result set using traditional search or query mechanisms. In other words, you need
From the enterprise perspective, which of these vendors proved the best combination of security, availability, performance and pricing will matter. when we run benchmarks on our (self hosted) LLMs, we do not a clear idea of where we have bottlenecks and we end up assuming its the GPU/memory. And our pilot implementation will never go into production as the security model is nearly non existent in our implementations; the execs AND qa are getting the same RAG outputs. It is all very new to us and our teams. If a vendor can outperform its competition in our tests and show credible security model with segmentation of knowledge, that would be the choice.
I ask because this was the statement from the PP
There are a lot of OSS vector search databases out there, we could probably list the main ones
...
What else?The cases where I've seen this work are when the DB offers something way ahead of what their competitors offer. For example, KDB+ was historically unrivaled when it came to ultra high performance time series storage and Aerospike is very hard to beat for extremely high performance multi-node K/V.
Otherwise there's little to stop a larger company from offering the OSS competitor to your DB as a service for a lower cost and invest eng resources to close the gap.
I also like https://github.com/lancedb/lancedb
(discovered from sibling comment here: https://news.ycombinator.com/item?id=39103322)
I’ve been working with vectors for over a decade; particularly with embeddings used in AI. We’re talking projects from 100k to 100B+ records, used for AI applications
Postgres, particularly with pgvector and derivatives, can handle to millions of records very rapidly no problem. It’s very cheap, scales great, and is accurate.
I’m sure some of these open source solutions are improvements. That said, weigh vendor lock in, cost, risk and in the end it usually makes very little sense.
https://www.capitalone.com/tech/machine-learning/learning-em...
Running an R&D group in the department.
Those high profit margin part of the LLM business is for big players only, they don't burn hundreds of billions to offer you opportunties to cut their profit by capitalizing on their core business.
Communism doesn't exist in high tech. People don't work their xxx off to pave ways for your free lunch for life.
I can see them replacing it if Qdrant isn't able to scale to their needs - thats why we ended up with Dynamo, Spanner, MyRocks. However its likely that its probably easier to just acquire the team - like Apple did with Foundation - and become project stewards than trying to invent a new datastore to save pennies.
I know that, and you know that, but tell it to a jury of East Texas hayseeds.
it is also worth to mention that even if there is going to be a market called "vector databases", which is highly unlikely, you can't just written off all existing regular databases and pretend that they are not going to just walk in and take over.
all in all, there is no reason to believe it is a hot market. it is much better to ask is there going to be a market at all.
They are also a Rust shop.
Who says Germany has no cool startups.
EDIT: Yes, it was Grok.
What have your experiences with vector databases been? I've been using https://weaviate.io/ which works great, but just for little tech demos, so I'm not really sure how to compare one versus another or even what to look for really.
Once in a whole supabase slips into a project, but that's basically just Postgres with some bells and whistles on top.
I got nothing bad to say about Pinecone, We aviate, Chroma etc. but when it comes to dbs, I like to go with the devil I know.
I think there are some things I wish were easier, for example finding and updating points, and the UI could be better.
for now is the keyword here.
When the company grow to 10k or 30k people, there will be teams competing for visibility, someone is going to build their inhouse "vector database" to get his/her slice of the pie. Do you still believe that any AI major player is going to reply on some external vector databases?
exactly! if there is a real & strong demand, we'd be seeing open source ones get upgraded & ready in months. it is more like just one of those "I want to build something easy in the core but fancy in its name to get some quick VC $"
companies like openai will fill the gap by offering such features out of the box. there is no logical reason why and how a big AI tech giant is going to take all hard work and letting someone else to take the profit by ignore the last mile issue. in fact, openai has already released such APIs in their last devday event.
I don't even know any 10-30k people companies that build their own search, most I've known use elastic search or lucene.
It's had to imagine that a few of these vectordb companies don't establish themselves as the standard solution, being the equivalent to MongoDB in their space. The other competent players will very likely get acquired.
Certainly these vectordb companies are in a better long term standing that the bajillion companies rushing to build products that are just calling an API endpoint at the end of the day.
for your question, google built spanner (that was like 12 years ago) and leveldb when it had about 30k people. Meta/facebook forked leveldb and started building rocksdb when they had less than 30k people.
no one is saying any banking, insurance, retail companies with 30k employees should be building their own databases/search engines. They shouldn't. That is actually my main argument - those companies shouldn't get too involved for infra like that, they should just take the offerings of the big tech companies.
Small revolution indeed.
$ git grep "Arc<" | wc -l
451
I genuinely ask - there are a lot of other problems in the RAG, fine tuning, AI/LLM, retireval space, to solve. And more and more vector retrieval is, while not 100% solved, at least is something the community has a grasp on the tradeoffs. Solved to the point that squeezing a bit more recall out of vector retrieval isn't the problem anymore.
I think this is a bit of a strawman. I don't think recall is the main point these systems are trying to sell us on, it's more about robustness and ease of use compared to building something inhouse or using a lower level library to build a system on top of it just for this small part of your overall project/product (be it RAG, search, whatever).
I guess Lucene-based solutions, while very mature overall in terms of engineering, lagged behind this functionality (out of caution, trying to build what's going to be long term useful) and are also perceived a bit too cumbersome. So these stores do make sense, I think. The core functionality is nothing too complex (at least HNSW), but hiding it behind a stable black box with just a few inputs and levers, has value for people that are likely to use these stores.
There is still work to be done in vector databases. None of the products have perfected hybrid search yet, for example, and performance varies a lot between products; they are not fungible.
What do you think you milvus? https://milvus.io/. The difference seems significant from the architecture perspective.
Re embeddings, you would likely get better results if you train your own embeddings model. A popular approach is ColBERT, which anecdotally outperforms vector search in border cases[1]. Second is training an embedding model using initial layers of an LLM. [2]. In Colbert's case once it's trained, you dont need a db to store the vectors.
[1]: https://twitter.com/arjunkmrm/status/1744741903646773674 [2]: https://huggingface.co/intfloat/e5-mistral-7b-instruct
Support from the team is great however, and congrats to them for this round!
We use 768 dimensional vectors for our items with several other payload filters (e.g. language). Performance has been good and I think the qdrant team focus on the right features without creeping into other areas.
It's probably going to stay, but I'm also evaluating databricks new vector store as we're using databricks for all the analytics parts of the app already, and having them all on the same infrastructureis appealing.
(now I'm gonna plug what I work on)
If you're interested in a more scalable vector database written in Go, check out Milvus (https://github.com/milvus-io/milvus)
Let’s see what they can do in a year or more with that new capital.
"We are getting many applications for this position. Usually, a test task would help preselect suitable candidates. However, since we develop open-source software, we rely on contribution.
You can build an open-source Qdrant connector to another framework or library. The simplest one would be, for example, a Streamlit data connector. But other ideas are more than welcome!
No limitations and no deadline. As long as this job position is online, we accept submissions. After you are done, send us an email to career@qdrant.com with the link to the repo. We will review it and get back to you asap."
No interviews, conversation before this email. Hope they see and fix this.
Edit : No Pay.
Curious if the co-founder who was posting in here will share his take.
For a job switch, I need to spend time in three different stages:
---------------
Preparation:
Leetcode (Blind 75) : 150 hours
System Design + DBMS + OS + Networking : 100 hours
Behavioural Questions (preparing STAR format answers): 10-20 hours
------------
Application:
Avg time for sending 500 applications: 20 hours (Assuming 1 application every 2.5 minutes)
-------------
Interviews:
Let's say I got 25 callbacks and 10 of them asked for takehome.
Person to person interviews time: 25 * 3 = 75 hours
Takehomes: 10 * 6 hours = 60 hours.
-------------
All in all, I'm already spending 415 hours of unpaid work to get x% of salary increment. Not including the side projects or hackathons we may need.
So having a takehome exercise asking to make an active contribution to the company is....bad. Sure i can reject it but not everyone will. which is what led us into the multiple rounds of algo interviews hellhole.
I apologize if what I'm saying is harsh. All I want is for leadership to see us as humans with families and not monkeys jumping through hoops.
If they're not then it's a scandal.
Like, you had to have actively developed mods for them, for free, for years, and be famous in the community, then they'll hire you (If you want).
This works because the working conditions there are far far better than your average game company. And probably much more fun than say a bank.
Over the years I've read/heard plenty of stories (here on HN and elsewhere) of people getting hired for their open source contributions to some stack that some company is using/developing.
So here I am willing to give some slack here to Qdrant. They get extremely qualified candidates who can jump right in, and candidates get told the rules of the game up front. It feels fine?
Surely much better than fake take home tests, whiteboard tests, leetcode onslaught, and 7 layers of interviews.
So if creating a high quality repo is 60-90% of your job interview that seems pretty good. As long as they are not ghosting high quality contributions that is.
I will change my view if they get 20 high quality connectors out of this and noone gets hired from that pool of candidates.
From this job posting [0]
> Compensated Interview Tasks: We value your time and effort; candidates will be compensated for completing interview tasks.
[0] -- https://join.com/companies/qdrant/10275180-core-rust-enginee...
From your perspective, you're filling out an application, maybe writing a cover letter, but on the other side, there are 100+ applications like yours. Not all of them are qualified, CVs are not a trustable source anyway.
That's why companies add tests to filter first, then interview later.
As a thumb rule, I'm happy to put 4x more effort than the company. If they interview me for 1 hour, I spend 4 hours doing the take-home. Anything more feels like exploitation.
HR gets paid to talk to candidates. I don't get paid to apply. The initial screening call is what allows a company to gauge the relevancy of a candidate. Let him speak about some of the topics and see how in-depth they go. Either the HR is familiar enough with the tech to understand proficiency (think a student listening to a maths professor) or they let a TL have a short conversation. I've overheard unqualified HR do their jobs badly, too; They laughed at picking them by looks and "feels". But, that's out of scope here.
A large company has millions to invest in different areas. Intrinsically, it has a much larger margin of error. You accidentally overprovisioned some resources and cost the company 10k? Tis but a scratch. You POC some personal project and accidentally get billed 10k? That is not the same.
A company can spend money on hiring. It is expected to. A private person can't spend money on applying to jobs. It isn't expected.
It's interesting to see how the shift goes from the self to the company [and to the country]. A little bit of communist propaganda goes a long way, eh, comrade NPC?