In my experience, semantic search systems are very hard to test and don’t always return great results when given a very small or vague query, which seems counter to how people are taught to think about search.
And then if you want to do a hybrid search with reranking, those network calls start to add up and degrade the UX.
And then each month, there’s a new post about some new architecture that promises another search accuracy or performance boost, but you’ll need to rebuild the entire search system to find out if it’s effective for your use case.
I assume the LLM understands my intent through a well described prompt more than a direct input vector database would. Assuming this, why does the database matter? Just have a great db backend and then on top an LLM that helps you find the data you want?