Skip to content

Top Best Ask Show New Jobs

Ask HN: How difficult is a search engine MVP that just works?

9 pointsbluu005y ago8 comments

what it is, i mean, what's stopping you from making the next google search?

Tf-Idf? No. Just kidding. Infromation retrieval, text mining, ML/DL ?! what is going on with this field ! Every other resource seems outdated? What is the state of art ?

Reading some of these posts : https://boyter.org/2010/08/build-vector-space-search-engine-python/

https://www.dr-josiah.com/2010/07/building-search-engine-using-redis-and.html

https://stevenloria.com/tf-idf/

https://stories.algolia.com/a-search-engine-in-css-b5ec4e902e97

8 comments

8 comments · 5 top-level

a3camero5y ago· 1 in thread

I’ve done this over the last year on a tiny scale for my own needs: gorillafind.com. It’s from scratch and just for government sites to sidestep some of the challenges (but so far only has 50 sites). The cost per site is around $1/mo for crawling, indexing, converting file formats and then serving up results. It’s difficult but not impossible and very educational. If you’d like to hear more about doing it yourself and some of the challenges feel free to email me with the contact info on the site. My system isn’t open source but I’m more than happy to chat about the research I’ve done and how you can make one.

I’d start off with not doing state of the art because it’s overkill for an “MVP”. And if you don’t need proper browser rendering of pages, there’s open source crawlers out there like Nutch that might work. If you’re making one yourself, the outdated academic papers and presentations by search companies are a good resource as the basic ideas of crawling and indexing haven’t changed too much (even if ranking and other components have changed a lot). A search engine is really a set of related components and there are many examples out there to use as inspiration for your MVP.

bluu00OP5y ago

> gorillafind.com

Interesting. (ﾟヮﾟ) You seem to know well that search is beyond ranking and retrieval. [1]

> I’d start off with not doing state of the art because it’s overkill for an “MVP”.

Can a search engine "MVP" sustain without using sota methods/practices?

What should be the initial scale for a search MVP for which it just works? In the modern realm.

[1] : https://medium.com/startup-grind/what-every-software-enginee...

just a note (you can ignore) : I am actually a believer of iterations in development and incremental optimisations. But even a minimum viable product should be able to work.

ploika5y ago· 1 in thread

Have you read this post?

https://danluu.com/sounds-easy/

It describes some of the various difficulties in building the next Google Search, much better than I could.

bluu00OP5y ago

interesting read

ioli5y ago· 1 in thread

I don't know how feasible it is, but couldn't you use ElasticSearch ???

bluu00OP5y ago

what is elastisearch? IDK ¯\_(ツ)_/¯

EDIT #1: you're cool. but i really dont know what elastisearch is. ¯\_(ツ)_/¯ And a google "search" hurts at the moment.

EDIT #2 :

> multiple people respond and say that Lucene basically does the same thing Google does and that Lucene is poised to surpass Google's capabilities in the next few years. It's been long enough since then that we can look back and say that Lucene hasn't improved so much that Google is in danger from a startup that puts together a Lucene cluster. [1]

[1] : https://danluu.com/sounds-easy/

bluu00OP5y ago

And to add, i am not looking for a "how to learn"?

Neither it is some site-wide search que.

And not a business model as well.

Not a "privacy first but no results" search.

Something that works.

techdragon5y ago

I built a “search engine” but unfortunately my approach lacked any implicit ranking mechanism.

TLDR: my regex search engine needs result ranking to be more useful before I consider showing it to other humans.

j / k navigate · click thread line to collapse