undefined | Better HN

0 pointsthomassmith654y ago0 comments

  Honestly I think people are completely underestimating 
  the difficulty of a good search engine

I suspect that is not difficult so much as expensive.

Boutique search engines pop up all the time here on HN, but they can't compete fairly against Google, without the resources to crawl a billion webpages day after day.

0 comments

charcircuit4y ago

Google also keeps a copy of the "entire" web in RAM to search it faster.

1_player4y ago

Source? But it's definitely possible nowadays when RAM is cheap and modern Google makes the Internet feel incredibly smaller than it was 20 years ago.

At the time you would get tons of results from a myriad of small blogs, forums, niche websites. Nowadays it's Pinterest, blogspam and more SEO optimised algorithmic crap. If you're lucky you get a forum result that might actually be relevant.

Given the joke that if you can't Google it it doesn't exist, it follows that the Internet has considerably shrunk in the past two decades.

cromwellian4y ago

Here, let Jeff Dean explain how it works (and why this is a complex problem that a lot of the naive 'build a better search engine than Google' miss)

https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8...

The original article's description of how Google search signals work doesn't even scratch the surface of all the signals it's using.

1 more reply

charcircuit4y ago

It looks like I was mistaken. They don't keep all of the documents in RAM, but I'm pretty sure they keep the search index (100 petabytes) in RAM. I'm struggling to find a source but I remember hearing it in a talk given by a Googler on YouTube. I also read somewhere that moving their whole index to RAM is how they brought their response times down from a couple of seconds to a couple hundred milliseconds.

j / k navigate · click thread line to collapse

0 pointsthomassmith654y ago0 comments

  Honestly I think people are completely underestimating 
  the difficulty of a good search engine

I suspect that is not difficult so much as expensive.

Boutique search engines pop up all the time here on HN, but they can't compete fairly against Google, without the resources to crawl a billion webpages day after day.

0 comments

charcircuit4y ago

Google also keeps a copy of the "entire" web in RAM to search it faster.

1_player4y ago

Source? But it's definitely possible nowadays when RAM is cheap and modern Google makes the Internet feel incredibly smaller than it was 20 years ago.

Given the joke that if you can't Google it it doesn't exist, it follows that the Internet has considerably shrunk in the past two decades.

cromwellian4y ago

Here, let Jeff Dean explain how it works (and why this is a complex problem that a lot of the naive 'build a better search engine than Google' miss)

https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8...

The original article's description of how Google search signals work doesn't even scratch the surface of all the signals it's using.

1 more reply

charcircuit4y ago

j / k navigate · click thread line to collapse