undefined | Better HN

0 pointsBeefWellington1y ago0 comments

Interestingly:

- Bing: Goodreads is third result

- DDG: Paywalled NZ Herald article appears to be the only result.

Of course what you've really uncovered is that Google search is the only thing that respects robots.txt: https://www.goodreads.com/robots.txt

0 comments

3 comments · 2 top-level

atombender1y ago· 1 in thread

The robots.txt does not prevent Google from scraping the quote page, which is the one that matches for me:

The robots.txt seems to exclude things which are genuinely useful to exclude, like RSS feeds.

While true, it seems like every individual quote page also includes:

    <meta content='noindex' name='robots'>

Worth noting as well, their sitemaps all 404.

No, you've uncovered that.

I gave a thought on whether this could be some copyright issue with Amazon but didn't bother to check the robots.txt file, nicely done!

j / k navigate · click thread line to collapse