undefined | Better HN

0 pointsgreglindahl9y ago0 comments

A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.

0 comments

2 comments · 1 top-level

sjwright9y ago· 1 in thread

I agree, it's a difficult conundrum. It sucks.

There are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...

j / k navigate · click thread line to collapse