undefined | Better HN

0 pointsmarginalia_nu4y ago0 comments

I've honestly not had much problem at all crawling the web as an indie search engine operator. If you want to get past CloudFlare you can register your bot fingerprint with them.

A small number of sites has blocked my crawler , but that's almost always been my own fault, and happened a few instances when the crawler was misbehaving and actually fetching too aggressively (or repeatedly). In every case just sending an email to the site explaining what happened and humbly asking for a second chance been enough to be allowed back in.

Most website owners don't seem to mind small search engines at all, what they don't want is scrapers that aggressively scrape their entire site 10 times a day, ignoring robots.txt, and being a general nuisance.

0 comments

kristjansson4y ago

There's also the alignment of interests between you and the site operators, even for a small search engine. Cooperation improves outcomes for both sides. Whereas the more aggressive scrapers almost certainly have interests that conflict with the site operators, regardless of the costs to serve the traffic itself.

gzer04y ago

How did you go about registering your bot fingerprint with Cloudflare? I did a quick search but cannot find anything with regards to that.

marginalia_nuOP4y ago

Good question, I can't find the link now either. I found it somewhere on Cloudflare's site.

--edit--

Nevermind, here it is: https://support.cloudflare.com/hc/en-us/articles/36003538743...

It should skip to "I run a good bot...", but if it doesn't, that's where you want to scroll.

j / k navigate · click thread line to collapse