Ask HN: How do I crawl responsibly?

1 pointsedward_rolf9y ago2 comments

I've been developing a in-process search engine for some a while. Now it's time to experiment in distributing it over many machines and also serve up a public GUI but I am wary because I have never crawled the web before.

To start with I'm just going to index as much data I can fit on an entry-level cloud machine and because I am very poor I shall be asking for donations to further the scope of the index.

Say I start with Wikipedia and The Gutenberg project and a couple of news sites. The first two will be easy, they have dumps of their data and I also don't think Wikipedia would mind at all if I put a tiny amount of preasure on their servers for the good cause of building a free, anonymous and open web search. But what about the rest of the internet? Will they mind?

People crawl and scrape the web all the time for different purposes. I'm looking for some advise so that I don't piss anyone off with my crawler. What tools/strategies do you suggest I use?

Cheers!

2 comments

2 comments · 1 top-level

Petrakis9y ago· 1 in thread

Check the site robots.txt

https://en.wikipedia.org/wiki/Robots_exclusion_standard

But basically scrapping is a loophole, you are not doing nothing wrong because everything is accessible with browser. Still web admins get upset.

For tools I use now selenium with phantomJS, but you can also use scrappy.

As long as you don´t claim alot of bandwith that other users could use should be fine

edward_rolfOP9y ago

Thx!

j / k navigate · click thread line to collapse