Ask HN: Crawl the entire web?

2 pointsnullundefined8y ago1 comments

If you were tasked with crawling the entire web, how would you go about doing it?

1 comments

I'd grab the CommonCrawl data and keep moving on what my actual problem is.

j / k navigate · click thread line to collapse

2 pointsnullundefined8y ago1 comments

If you were tasked with crawling the entire web, how would you go about doing it?

I'd grab the CommonCrawl data and keep moving on what my actual problem is.

j / k navigate · click thread line to collapse