Ask HN: Crawl the entire web?

2 pointsnullundefined8y ago1 comments

If you were tasked with crawling the entire web, how would you go about doing it?

1 comments

1 comments · 1 top-level

I'd grab the CommonCrawl data and keep moving on what my actual problem is.

j / k navigate · click thread line to collapse