Extracting Subset of Common Crawl Data on Laptop (opens in new tab)

(avilpage.com)

1 pointschillaranand3y ago1 comments

1 comments

Each Common crawl monthly data consists of ~100 TB. For some use cases, we don't need entire data set. We just need a subset of the data.

In this post, lets see how we can extract sub set of the data from our laptop itself.

j / k navigate · click thread line to collapse

Each Common crawl monthly data consists of ~100 TB. For some use cases, we don't need entire data set. We just need a subset of the data.

In this post, lets see how we can extract sub set of the data from our laptop itself.

j / k navigate · click thread line to collapse