undefined | Better HN

0 pointstoast03y ago0 comments

It would be nice if compressed HTML was on a similar order to filesystem block sizes, but most pages are much larger than 4k bytes even compressed. May depend on the specific site that's being scraped.

0 comments

1 comments · 1 top-level

marginalia_nu3y ago

This isn't a hypothetical. I've used this exact workflow for this exact use case. I'm telling you it's no good.

I run a search engine crawler and my average across 100M docs is about 7 Kb when compressed with zstd (fs block size is typically 4 Kb). Some much larger than that of course, but many smaller still. HTML in general compresses absurdly well.

j / k navigate · click thread line to collapse