This seems like a good use case for some sort of spam filtering. Maybe the web is that big, but what percentage of it is data someone will ever care about? I wonder if you could make a good-enough search with agressive filtering of pages before they enter the index.

A search engine doesn't index the HTML code. You're looking at a few Kb per document. You also don't need multiple historical snapshots of the document like WM retains.

So you're looking at maybe 20 bn docs, 4 Kb each. 100 Tb, before compression.

>100kb max per page?

They changed that limit somewhere in the mid-2000s. Just as well, there's some CMS's out there where there's several hundred kilobytes of inline JS and CSS before any body text.

> Internet archive snapshot the whole page while google only index the first few kb of text data

Is that a good way to build an index?

>so maybe it'll cost lower to build a search engine index compared to an internet preservation project?

An index is just hashmap of words and list of urls. So you have to parse the page, and add the urls and word frequencies to the list.

In terms of storage is cheaper, in terms of computing power is more expensive.