http://bigthink.com/series/62#!selected_item=4845
He starts at 8 mins in, and although the whole video is 2 hours long, he makes a bunch of major points in the first 3 to 4 minutes of his talk, after which he answers questions for 3 to 5 minutes.
His central point is that there is a lot of potential value in getting the fixed costs of the search business down, and that he would prefer to invest there, rather than in playing the zero sum game of trying to take away make share in the $25 Bln search business.
Also worth noting: anybody who can lower the fixed costs of search by an order of magnitude is also going to be in a position to make money from businesses that have similar cost structures.
Indexing is more complicated. You could argue that 20% of Google's index contains at least 80% of the information people need. The problem, like the old advertising saying, is figuring out which 20%. So if you have a clever way to address content quality and uniqueness, suddenly your crawling and indexing costs plummet.
First, the bulk of the search seems to be for the usual stuff. Thus the bulk of revenue from ads for the provider and _perhaps_ the bulk of added value for the users stems from searching for the usual stuff.
Second, as the obscure stuff oft uses uncommon terminology and names, it's technically easier to search for it and display relevant results. The trick is to give useful results for the most common things. Consider how hard it may be to automate (!) picking sensible results for words like `the' or `free'. Or for dates that are represented in one zillions of formats. Of course various approaches (like lists of & criteria for stopwords) have been implemented.
Naturally this holds true for general search engines. Vertical ones are a very different beast. And I think Peter Thiel was not talking about specialized vertical searches like patent search.
To go back to the Blekko example, I'd say it's pretty clear you couldn't build something like that with $10-15 MM (assuming they've gone through maybe half their funding) a few years ago.
