People are likely to use a standard English dictionary. In my experience (which is exactly within this field) people use a fairly tight subset of the English vocabulary.

So I would be quite happy to test for a dictionary of, say, 100,000 words and be hopeful of a good hit rate (note that XKCD says common words, which is easily missed)

Our software has a test (which runs about third in its list of tests) which does dictionary combinations up to three words (two words is quite a commonly used password based on our statistics) with a dictionary size of <s>175,000</s>17,500. (Edit: sorry, apparently it is an order of magnitude smaller, I checked with one of the engineers :)) This includes English words plus a few commonly used foreign/slang terms. The hit rate on this is fairly high.

(we crack document/windows passwords mainly)

You could of course choose deliberately obscure words to invalidate this - but they aren't so easy to remember (so people will tend not to).

If someone is going out of their way to secure a password, sure, you're going to hit a brick wall. But what every password scheme tends to forget is the "human factor" whereby people not concentrating on being secure will introduce attack vectors.