Part of the IQ test includes vocabulary, and it's not that much of a leap to say that more highly-educated people have a broader, deeper vocabulary, even when controlled for cultural effects. A smaller vocabulary will give you a lower IQ. There are likely to be other factors affected by exposure to wider experiences than you might not get when impoverished; for example, in general, the more you handle numbers, the better you get at manipulating them.
Highly educated people might also have higher innate intelligence, which allows them to pick up words easier and develop larger vocabularies. Let's not forget that IQ has been shown to be significantly heritable by adoption studies.
