Yes, but remember the "bottleneck" from 70,000 years ago, when the human race was nearly wiped out. At that time, because of a nuclear winter brought on by a volcanic supereruption, there may have been fewer than 10,000 humans in the entire world.

http://en.wikipedia.org/wiki/Toba_catastrophe_theory

So, based on that, we're descended from fewer people in the past than most people realize. Another way to say it is that most of our ancestors were cousins.

That overlap is more than large, it's actually the most important factor in the calculation. You only need to go back about 6 generations (200 years), and there is practically no interchange between geographically separated groups, which means that I probably don't have any genes from some random asian living 1000 years ago (although we both almost certainly share genes that we inherited from some far more ancient common ancestor).