I am going to give Buffett the benefit of the doubt in terms of what he meant. Not that it was a completely clean slate and no one was here beforehand, but that the economic productive output that existed here before (beyond bare resource value) was not assimilated and did not contribute much to present day economic output. For all intents and purposes, it was "starting from scratch". Contrast this with, say, the Roman conquest of the Greeks, which was almost a kind of cultural conquest of the losers over the winners.

This is not to minimize the genocide of the Native Americans that occurred, or the horrors of slavery. But in an economic report looking at compounded returns over the history of the US, starting from scratch is a fairly reasonable assessment.