Either you have an answer or you don't. The history of the U.S. is completely irrelevant to this question. I don't live in time when blacks couldn't vote or women couldn't vote, it's irrelevant to the conversation what the U.S. used to be or used to do.
Perhaps your panties are in a bunch because you think I think you're advocating a return to monarchy; I don't think that, so let it go. I think you're advocating for a return to the 1800's view of the constitution because you think it was a golden age of freedom before the federal government got so powerful. This is a typical anarcho-capitalist view and one I also disagree with.
However, trying to prevent people from voting is authoritarian whether it's from a king or from a congress and I won't support it under any circumstance. The people would certainly never vote to remove their right to vote.