I would argue this is not quite the problem you make it out to be, because generally speaking, people tend to be strong or weak in a subject overall.
If you're generally good at math, getting tripped up by vectors is not a death knell for pursuing coursework at that level. But there is a massive divide between the ability of someone "generally good" at math and "generally bad" at math. Tests do a really good job of capturing that, actually.
