It would go a long way toward distinguishing between those two possibilities if a computer judged the faces - find the eigenface(s) of different mathematical ability levels and see if there's a lot of correlation between those and ethnicity, or if features alone are the predictor.
Even if it is features alone though, I would expect a major involvement of education and self-fulfilling expectations: take a starting state where there is no correlation between facial features and math aptitude. Everyone would have their own beliefs about what faces are good at math (humans always see patterns). Some of those beliefs would happen to be similar, so people with certain features would be steered more toward math. As time passes, the bias becomes more legitimate and more self-perpetuating. The end result is a socially-imposed link between particular visible traits and ability in math.