The comparative difference is closer to the latter than the former, actually, though neither is complete and the reality is more complex: the difference in racial stereotypes is not mostly due to real differences in the groups to whom various racial identities are ascribed, but due to different perceptions that are a combination of deliberate engineering (in the case of Asians, by elites who had a positive interest in importing their labor to undercut domestic wages for the "hard working" bit -- in the case of persons of African descent, by elites who had an interest in keeping them in "their place") and different responses of not-necessarily-representative subsets in often non-equivalent circumstances, plus subsequent reinforcement through confirmation bias.
Sure, there are plenty of hard working, intelligent persons of asian descent in the US -- also plenty of hard working, intelligent people of hispanic or african descent in the US.