This is entirely supposition on my part, from someone far too young to have lived through any of it, but I suspect it's less that Japanese Americans looked different, and more that Japan was a non-Christian culture, which made it easier to dehumanize people of Japanese descent.

The Nazis might have been evil but at least they were "like us" in that they shared a common heritage, religion and linguistic root with Americans. Japan, meanwhile, was portrayed in American propaganda as an inscrutable hivemind run by a primitive death-cult.

You can see the same strange mistrust of non-Christian culture applied to Muslims in America today - even though the vast majority in the world are not violent terrorists, many Americans suspect that Islam taints and "radicalizes" the mind with evil in a way that Christianity doesn't.