I think that makes sense.
When another group with (possibly) different norms has no affect on you, it's easy to ignore them.
When that group can exert some effect on you, but the effect is unknown, people get scared, and act accordingly. This is exacerbated by people that like attention and/or power that will stoke these fears to further their own agenda.
When that group is finally encountered, often there's understanding, a willingness to compromise where needed from both parties, and as the unknown subsides, so does much of the fear (and the power that grants those that use fear).
