There are exception to this when people of high income, wealth, and education moves to an area where the relative socioeconomic status is lower. A typical example is popular (often warmer) tourist locations, and in those cases we don't usually see anti-immigrant views gaining popularity. Following this we could draw the conclusion that anti-immigrantion politics is simply a proxy for politics in regard to an a influx of low socioeconomic people regardless of ethnic or religious backgrounds.
Citizens of Marseille might disagree. https://www.independent.co.uk/news/world/europe/marseille-eu...
Front National is very popular there.