It's worth distinguishing because the needs/desires of expats and immigrants are different.

I'm an American living in Tokyo. There's a fair number of American expats in Tokyo, usually high-level managers of international corporations. They socialize mainly with other expats at places like Tokyo American Club[0], don't learn Japanese beyond the basics, and send their kids to "international schools" where the curriculum is in English.

However, as an American immigrant in Tokyo, my social group and goals and so on are different. Immigrants learn Japanese, socialize largely either with locals or other immigrants, and their families may speak little (or no) English. Our needs involving banking, legal services, licensing (e.g. driver's license), and housing are different from expats -- they care about rent at high-end furnished apartments, I care about rent of normal apartments (short-term) and mortgage availability (long-term).

  > I mean if this definition was applied consistently, I
  > would be happy to use it. But when speaking about e.g
  > H-1B visa holders working in the US and taking jobs or
  > raising house prices, they are called immigrants
  > not expats.

It's easy to twist yourself into knots trying to understand thought processes of people who don't think. Someone who calls H-1B holders "immigrants" and blames them for housing prices hasn't put any thought into their position, so you'll only do yourself a disservice by trying to meet them on their level.

[0] https://www.tokyoamericanclub.org/index.php/en/