But isn't that exactly what the Europeans did in the US and Canada, Australia and New Zealand, even Argentina? Import new people, who vastly outnumber and outvote the descendants of the original inhabitants? (And not just Europeans – it is also what China did to Taiwan.)
The fact that Russia's annexation of Crimea is only a few years old is a very large part of the challenge to its legitimacy. Similar events which happened 70, 100, 200, 300+ years ago, far fewer will seriously propose overturning them (indeed, the idea of doing so is liable to be condemned as "irredentism").
> but would the Gibraltar population really be so much worse off if they were ruled by Spain?
Many people in Gibraltar are by heritage neither Spanish nor British. The UK imported a labour force – some Brits and Spaniards, but also very many Italians, Portuguese, Maltese, and North African Jews. Many of their descendants would rather be independent, but given that's unlikely to be an option, British rule is closer in practice to independence than Spanish rule would be – the UK, being much further away, is naturally going to be inclined to give them a great deal of autonomy over their internal affairs, Spain (whatever promises it might make to try to get it back) will inevitably desire to integrate it into the Spanish state and treat it as just another part of Spain.