EU citizens can freely live and work in Switzerland and vice-versa. It would be difficult to reliably cap immigration from other EU countries and stay in the Schengen Area.
Worse, being at CERN wasn't a plus for the hiring process, I would need to apply to the position as if still living in Portugal, as my VISA was tied to CERN directly with a three month deadline to leave Switzerland after the contract duration.
It also did not help, that my fellow country folks do not have a positive image across the country, for various kind of reasons, which is another issue I experienced while living there, like being refused entry in clubs when showing a Portuguese ID card.
Eventually I moved back to another EU country, still I do visit Switzerland, from time to time.
Pity that right wing movements are taking off all over the place.