There is no demand for modifying freedom of movement within EU. It’s not even a topic in most EU countries.
What IS a topic, is preventing non-EU migration, and that has broad support and slowly all parties are moving in that direction.
And we are NOT EU. But for now, they basically go „yes yes, but we think of you as EU because we are so tightly connected“.
So what do you expect to happen if we push the point and make them treat us as non-EU?