> Vedas says one is born according to his past life karma.
It seems like there are some gaps in your understanding of Hindu and Vedic thought. Yes, you are born according to your past life karma, but the soul (in a body or in the other loka) always have free will – so you chose your karma – you chose to do actions that created future karmas to be resolved. By extension, you chose your birth. This is the full understanding.
> If people were allowed to change caste most people would have changed to priestly caste as it is the most easy life one can have.
You are misconstruing the abuse, misapplication and misunderstanding of the varna dharma with how the Vedas define it.
1. In the original varna dharma, there isn’t an “easier life” for one caste or another. Everyone does there dharma and that takes work. According to the Vedas one caste is not superior to another, they are equal but different.
2. Your natural abilities are the way that you choose your caste. One person has no interest or ability for the spiritual life, but they are good at business. Your natural abilities are chosen based on past karma, which you also chose.
> No father would want his child to convert to lower caste.
Of course that is so when in the abused and misapplied varna dharma. This would not be a problem if the varna dharma were applied as it is in the Vedas. A person of the priest caste would offer equal respect to the worker caste, for example.
> but still mass education happened only after their conquest.
Which has been very problematic. Education for all is good, but they have enforced Christian education and many other negative things. I am not arguing that before the British things were perfect – the argument here is how the Vedas defined the varna dharma system vs how it has been misapplied, both before and after the British. Please understand this – you originally stated misinformation about what the Vedas say.
> Assuming this is true, who has the authority to decide who should belong to which caste. Obviously the people in the highest caste will decide and they will try to main the power. Its simple as that.
Continuation of the misapplication of the varna dharma, against the teachings of the Vedas. The authority should be by one’s own volition and the community.
If the varna dharma is actually applied as it is outlined in the Vedas there is equality and justice for all. But it has been abused and used to keep certain people in power. That is not the way it is outlined in the Vedas. This inequality is wrong and against Vedic teachings.