The short answer is law. But even with a good system, it seems very difficult to maintain.
In the U.S., the idea was that its founding legal document, the Constitution, would put some very strict limits on the power of government. Thus, there would be many things the government could never do, like restrict one's freedom of speech or unreasonably search and seize a person's belongings or effects. A process was included to amend the Constitution in the future if that was ever necessary, but as one would expect, it takes a very large amount of agreement to do that.
Under these strict limits, three different branches of government would often work adversarially to keep one another in check. The Congress would make laws, but it had no power to enforce them. If the Congress was not happy with the execution of a law, they could repeal it. The President would execute the law and set up a system to enforce the law, but could not change the law and was not given the power to interpret the law or determine whether the law had been broken. The judiciary would interpret how the law applied to specific situations, determine whether law was being followed, and make sure new law complied with the country's ultimate law.
In theory, each branch doesn't have enough coercion to abuse it, and the other branches can stop the excess of a single branch.
It appears, however, that currently this isn't the case in the United States. The executive branch interprets the law to mean whatever it wants the law to mean. Then it claims "national security" at every attempt to subject those interpretations to judicial review. And it doesn't tell Congress its interpretation of the law or how it's carrying out the law. The Congress and judiciary aren't acting adversarially enough to restore the desired balance
Can this be prevented or corrected? Is it really a workable solution? I think we may be about to find out...