It's not ideal but in order to make utilities that work on every page, these permissions are needed.
The code inside the extension bundle (.crx) would need to contain the potential for abuse, and if it gets popular enough, security researchers will look at it. Even if it's not popular, incentives will be at work, because it would be a foolish risk for a company to ship code that could expose a user's entire browser history into the extension, because at any point someone could take a look at the bundle and find the flaw.