For the extension to handle prompting you before you go to a site, it has to know you went to a site and then show the prompt on it.
It would be nice if browser permissions were expanded on in greater detail with newer apis to handle different cases (and if Firefox and Safari could join in supporting unified standards here for privacy reasons), but for now such permission is required.
Chrome offers the ability to limit that per designated sites.