I'm with you about not wanting the autofill, but I use the key combo mentioned below in nearly 100% of cases.
The vast majority of the time I'm opening the extension popover it's because the key combo failed to autofill (site doesn't support it) and I need to manually copy/paste.
For extra fun - the Key combo is customizable if you don't like ctrl-shift-L
Just hit up chrome://extensions/shortcuts and change the combo to something you'd like.