Hmm, that's surprising to me, since all the cards are loaded into the DOM regardless of whether or not the extension is active. The only memory footprint should be from the extension itself, which should be minimal since the code is so simple.
In fact, you might even observe a speedup because fewer elements are being painted, but the difference as a whole should be marginal.
Are you experiencing actual slowness from using the extension? I'd be really curious to know.