Do you have actual memory usage issues? Is a lack of memory causing problems for other applications?
I am asking because RAM is meant to be used, high allocations to Firefox on a (mostly) idle system isn't too interesting as a metric on its own and should automatically decrease if others need it.
I had ~1GB swapped out on 32GB 3.4GHz quad i7 machine.
5GB was taken up by ~8 tabs in FF. I noticed it because IntelliJ and Android Studio started to be suspiciously laggy. I mean laggier than usual :) After stopping FF, the IDEs went back to normal. Processor usage was negligible when I was not touching the system, so it wasn't caused by some runaway js code on some webpage.