Quirks that only work in Chrome may stop working in a future release. Things that work on Chrome and FF are more likely to continue to work.
Ideally, you'd have reliable usage numbers by browser and platform, as well as a threshold of usage that demands testing. Maybe that's 1%, 5%, 10%, 20%? Aggregate up, so if FF desktop doesn't hit the numbers, maybe FF in general does, or maybe 'other browsers' does. Or maybe commit to testing the top N platforms over 1% usage.
Lower on the usage scale doesn't need to be pixel perfect or pretty at all, IMHO, but major site features should still work.