You absolutely do want to do dithering. Dithering converts distortion (error correlated with the signal, which is bad) to noise (error uncorrelated with the signal, which is less bad). This means that even though the noise floor is higher, you can recover more of the original signal. There is virtually no case where that is not desirable.

You're right of course that it will compress less well, but that's to be expected because you've lost less information!