And interesting thing about entropy is you can never make it worse - you can never reduce entropy by mixing in an independent stream from elsewhere. (A dependent stream is different, it might cancel out something else.)
So you can still use the hardware RNG without hurting anything, unless you think it's so sophisticated it's able to figure out your other sources of entropy and exactly cancel them out.