The "distortion" you're hearing is actually just the leftover noise. When there's noise and speech at the same time and frequency, then you can year some of both and it sounds a bit harsh -- like distortion.
In general, this problem is tough because when there's a change in the signal, you have just 10 ms to decide whether it's the noise or the signal changing.
