I'd usually agree, especially as things get big.
But Kent is also pretty famous for throwing out code if things aren't shaping up. He does this in micro increments however, usually with just-written code.
I've just spent years wrestling with someone else's poorly written, ill-intentioned code, bringing it into line. I've taken the above approach of slowly reworking it. Sometimes I wonder if I just kept the tests and jettisoned large bits of it if I'd be better off?
Very contextual of course, but sometimes you have to explore a little bit to know the right places to make tradeoffs.