That might be true of some complete rewrites, but refactoring - when done right - improves all the good stuff like composability, readability, and simplicity, without changing any logic (unless you count fixing bugs which it will likely uncover).

There’s also a bit of learned helplessness on some teams averse to refactoring, as in “things were always done this way, don’t touch it if it works, the opportunity cost is too large, etc. etc.”. In my experience, this sentiment evaporates when presented with the results of properly done refactoring - when it becomes obvious how much faster the team can move without compromising on quality (and frankly the work just becomes more enjoyable).