To clarify, the unit's own behavior is indeed not external to itself. But the way it behaves from the perspective of a user of the unit is external behavior. Changing the behavior, from that perspective, of a unit will necessarily invalidate its tests (some at least) and is not a refactor. That was my point, if you are changing an API you are not refactoring, you are changing its external behavior.

But this isn't just a problem with unit tests, it's a problem with all tests. And really, it's not a problem. It's the cost of doing business. Anything in flux (where that change changes its behavior) will force you to change anything that depends on it, whether it's a test or a user/client. If you aren't willing to discard the tests then the tests "freeze" the system in place, regardless of their level or kind (unit, integration, end to end).