What operator is used to modify the value doesn't matter, just the fact that you can do so. At the point where you use an assignable lvalue, the optimizer needs to know where that lvalue came from, which means that pointers need to be tracked like references.

My point is that because pointers act like both values and references, they are neither values nor references. This makes it impossible to soundly reason about them.