Suppose, as you do, that your body is not who you are. Then your identity lies solely in your brain. But if that's the case why would disphoria even exist?
If you feel like a man but you don't look like a man, why could that cause any identity problems? It must mean your body is actually part of your identity, so changing it changes who you are