The VR technology tries to eliminate any real world point of reference which makes us use reference points inside the VR which, I believe, is one reason the experience for us gets disconnected.
The most common symptom of this is that you get nauseous. My guess is that the (sensory) disconnected experience makes us lose balance.
What would solve this problem, I think, is to make the VR work with the whole body. So wearing a suit with goggles and standing on a plate that moves as you move which gets translated to in-game movement.