I can touch type both QWERTY and Dvorak. I usually type Dvorak, and the longer I type QWERTY, the more I get this mental fatigue of continually overriding what my fingers wanted to do. Using vim gives me a tinge of that feeling, that I wouldn't recognize as a hint of that feeling if I didn't know how the bigger version of the feeling feels after a day of typing QWERTY. It has something to do with the impedance mismatch of hitting keys in the wrong mode/forgetting what mode you're in (humans are terrible at moded interfaces) and throwing around a minilanguage for text editing in my head besides the actual task I need to do.
Saving 2 or 3 minutes of text editing a day isn't worth the aggravation.
I would say you can feel like you have a godlike sense of power over the text, like you're playing Age of Empires. Some people probably like that, and equate it with productivity and time-savings. It's optimizing for the wrong thing.