And casual keyboard-commands just don't scale well. Moving cursor x chars means you must type the movement-key x times. With vi you just type the number x and the movement, and usually save big time.
Aditionally, vi has more knowledge about textual objects, which are all interchangable. So whether you move 3 chars, words, paragraphs, pages or functions, it's all the same, just with different letters.
Vi's command-language is highly composable and thus very dense, so it will alwys be faster than mouse and casual editor-commands. Of course as long as you habit them.