I guess I'm the opposite, probably because I've learned to use Vim not as a standalone editor / IDE, but as a preferred set of keybindings and editing within other environments. I hear a similar argument to yours all the time - specifically that Vim as a plugin is not quite enough - but don't fully understand it.
Essentially, to me it's all about ergonomics of editing. Hands straight on the "home row" of the keyboard, and with modal editing, almost everything is a single keypress away. To me, this feels a lot faster and easier than reaching for the mouse (or arrow keys) every five seconds, and I get the additional benefit that my core editing commands stay consistent across different environments (where I have a Vim plugin, that is). I leave all the fancier, IDE-specific stuff (e.g. refactoring, etc) on their default IDE bindings, and learn them. It's a good compromise, I'm not sure why people don't like it.