You were doing it right the first time. Vim's "insert mode special keys" are basically incredibly basic terminal controls that are the same as Readline anyway (like backspace, or ^U to delete to the beginning of the line), or incredibly obscure things like inserting the contents of a specific register. Vim doesn't expect you to move the cursor much in insert mode, it's just for typing.
As it turns out, the extra Readline bindings this emulates are basically emulating Emacs' standard bindings, so if you want those bindings in an editor, you know where to look.
Personally, I just stick these settings in ~/.inputrc:
set editing-mode vi
set keymap vi
show-mode-in-prompt
If set to `on', add a character to the beginning of the prompt indicating
the editing mode: emacs (`@'), vi command (`:'), or vi insertion (`+').
The default value is `off'.
Chet accepted a patch I submitted for custom indicators, and this will be coming in bash 4.4/readline 7.0: http://thread.gmane.org/gmane.comp.shells.bash.bugs/22580/fo...
I have patched versions of bash 4.3/readline 6.3 available on github (the bash page has gifs showing the functionality in action): https://github.com/calid/bash https://github.com/calid/readline
Note that the '\m' escape hasn't been accepted into the mainline yet, so as of now the custom indicators will still only display at the beginning of the prompt in the new versions.
The general solution is to just tap Esc a couple of times to ensure you're in Normal mode and continue from there.
There is an [episode of vimcasts](http://vimcasts.org/episodes/modal-editing-undo-redo-and-rep...) which compares editing in vim's insert mode to making a brush stroke in a graphics program. With this plugin one must be careful not to make the brush strokes too long.
:-)
Ah, well, whatever peels your banana...
What, more than usual? How could you even tell?