Main thing for me is size in the hand and squared off sides. I find anything with rounded edges I can't hold on to and just hits the floor constantly... little better with a case, but that makes the phones just that much bigger and less comfortable. I can't bring myself to let go of the old SE's.... of which I keep two around just in case.
I found the iPhone 5 the slipperiest phone I’ve ever held. The texture seems designed to minimise friction between the phone and your hand, and the square edges make sure as little as possible of the back is in contact with it too!