Performance has always been a finger in Ruby's eye, but with 1.9 that doesn't seem the case.
Regarding libraries; you are right. Ruby has a lot of code out there, but much of it is in random repositories maintained by random people. Hopefully this changes.
I'm not convinced that's true, as the depth and quality of rubygems are inferior to the depth and quality of eggs available in python. Find me e.g. a "gem install scipy" equivalent for Ruby.
However - "easy_uninstall scipy" REALLY needs to work!
Rubygems is a separate system from deb/rpm, but works roughly the same way. Typically it doesn't do binary packaging the way they do it, though, and if native (non-ruby) code is required it is compiled during the install.