This is definitely a pain you'll feel with rails, and any other combination of software pieces which you bring together.
For what it's worth, within the last month, I've touched rails 3.2, 4.0, 4.1, 5.0, and 5.0.1 apps which are running in production. It's possible to maintain these apps on old releases, but the idea of never upgrading will be painful in the end.