Instead I think what is happening is that massive objects actually stretch the fabric of spacetime somehow so that the closer you are to the object, the slower you travel through both space and time. And the more massive the object, the more stretched space and time become as you get closer to it.
Hence if you go near a very massive object, from an outside observer it looks like you are frozen in time because time is so stretched it takes forever for you to move through it.
As you mentioned though, from any given frame of reference time will always feel the same. 1 second will always feel like 1 second.