For a 60" TV that sits 10 feet away, you're not going to see much difference. Move that TV to 4 feet away, and you'll appreciate the extra pixels.
For virtual reality, and other applications that put the screen much closer to your eyes, the extra pixels will make a huge difference.
It's all about the viewing distance.
For a 6" screen, that works out to ~1800 pixels. So maybe not 4k, but the next step down from 4k generally is 1080p, which is less than the figure I gave above.
So yes, there is a point.
(And that's not getting into other factors either, such as panning and stroboscopic effects.)
I absolutely love the look of the crisp text on the Nexus 6, but I don't think that it would be very visible in most video content.
That's not what VP9 or HVEC are good for anyway, so I would bet that scaling down 4k video or scaling up 1080p (for the same bitrate) won't make any noticeable difference.
But of course that's just a matter of opinion unless we do a blind test with real use cases.