The only people I see obsess over these metrics have either their ego or pay tied to them.
> Do you actually go interview users on what they perceive?
Yes, regularly, because that's what effective teams do. I also limit bandwidth to 3G and record the sessions so that everyone can share in the learnings, which very rarely circle back to things like TTFB/TTI.