You're right. Let's completely change how humanity has done it for centuries so you and your friends don't spend 32sec looking up "current time in X" 2 or 3 times a day.
You would still need to do the effort of understanding what 10pm means for the other party. 10pm may be your lunchtime while it's the middle of the night for them.