I feel like you're missing the point... Nobody eats lunch at literally the same time, but 2pm for me is not the middle of the night for you. Those are two completely different times of day. In the current system, because we both have a common understanding of when 2pm is, it is easier for us to communicate.

The alternative you are proposing doesn't solve anything: I still have to convert from "my 2pm" to "your 2pm". I still need a lookup table, but now the actual time itself gives me no information. It's a downgrade.

Here's a question for you: in your system how do you communicate what "time of day" it is at your location? When you arrive in a new timezone, how do you know how much to adjust your schedule by?