That actually is a good point I hadn't considered; I think it would still make enough sense in the local context ("tomorrow" is "the next time most of us will wake up"), but yeah, it adds a lot of confusion for short term relative terms...
I could maybe argue "well, 'yesterday' and 'tomorrow' aren't very precise anyway - you need the entire context of the conversation to understand what they're referring to", but they're well established and understood... Darn.