I think people on both sides need to have more empathy, then. I'm generally one of the prompt people, and I'll try to start on time. If people are late, they'll arrive after we start, but that's fine.
And the late people need to understand that sometimes they will miss the beginnings of things, but that's ok too; their inability to be on time (for whatever reason) should not waste the time of those who get there on time.