I was in Mountain View and my manager was in NYC, so we started working at the same time, given the 3 hour time difference.
Also, the manager of everyone in the ocean of people around me was my peer, we had the same manager, and he came in almost as early as I did.
So, no problem, or at least no problem that I worried about.