OK, I'll give an example. My team leader in a previous job used to regularly turn up at 11am (officially we were supposed to be in by 9.30, but it was academia and no one complained too much). I was usually in before 9.15 - like the majority of the staff.
So sometimes I would need to ask him something, and he wasn't around - that would hold me up.
Sometimes other people would need to ask him something. He wasn't around, so they asked me instead - usually when I was trying to get in the zone.
I am all for giving staff flexibility but it should not come at an expense to productivity.