I agree overuse can be distracting. But I try to think of it as flattering that they respect me enough to take the effort to carefully vocalize their thoughts rather than blurting out the first thing that occurs to them.
This doesn't always hold, but a trend I've noticed at work that someone talking up the chain of command is more likely to use filler words than when that same person is talking down the chain of command.