It's a speed upper limit, not a minimum. Are you upset when an agricole machine is using the public roads?
This "annoyance" against people not travelling at the speed limit on public roads is based on a lack of time management and the expectation that the road is empty. This is a selfish point of view.
When I want to control my travelling time, I walk or cycle. When I want comfort, I use public transportation or my car.