Not sure it's just an annoyance at sharing the road - like you said, there are other motorists. You don't see epic forum rage-discussions about garbage trucks holding up traffic, street-sweepers, buses, parking lanes that revert at rush hour. Those get discussed, but there always seems to be a special rage reserved by some people for cyclists - the quiet, small road users. I think you could explain a lot about a large segment of people by isolating and understanding this.

If I'm held up by a cyclist, I think "Good on them, they're braver and fitter than me."