Not irritating at all. You are saying that you can't swap the pointed at object for another one AND that you can't change the pointed at object. Two very different things.
You have four possible combinations all with different meanings.
It's also why I always I insist on putting const to the right of the declaration part the const is referring too. The C++ standard allows const to the left IF it is the first const. But I think this flexibility is kind of garbage....