Fischer refused to play Karpov in 1975 and lost his World Champion title by default. Granted, Niemann is not Karpov, and no one is Fischer, and the situation was entirely different, but this is known as counter-example. The salient point is that if Carlson has a serious challenge to his title, he couldn't refuse to play and still keep his title.