This is time that could be spent playing tournaments.
This would be okay if the world championships was every four years, but it's every two, so a large fraction of Carlsen's time is spent preparing for this one match that everybody knows he's going to win anyway.
It's similar to the way that many professional teams are reluctant to allow their players to play in the Olympics (in e.g. Basketball, Ice Hockey, and Olympic Football is completely neutered). Carlsen clearly feels that the World Championship is not important enough to sacrifice a large chunk of his career for.