It’s physically harmful in the long term, can lead to serious vascular problems if left untreated.
In fact it turns out that REM sleep can sometimes compensate for apnea in some people, that is, the onset of REM can happen more quickly. Some people can get enough REM sleep even with apnea.
To be clear, ultimately all of these things can become problems, I’m just disputing that lost REM is the main health concern. It’s one concern, but AFAIK, in the long term venous damage is what will kill you.