The question is whether coffee is acidic enough to do damage even without the sugar.
Tooth decay requires several parameters 1- bacteria 2- bacteria food source 3- acidic environment 4- time
We all have the bacteria but the other ones are somewhat controllable. What many people don’t account for is the 4th parameter of contact time. It would be better (for your teeth, not overall health) to rapidly drink several sodas than periodically sip one for several hours.
Decay typically only occurs in a pH environment of 5.5 or lower, otherwise the enamel is resistant to the acid. So the longer the pH is lower the more likely you will develop a cavity.
Also: it has definitely stained my teeth. Is there a way to remove the stains that doesn't weaken the enamel?
People often drink coffee slowly and without a straw, this leads to staining but not tooth decay. However the slow part contributes to the teeth not getting a chance to neutralize acidity normally. This gets worse when people then brush their teeth after their morning coffee to prevent bad breath, which the abrasiveness combined with the acid environment does cause havok.
Both of them can be avoided in a variety of ways, to reduce both staining and decay.