Any harmonic distortion in the headphones will be introduced relative this, as well as being in the higher frequencies where it's less noticable, but still may cause hearing loss.
I've used ANC IEMs instead of proper hearing protection around loud machinery before and although it may have felt quieter, my ears still "rang" afterwards so I was causing permanent hearing damage. People who think ANC "sounds quieter" and use it instead of proper hearing protection (or just turn the music too loud) will definitely suffer hearing loss, though not directly because of the ANC.
I am not an audiologist, but an interesting similar issue happens with our eyes. If you wear tinted lenses with no UV filter, your body will be happy that it looks darker, and open your iris more. Now since your iris is open more, you actually cause more UV damage to your eye without noticing. I remember this being a big story when polarized 3D movie glasses came out and people were keeping them on outdoors.