THe biggest difference that most people hear is EQ. (oh these are very bassy, or too clean, etc, etc)
The people that have external DACs are almost certainly hearing a difference in EQ rather than _quality_. Is that a problem? for me I couldn't care less. However when that starts bleeding into advice or gatekeeping, then it becomes an issue.
(I am a former sound technician for both recording studio (analogue and digital) theatre and TV)