Think of it this way... in your lifetime of genuine interactions with people (e.g. conversations), how many XX people have you mistaken for XY people, and vice versa? For most people the answer is very close to zero, but why is that?
I would argue it's because billions of years of evolution (a biological process) have already helpfully trained us to detect even the tiniest differences between males and females. We have been a sexually dimorphic species for a very long time. And those differences (whether massive or tiny) are directed in large part by the chromosomes inside our trillions of somatic cells - skin, hair, bones, eyes, brain, etc. etc.
Also, for those rare cases you mention, the chromosomes often are a major deciding factor in how we do the classification, e.g. we might be confused about someone with a DSD because although they were "assigned female" since birth, they continue to show some male characteristics, due to the Y chromosome present in their organs, skeleton, etc. Surgeons can operate on genitals etc., but they can't alter all of the trillions of somatic cells in the body.z
