This system can break down for some trans people (although I bet most make excessive effort to present as their preferred gender), but it also breaks down for effeminate males and masculine women (and I bet this is the larger category). Despite these difficulties, it's worked for literally all of history until people started pretending they don't know how pronouns work. Test it yourself by taking a look at Lynn's picture and asking which pronouns you'd use.

I'm not sure how invoking genetics simplifies anything. Have you had your sex chromosomes checked? I haven't, and triple-X and KS are often undiagnosed! I'd hate to have to update my drivers license to "unclassified" in my 30s.