Let's take visuo-spatial ability. Men do better in testing than women generally. So for arguments sake let's say they are correctly measuring a deficiency.

So now the question is, is this for cultural reasons or genetic reasons? Laszlo Polgar was a psychologist who raised his daughter to train on visuo-spatial thinking, mostly with chess. She became the youngest grandmaster ever, the youngest person to break into the FIDE 100. She became the 8th best chess player in the world. It's anecdotal, but if she is capable of this, presumably other women are genetically capable of this level of visuo-spatial thinking, so the tested deficiencies might point to culture over genetics.

As Gould said, it's not just IQ but the blanket if assumptions around it - it is genetic not cultural, it is not remedial etc. As I said, people don't get this worked up over light year measurements.