For references on the biology, read Simon Baron-Cohen or Louann Brizendine.
Regarding your statistical "argument", obviously the example of Lisa Leslie does not mean the median height of women is greater than or equal to the median height of men. Overlap does not imply equality.
I often wonder why people who can understand fine statistical distinctions in other contexts, and are capable of calculating over/under representations down to three decimal points, lose about 50 IQ points on this topic.
It is similar to the selective and highly asymmetric demands for evidence. I assure you, there are reams of papers on Pubmed concerning the effect of fetal testosterone levels on spatial reasoning. Indeed, there are many more such technical papers than there are well-controlled studies on the disparate impact effects of "competitive job postings".
This hard evidence differential obtains in spite of the fact that researchers must conduct their studies under dark of night, lest they meet the fate of Larry Summers.