Damore's manifesto purports to present evidence that there "maybe" are inherent differences between genders ("may" and "might" are also sprinkled all over the place for good measure.) He even acknowledges that this is far from a complete picture. But then he goes on to conclude that:

1) That must explain the existing gender distribution in engineering

2) Google trying to explore doing outreach to increase diversity must necessarily imply discrimination of the unfairly oppressed white heterosexual male

So trying to dismiss all criticism by splitting hairs over whether he meant that the average, the median, or all women are uninterested in engineering completely misses the point that either way the conclusions are unwarranted. The whole thing smacks of persecution complex first, finding "evidence" later.

1. Saying that women are "on average" worse programmers is exactly why people are upset with him. He doesn't need to be some kind of insane absolutist in order for what he said to be very problematic.

2. Even if there are statistically significant sex differences with regard to programming ability and inclination, there are also historical and sociological factors that are working against women in tech. Damore's memo was explicitly a negative reaction against some of the programs designed to counteract these systemic issues. As such, even if he were 100% right about the sexual dimorphism (which he isn't, see my other post in this thread), using that as an argument against trying to solve the sociological problems is not only logically unsound, but exhibits bad faith and bad motivations.