I would care if my coworkers were bigots, and I would especially care if they were prejudiced against me, how can you effectively work with someone who doesn't respect you? Pretending bigotry doesn't exist is the same thing as pretending that race/sex aren't visible. You will also have to apply the same affirmative action when promoting people, as a couple of years work experience doesn't erase the generations of privilege you're competing with.

I never said that minorities can only be hired by applying insufficient scrutiny. What I will say is as a group, minorities can only be hired in a proportion that is different to the proportion of minorities in group of qualified applicants if some weighting is taken off the job relevant qualifications, and applied to their membership in minority groups.

If you could come up with a huge list of entirely fungible, interchangeable resumes and interview notes, you could consciously pick minorities first without harming the quality of your recruiting. What kind of roles would have that weak of a job market though?

If candidates aren't fungible weighting any importance to race/sex means taking some weighting off something else, and it means that bigots will assume individuals from minority groups are less qualified, because of the fact that the group as a whole is less qualified, because of your own policies. It perpetuates both inequality and bigotry.