I suppose you could make the same comparison with whites as well, I get that there is somewhat of an issue there because it's so inflammatory to discuss whether these things are fair or even effective at helping people etc.
I think the discussion should be had anyways. There might be some value that comes out of it. Like in my own case, I'm on the losing end of these policies in theory despite having very serious disadvantages in my life that aren't on the list of things to account for. Broadening the definition of diversity could alleviate feelings people have that they're not being treated fairly. But because we aren't willing to have difficult conversations about this stuff we have a supreme court case about it instead.