You're right in principle, but the point here is about the reasons for rejecting a casual model. The issue people seeking fairness in statistics run into is rejecting models based on what ought to be, instead of what is. A casual model can be totally unfair, and yet also correct (insofar an approximation is considered correct).

Taking the example from our parallel discussion, if the data says being male is correlated with risky driving, and it seems to fit the casual model of "male -> risky", it would be wrong to reject it just on the grounds of "we're using this model to set insurance rates, so by penalizing males, the model is sexist". It may be that you can come up with a better casual model explaining the correlation - say, cultural history and path dependence - but until you can, rejecting a fitting model based on "it's unfair, reality ought not to be so" is just wrong.