You need a moral framing for big infrastructure projects, or else that's how you get redlining and the destruction of minority neighborhoods for "urban renewal" and the inner-city highway system. You can't do a "cost-benefit" analysis without some sort of moral system inherent in the costs and the benefits, or else how can you calculate the effects on humans. Your "just the numbers" has its own moral system you are ignoring and instead saying everyone else isn't a cost-benefit and is only morality.

I want to build infrastructure too. Just not at the cost of the destruction of the world we live in.