First, there's pretty obvious harmful effects of religion: on the minor end, feeling ostracized if you happen to be different than what the stated beliefs say are correct, i.e. being queer in the Christian church and ostracized or driven to suicide for it. (Or even more fundamental issues, like taking issue with men being the head of households.) On the extreme end, you have crusades, pogroms, wars that engulf nations. Obviously relatively harmful. Or even on the individual scale, ritual killings of those who defy the religious order.
I'd also posit that, again from the Christian faith, believing there's an all-powerful god who will sort out justice in the afterlife leads to negative outcomes here on Earth: believing you'll have everlasting life in heaven is of course going to limit your investment in systems here on Earth. I've had religious people cite (paraphrased) that "god will sort it all out" when I ask why they don't push to directly address more of the problems in the world. There's an abdication of responsibility I've found relatively common amongst my religious family on such issues. (Not universally, notably, but still.)
Regardless, for those who have "lost faith" on epistemological grounds, what is the alternative to irreligiosity? I don't "choose" not to believe in the Christian (or any other) god, I'm unconvinced by the evidence put forward by their evangelists. I have no philosophically sound alternative to atheism unless and until I encounter convincing evidence to the contrary.
I also worry about people "giving up" some of their philosophical beliefs just to belong to that kind of community. I've talked before [0] about how other more extreme communities (e.g. flat earth) aren't so much comprised of people who believe the premise of the group so much as people who _want to belong somewhere_.
> But irreligiosity does not scale
What's the alternative? The atheist segment of the population giving up portions of their ethical and philosophical autonomy to churches in exchange for a sense of community? Continuing to enable the hold of said autonomy over those who believe largely because they aren't exposed to other epistemologies because their community center discourages it?