It is absolutely a label that is applied to people. Sitting here, I feel like I am part of the specter of brogrammers that looms over the industry. Whether it was their intended usage or not, I feel like this term is referring to programmers like me.
Their job ad is not going to have the context of this blog post. Their job ad doesn't say "no brogrammers, but we don't mean men who work out and code, we actually mean the toxic culture of exclusivity that you would see in a frat." Nope, their job ad just says "no brogrammers". If I saw that I would assume that their culture would be toxic toward me.
Please stop using this term. There are people out there who feel like it applies to them and it alienates us. I agree with OP, it is not a term that encourages inclusivity because there is a stereotypical image of a person that comes to mind when it is used.