Part of the problem is there most groups can be divided into those who share a common purpose and those who share common beliefs. Most social groups fall into the latter category, whereas most project groups fall into the former. (There are also, of course, groups that buck the trend. The Xiphos developers, for example, are simultaneously both types of group and there are lots of groups that arise from circumstance that fit neither; enough, in fact, that they may be best described as a third category)

With belief-based groups, it's somewhat reasonable to use one person's beliefs as a predictor of the group's beliefs; if the guy leading your kid's youth group starts spouting racist nonsense, then it may be time to find a new youth group. However, that's not true of project groups outside of the radical left¹, where people often put aside their differences in beliefs to further some other goal. Circumstantial groups are similarly heterogeneous: just because one Redditor is a staunch /r/MensRights follower doesn't mean that another can't find that subreddit abhorrent and instead follow /r/shitredditsays.

I think the reason that making the separation feels difficult for many people is that they find themselves in far more belief-driven groups than purpose-driven groups, and don't notice the circumstantial groups that they're part of.

¹ The American radical left focuses somewhat horrifyingly on ideological purity, to the point that many gay gun enthusiasts find more acceptance in gun clubs than gay rights groups.