You're on to something, but libertarianism probably isn't the right target for your conclusions. This is more of an example of how movements that seek to protray themselves as structureless end up with power structures that are less responsive to their communities. See the 1970s feminist essay The Tyranny of Structurelessness[1] by Jo Freeman for a timeless example.

[1] http://www.jofreeman.com/joreen/tyranny.htm