Prior to the civil rights movement it was assumed that by freedom of association, it was beyond the scope of the government to prevent things like segregation by private citizens. e.g. if a business owner did not want to serve black customers, then that is his right. So yes, someone who does not support the government having that power to infringe on freedom of association will not support any government intervention to prevent voluntary segregation.

This is why he distinguishes between the civil rights movement before the Civil Rights Act of 1964 which was only concerned with preventing government enforced segregation.