I fail to see the critical distinction between refusing to associate and refusing to engage with, or how the freedom of association can be upheld without the ability to refuse such engagement.
We must provide social media services with protected freedoms, or we risk impinging upon the freedoms of its users. Users must be free to choose social media which allows them to _not_ associate or communicate with those they do not wish to. Nor should we compel them to support speech that they do not wish to, for similar reasons.
If your concern is whether certain legal speech may be effectively completely deplatformed, then may I suggest the solution is federated social networking and not to impugn the freedom of association.