Controlling who can see your content is actually a very reasonable feature; some things are not meant for everybody, but just for the people who actually know you. But it's not a use case that ActivityPub was designed for. Google+ had a really nice feature where you could easily control who can see your post. Diapora has something similar, but considering it's federated, I'm not sure if you can really guarantee it.
I think the only way to guarantee this control in a federated system, is to encrypt everything that's not completely public. If everybody has a public key, you can use that to encrypt the secret key. It's a hassle, but I think this would work.