> E2ee helps mostly against server owner, and you eliminate this risk by being your own server owner.
Unfortunately you only eliminate it by being your own server owner _and_ your recipient being their own server owner. Take a look at email: I might not want to use e2ee because I self-host, but the second I send an email to a friend hosted on gmail, Google gets all the content.
I think federation does have its place (for different reasons) but it unfortunately isn't enough for privacy.