First of all, I must clarify that I didn't truly address your initial statement, that "all" ActivityPub elements should be E2E encrypted, which is a notion that I find absurd, because it increases the complexity at least an order of magnitude and except "Create", no other activity in the specification has side effects that pass "content" between servers. In my opinion the only part of an ActivityPub request that can be sensitive and could require encryption are dereferenced objects inlined in a "Create" activity.
What I addressed with my (granted too terse answer) was that E2E communication is actually possible, but it would require one of the following options:
A "Create" Activity can wrap any type of object that has a JSON-LD document describing it (the ActivityStreams vocabulary is extensible[1]) so adding a new "E2EEncryptedMessage" object that contains all the properties required (and which are not already exposed in the existing vocabulary, eg, public keys which exist for the actors that participate in the exchange) in such a communication is possible. As long as all the information required for E2E can be encoded fully at each communication step (ie, nothing needs to be communicated off-band) everything will behave as expected for the clients that support this protocol extension.
Another option would be to piggy back on existing Object types like "Document" and encode a full E2E payload in it's "Content" to obtain a similar effect but with less expressiveness from a ActivityPub point of view. This probably needs a custom "MimeType" to accompany it so other clients can avoid processing these objects.
In conclusion, that's what I meant by having ActivityPub behave like "wire protocol", and apologies if I created confusion.
[1] https://www.w3.org/TR/activitystreams-vocabulary/#extendedty...