I'd also be interested to hear Thomas clarify this. I saw a recent thread on Twitter where he and bascule were talking about it and it still wasn't super clear, but one specific point I recall is that Matrix has a significant amount of metadata stored on the server side which constructs a social graph. As opposed to something like Signal which has close to nothing stored on the server.

To me this seems like an issue of use case. If my goal is to be able to talk to my family and friends, and I don't care that it's known that I'm talking to them as long as the contents of the messages are private, that is fine for me. For a case with more stringent requirements, I can see Matrix not being a good recommendation in its current design.