The fancy crypto is what allows them not to (effectively) keep those logs in the first place. If you build this feature without the fancy crypto, then even if you say you're not logging this stuff, you (effectively) are, because your system depends on durable access to that metadata in order to function. This is, for instance, the major security difference between Signal and Wire.

One strong indication you have that Signal isn't logging this stuff is that they had to wait until they were able to advance the state of the art in anonymous credentials in order to implement group access control at all.

> You either have to trust the server operator not to keep logs

You don't have to trust, you can verify. This has been proven in court: https://signal.org/bigbrother/eastern-virginia-grand-jury/