In practice it works by each device having their own encryption key and then those devices are bound together with a cross signing key, so your peer can robustly identify all your devices at once (and the list of devices can change as long as they are bound by the cross signing key). Certainly the server is able to correlate device ids (and thus keys) and IPs.

The way threema does it sounds a bit how room encryption works in Matrix amond multiple clients.