That's a good point. You do need something to do the negotiation. However this is not an intensive task and there are a handful of approaches that can avoid needing a dedicated third-party.
1. IPFS PubSub can be used for sharing this info (although you do still need to bootstrap the IPFS DHT).
2. You can share blobs over text chat. (Including services like Jami which are distributed)