But you don't have to actually know them. If they posted their Tox ID on 4chan, which happens all the time, and they accept my friend request, I would have their IP address, correct?
Ip addresses change. Sometimes they don't even work for P2P. Without ICE or something its hard to even know your public IP address programmatically. So often some kind of proxy or server is necessary for most people. True decentralized connections are a sketchy proposition.