I'm not sure if BitTorrent is a reasonable solution to the problem of P2P video chatting, since the power of BitTorrent is in having many nodes all sharing the same content and able to send parts of it to a new node seeking that content.

Realtime chat doesn't seem amenable to that type of optimization.

In any case, I don't think the problem here is primarily in the technologies being used; I wouldn't exactly call real-time video chat a solved problem (I'm sure there's room for plenty more optimizations in it over the next few decades), but at least as I understand it, it's definitely one for which there are plenty of available open-source stacks available.

The problem is in a) designing a robust protocol/API that can be implemented by many clients and servers (that fulfills my basic requirements above of "basically Discord, but not controlled by one company"), b) writing a solid reference implementation of both the client and the server, and c) (perhaps most importantly) getting other people to adopt it so it's more than just Failed Competing Standard #27.