What's more, if you can't vet who joins the channel, then you can still have eavesdroppers listening in. So it's really no more secure than IRC over SSL (in fact probably less so because at least with SSL, eavesdroppers would need either access to your channel or to the server. So PMs and private channels are secure. With this, everyone shares the same certs so even your PMs are at risk).

> I don't see any mechanism in place to prevent the server from replacing that public key with their own

I'm not sure that's possible without exchanging the cert via peer-to-peer. In which case, you've already solved the toughest bit of the chat protocol (the handshake and coordination across the clients) so you might as well go fully peer-to-peer and do away with the server entirely.