If you know your requirements up front, then it's not hard. In fact, it's a good exercise to do so -- probably one reason why messaging systems are a dime a dozen nowadays. What's hard is making a messaging system that is everything for everybody. If SIP or XMPP has taught us anything, it's that "extensibility" is a killer. Requirements evolve, and often it is perceived as easier or "cleaner" to start from scratch, rather than work around the idiosyncrasies of existing systems.

On the other hand, if you are trying to design a protocol to solve a domain specific problem (eg, in cryptography, multimedia, distributed systems, etc) then it becomes more of a research endeavor with all the associated pitfalls. In that case, there will always be the "shoulders of giants" to build off of. Unfortunately though, for every pioneering TextSecure, there are a dozen CryptoCats repeating the same mistakes of years past...