What's the message format you're using for RSA? I see RSA in the "pq" code of your Java client library; I see you asking Java for an "unpadded" instance of the RSA encryptor; I see you using RSA in its encrypting operation (not signing).

Great crypto software avoids RSA. Good crypto software uses OAEP when RSA is used to encrypt. Crappy crypto software (see: DNSSEC) uses PKCS1v1.5. What are you using?

I didn't look carefully; I have probably missed something. Please explain how you are using RSA in the handshake/authentication for your protocol. Like Moxie said: it's not the underlying algorithms you use; it's the constructions you use to to them together, and to the data you're protecting.

Incidentally, "I refuse to respond to comments from people who have only looked at my system for a couple minutes" is not a habit of highly effective cryptographers.