The problem is that even if you cache sessions, you still have to do the math to encrypt every packet. Encryption cost is a cube of key size. Please, contradict me. If you can, many cryptographers and mathematicians will thank you... :-)
Say what you will, but the difficulty of brute-forcing decryption is based on key length. You might be able to optimize the encryption using a certain length, but the key will still be vulnerable if it is only 1024 bit (as the article suggests).