I have about 20k messages in my All Mail and around 5k in inbox, and it's pretty unwieldy. Usable, but definitely slow switching mailboxes, slow at searching, slow at sorting and filtering, slow to bring up message windows and so on. It's more UI lag than anything else.
It is a very fast mail client with near empty mailboxes.