No it's noticeable. There is no hesitation in the responses. I recently switched because I setup something for Emacs which needed to run a shell on start up and I noticed just how slow it was because Emacs is not Async and would hang for that shell to boot to run a simple command.

On a side note, while prezto is faster, another culprit of slowness is nvm which when I disabled it made the terminal much faster to load.