Yeah its a trade-off between performance and convenience. One should know when to stop and how to slim down based on the usage. Often installing too many plugins when not using them is something that I have witnessed often with curious developers. While the real reason is installing too many plugins, the blame is on zsh ;)
Haven't done any plugins beyond what oh-my-zsh brings with it, but thanks for informing me that I should keep an eye on that .. pretty wild that we have plugins for shells, sheesh. Whats next, an App Store? ;P
oh-my-zsh comes with minimal set of default plugins. The others you can add. Check out oh-my-zsh github page for themes and plugins. I guess the comment in your plugin configuration section will also warn you about adding too many plugins having performance impact.