The API for web components isn't that bad. You have to stamp DOM templates yourself using whatever process, you have to handle attributes and properties yourself, and that's about it. Lots of boilerplate, but a library like Polymer can take a simple, opinionated approach to handle it for you.
From my experience Web Components have been pretty pleasant to use with or without Polymer, it's just a matter of wiring everything up if you go vanilla.