Why? The work you do in your spare time can directly (by reusing libraries you build) or indirectly (learning a new language, framework, etc) improve your performance at work.
Only reason you should ever feel guilty is if your side projects are meant to directly compete with the work you do at work. In that case, I would say it's a bit unethical.
