Well I think my skills suffer more because of work: after a day of "being productive" at work, I no longer have the time and energy to explore and learn new stuff, which is how I gained and expanded my skills in the first place.
Yes, the extra time you get from the chill job can go either way. It's whatever you make of it. If you use the time to learn, you can definitely learn more at a chill job than at a hard job.