The ability to take leisure is still conditioned on labor. To take time off, you have to save up. To save up, you have to work. Labor is distorting the real value of leisure if the production of that leisure needs less labor (leisure time still requires you to be fed, housed, and more if you go on holiday, travel, or consume to keep yourself entertained or educated).