As someone who wears headphones basically all day at work (painter...) I 100% agree. It's a whole different experience, in a different dynamic.
On a similar note, 80% of the time I'm wearing my Sony linkbuds -- the ones with the "they have a hole in them" passthrough, even though they have the shortest battery life of the multiple sets I own.
Realistically, beyond the (not dismissing) "they are trash when the batteries die" aspect of things, taking your headphones out for 10-20 minutes every 4 ish hours... Is probably a good thing for our ears and minds alike.