I think you missed the part where those other, less-productive hours in my workday are spent in meetings with others. As in, that's part of my normal workday. And yeah, it may not be productive in the way I like, but it's necessary time to effectively collaborate.

This idealized notion of "when I feel like it" workdays is particularly unrealistic in a workplace where you collaborate across timezones. I work best in the morning Eastern time, and I have teammates on the US west coast. If there's any chance of working with them instead of around them, I can't just say "eh, I'm less productive after 11am Eastern, so I'm just gonna call it quits."