I currently have a manager like you and honestly this is worth a lot. It's basically part of my compensation, almost like being paid in
time.
The peace of mind that comes with knowing that when I say "Hey, sorry, I need to take Wednesday off" the reply will be "Hope you've got something fun planned!" instead of the likes of "Hmm, well, we were ahead last sprint so if we slip this one it's okay" is incalculable. (I probably don't have something fun planned, but the sentiment is appreciated.)