Good question, one that probably has less to do with what Google wants, and what the music industry requires. If you turn off the screen, the "video" is indistinguishable from an audio stream. So they want to track for terms of royalties paid per "listen".

This is only feasible on a subscription basis (no one has a good microtransaction model with actual payments) and so Google and the music industry limited the feature to only subscription based.

Because why should it be? People are willing to pay for it.

This is especially also the case, as Google could abuse some sentence in the YouTube API ToS, which prohibits the separation of audio and video, to get rid of third-party YouTube apps which offered this functionality for free before YouTube Red became a thing. If those had been around, they couldn't really have charged money for it.

There are still a few of those apps available on non-Play-Store-stores, as Google would have had to deal with legal entities to get rid of those (and would have probably lost), but most people never use alternative stores, so that's not really a problem for Google.

One of those apps is NewPipe, if you want it for free: https://f-droid.org/repository/browse/?fdid=org.schabi.newpi...

I don't know.

Maybe it has something to do with them not being able to show you ads when you video ads when your screen is minimized. Or maybe it is to prevent people from using youtube as an alternative to paying for google music.

Regardless, I posted my original comment because it seems a bit disingenuous to say that this feature isn't available when it is (at least in the US). You just have to pay for it.