Of course, the experience of clicking play on a song and having it only start a number of seconds later is not something that'd sell particularly well, I guess. And then you'd have to renegotiate the BT profile if a call comes in that has to happen live. And switching back to the song will have another big delay.
And let's not forget this was a discussion of buffering. A buffer of 5 minutes (50MiB) buys you 5 minutes of not having to be real-time, or to be slowly lagging behind — if that covers 3h of continuous listening time, you probably covered 99% of uses where latency is not a big deal anyway (like playing music — calls and movies are another game).
I already acknowledge practical UX problems with just relying on buffering, but it doesn't make much sense to say how it can't be done because of the protocol either.