> I wonder if it's at least partly the temperature, since UHT milk is shelf stable (for a good while anyway) you might be drinking it significantly warmer than milk from the fridge?

Hardly. When I was younger I used to have UHT milk at my grandma, and she stored it in the fridge (after opening), so it was same temperature of what I drank home. Still a very noticeable difference with "normal" milk.