Considering that housing benefit was set for a long time at the median market rate, this surely drove rental prices upward.

If landlords know they are guaranteed the median by way of housing benefit, they set their rents at the median (given a housing shortage, they can guarantee to rent out any old dump at the housing benefit rate with no disadvantage: and this is evidenced pretty well by distribution of rent prices), which has driven the median upward - a vicious circle of rent increases.