My apologies, my phrase "milking" was meant to imply "overly burdensome resource taking" when compared with their value add. IE cost too much
To rephrase, all too often middlemen are there because of legislative/financial props from government or other incumbent advantages and - as you say - both sides would be better without them.
In this case - as in many middlemen cases - Google provides value - their search services. This "value" is the hook for their place in the transaction.
If Google provided a useless list of random results for every search term then it is obvious they would lose their place.
But where is the crossover point? Where Google doesn't provide enough perceived value compared to their perceived cost such that the market abandons them?
Peoples growing dislike of ads and - from this article - advertisers changing perception of ad value will change the cost vs value equation.
But I still think Google is safe for now.