Nothing wrong with middle men per se, but problems do arise when we all rely on the same middleman: those become way too powerful and can do nasty things.
By that time, no one can do without the nasty middle man as we have forgotten or never learned the skills to fend for ourselves and are thus beholden to the nasty middle man.