I haven't seen arguments that they're constitutionally unsavory, but I've seen arguments, that the President, as chief executive, does have almost CEO-like control over them. FDR did exert such control, in his case using it to expand the federal government, but he ran a fast-moving government.
So it's not like there isn't precedent for this, it's just that the consensus was as you said, the independent (some would say unelected) bureaucracy running things. But that was only ever a convention.