This is a reasonable criticism. Let me try to counter.
PoW doesn't protect you from a large enough cabal of power (in Eth's case, developers and whale bag holders) controlling the direction of the chain, or censoring entities they don't like from continuing to contribute.
What it does do is ensure that this large cabal can't erase history that happened. The only reason Ethereum was able to fork was because it was still young, they moved very fast, and there was enough external incentive from all parties to ignore the lost mining costs. And even with all that, Ethereum Classic still exists with all the PoW post DAO hack preserved.
On the other hand, PoS has no cost associated with writing the blockchain as far back as you want. If a majority cabal of stakers agree to rewrite history, they just do it. They can rewrite the chain from the dawn of proof of stake with whatever changes they want and it looks equally valid as the "true chain". No energy intensive hashes needed.
With PoS, you have to compromise a large enough number of stakers to mutate the chain. Very little cost after that.
With PoW, you not only have to compromise the miners, but you need to force them to burn tons of real world energy to catch up to the current block to rewrite the chain. Insanely expensive and effectively impossible the farther back in time you go.