We don't disagree. Many people act like the consensus algorithm is somehow magically obviating the need for a consensus among humans. One must inherently trust that, if one is to use something like Bitcoin.
Functionally, as long as the consensus is to install the latest version of Bitcoin Core, Bitcoin Core's maintainers control Bitcoin.
The whitepaper claims that Bitcoin removes the need for a trusted third-party to transactions. If you consider mining pools to be "a party", that claim isn't strictly true: on several occasions, one mining pool has held 51% hash power (e.g. https://www.extremetech.com/extreme/184427-one-bitcoin-group... is a notable example; before that, pools normally kept their hashing power below 30% voluntarily), so in reality, you're trusting all sufficiently-wealthy parties not to do that. However, the whitepaper doesn't make the widely-believed stronger claim about magical trustlessness.