Only if they already have local access. And the sshd is not listening on the port (if it’s on, you can’t just bind it)
And they still can’t show the host certificate, so your client would tell you the certificate changed.
Unless, of course, they have a local root exploit — but then port 22 is just as unsafe.
I fail to see how changing a port makes anything less safe.