In my experience, if you have an SSH port accessible from the internet, it has been probed today by a few Chinese/Russian IPs. Unless my raspberry pi home server is somehow a high value intelligence target...
In my experience a SSH service on a random, non-standard port gets surprisingly few probes. I look after several machines and I see less than one attempt per year (versus hundreds per day for port 22). I have yet to see somebody probe a SSH that listens on IPv6-only.
No, the best solution is to only allow login by SSH keys. No passwords => brute-forcing is impossible. So your threat model for someone gaining access no longer includes someone using weak passwords.
