The number of attempts were staggering though, i think there were requests every seconds non-stop.
Once I changed the ssh port to a large number, the OOM and heavy CPU usage stopped, and never came back. So, I think I'm safe, though I keep an eye on the logs, and for any unknown processes, but never seen anything out of ordinary.
The 64gb machine is my dev machine, as my IDE(intellij) runs on high memory config and I run some heavy process, it could've been combined with the ssh spam it went OOM. I still run all the things, without any issues now.