There are many things, but I think they all boil down to KISS: don't download, don't install and don't run what you don't need.
Most of the compromised servers on which I've done post-mortems have had much more software than they need. Some people think that they need every last tool, gadget, extension and plugin they can find, and they don't realize how significantly it affects their security surface.
One thing that people who have security issues seem to have in common is their willingness to pipe curl / wget to a shell. If some software is asking you to do that, the software authors are already showing that they couldn't give the slightest damn about security.
A few other little things help, like: Never execute where you can write, never write where you can execute (looking at you, Wordpress - that's the reason Wordpress is the #1 phishing site hosting platform on the planet). Handle dependencies yourself and avoid scenarios where you're waiting for someone else to implement security fixes (like dependencies inside of containers). Use chroots / jails / whatever your OS offers. Only expose services to the Internet that are absolutely necessary and well vetted, and use ssh for things that don't need to be exposed to the Internet. Use unprivileged accounts to run services. If you can, avoid binaries and compile from source using documented snapshots.
Generally, just don't trust stuff on the Internet. Everything should be vetted :)
For logs, find the kinds of things that indicate attempts at nefariousness, then write a few scripts that give you some stats on the kinds of nefarious attempts made. As long as some new vulnerability doesn't come along, and as long as you don't have a bad configuration, you should be fine.
