1) Experience - knowing what you are gonna want to know in the future and knowing what is likely to go bad.

2) Investing in logging early, so that your system is giving you metrics as it grows

3) Understanding that logging and metrics are a conversation that happens with a system over time. You don't do it once and then are done, it's like teaching a child to talk. You have to leave room to expand, and be actively building a 'vocabulary' in your logging and metrics that makes your day-to-day simpler.

4) KISS -- at least on the "big stuff". Your high level architecture should be 'clean' in the sense that issues can be isolated and readily assigned to a subcomponent. I find that isolated communication points that let you bifurcate responsibility are key to quick deductions. Ie the backend publishes data to a store, front-end reads data from that store, so if the data store agrees/disagrees with either you know exactly where the problem is.

Most of the time you have to write metrics into the code (and have a place set up to collect the data), that alone is a step a lot of places don't do or don't do enough.