Sometimes your problem turns out to be a problem with production hardware or wiring, or depending on interaction with external partners who may not be able to replicate their interactions accurately for your dev/test environment, or otherwise impossible for you to replicate in dev/testing without already having figured out what the problem is.

While I agree with your overall sentiment, in any kind of remotely complex environment you sooner or later come across problems that you have no sensible way of reproducing in dev, and which will require you to debug on a production environment without it saying anything about the quality of your development process.

> Never, ever debug on live. If you can't recreate the problem locally, you don't understand the problem yet.

I am guessing you have not been exposed to highly complex live environments. What you say is largely true but you state it as an absolute fact, which is a bit naive. Live environments tend to be far more complex than test or local environments and sometimes that complexity is what creates the bug. Sometimes the only solution is to debug live, but of course that is a last resort.

> Then you've done it wrong. Very, very wrong. You should be able to replicate your live environment locally and not being able to is a sign of a very broken development process.

I feel this is statement is a tad hyperbolic.

One of my key skill sets is debugging, I'm pretty good at it. One of the things we can't do is completely replicate a production environment because we're a web hoster. Sure I can run and test stuff against our OS "gold images" but when you've added third party client code running on 300-400 sites on a single server that environment is not reproducible without a herculean effort. We also don't have the luxury of time to "Keep trying to replicate it until you do. I'm talking days of trying".

Four or five times a year I need to dig out ADPlus, point it at a customer's worker process and tell it to snapshot and dump when certain conditions/triggers occur. I then whip said dump over to my WinDbg+SOS environment to find out what the hell was happening in the process when it went bad.

I think there's a lot true in what you've written. However the root cause fix applied in a development environment is not always the fastest way to stop a user facing problem. Things get weird on live servers. I'm not suggesting altering code willy-nilly but a solid knowledge of production (logging, service status, etc) can lead to more effective debugging scenarios and faster time to resolution when things do go bad.