At CMU, we used CS:APP (Computer Systems: A Programmer's Perspective). It's written by Randy Bryant, the same professor who used to teach it. I still have that book, it's updated for Ivy Bridge, and I truly believe that this book changed my approach towards debugging more than any other text. It will push you to the next curve.
Towards the end of the book you'd find some programming labs. I highly suggest you do them religiously. Many great men have failed at them.
Has anyone read this book and can recommend it?
the writing style is very engaging. some stories are fully resolved, others just conclude with a plausible hypothesis that explains the data, but due to lack of sufficient evidence cannot be confirmed.
Debugging with GDB: The GNU Source-Level Debugger by Stallman, Pesch, and Shebs http://www.amazon.com/Debugging-GDB-GNU-Source-Level-Debugge...
The Art of Debugging with GDB, DDD, and Eclipse by Matloff & Salzman http://www.amazon.com/gp/product/1593271743
Debugging: The 9 Indispensible Rules for Finding Even the Most Elusive Software and Hardware Problems by David J. Agans
Amazon link: http://www.amazon.com/Debugging-Indispensable-Software-Hardw...
What I really like about it is that it gives a number of high-level approaches to solving debugging problems and is written very accessibly.
http://blogs.msdn.com/b/tess/archive/tags/debugging/
She has written some labs covering common debugging scenarios:
http://blogs.msdn.com/b/tess/archive/2008/02/04/net-debuggin...
It has a bunch of examples of bugs, classified by type of bug, and gives you tools for diagnosing and fixing them.
To improve, try learning cracking; https://microcorruption.com/ is a CTF game based around gaining access to embedded devices, it's rather fun too. I think what the most common mistake I find is two programmers having different assumptions on how some subsystem should work, so try improving your working memory as being able to remember the seemingly ok code 10 frames ago that conflicts with what's in front of you could be the key.
try:
Working Effectively with Legacy Code Martin, R.
API Design for C++ Reddy, M.
Were both full of indispensable advice.
http://www.amazon.com/Why-Programs-Fail-Systematic-Debugging...
I don't believe books are likely to be particularly relevant with the diversity of tools and platforms. Google the platform name plus 'debugging tool'. Then practice debugging.
If you have to, step through every single line of code and observe the program state closely at each step.
What kind of distributed system is it exactly? If it is a highly distributed system you can mock parts of it or find/ build a tool for fine-grain inspection and integrated reporting of the state of the system.
A book is not going to be able to keep up with platform and tool development.
You want to learn to Google, find good tools, practice a lot.
Some more posts worth checking out in the JVM domain:
http://highscalability.com/blog/2015/1/7/the-ultimate-guide-...
http://www.infoq.com/articles/Advanced-Java-Debugging-Techni...
(Disclaimer: I'm from Takipi)
He emphasizes the use of the scientific method - observe your program's behavior, form a hypothesis as to what the bug could be, design an experiment to test your hypothesis. Just learning to be methodical rather than thrashing around mindlessly helped me tremendously.
It's out of print but I expect you could find it on Alibris.
