I agree with you that drawing the ethical line wherever it is most convenient is bad. In fact, I have discussed or debated ethics online a few times in the past, and I was always in your position, saying that the issue is black and white no matter how convenient it would be otherwise. Perhaps this is why I am so interested in this thread, now that I am on the other end of it.
Anyway, thanks for your insight. Although I am still not convinced that anything I did was unethical, I will think twice before looking at past work again.