For me, getting bug reports and feature requests is actually fun and makes the project more interesting. It is telling me that someone uses my software seriously and the bug was worth reporting. Many people don't report bugs because it takes some effort.
It can be, yeah, it just ground me down in the end. Deciding to stop was a huge weight off, I can only imagine the stress someone running a big successful project must feel.