> Should all environments be monitored?
Yes. If you don't collect data about the environment, you can't tell whether
the current behaviour is normal or abnormal. Though not every environment
should send all alerts possible.
> I was wondering how others handle these problems internally or if we are the only ones who are having this discussion?
Monitoring is a tough topic -- still, after more than twenty years of
development in this field. The problem is that we still treat monitoring
systems as monolithic applications with hardcoded data collection and alerting
logic, instead of making them what they are: event stream processing engines,
with some publish-subscribe capabilities to allow attaching some external
logic.
I'm working on a toolbox that helps in building such a system
(http://seismometer.net/), though it's still lacks tools and its website is
more of a sketch than a project homepage.
In the meantime, you should think about how to allow different people to
select or filter out different notifications, possibly in such a manner
that they can process those notifications programmatically.
