That's why it's so important to track the natural search traffic you're getting by search query, and line it up against the paid search traffic.
If you have strong natural search traffic on the term, then you should be able to scale back your bid on that paid search term without much risk.