SlateStarCodex's excellent post "Meditations on Moloch", (http://slatestarcodex.com/2014/07/30/meditations-on-moloch/) says it way better than I could so I hope you'll forgive a slightly lengthy quote:
" [M]any of the most important competitions / optimization processes in modern civilization are optimizing for human values. You win at capitalism partly by satisfying customers’ values. You win at democracy partly by satisfying voters’ values.
Suppose there’s a coffee plantation somewhere in Ethiopia that employs Ethiopians to grow coffee beans that get sold to the United States. Maybe it’s locked in a life-and-death struggle with other coffee plantations and wants to throw as many values under the bus as it can to pick up a slight advantage.
But it can’t sacrifice quality of coffee produced too much, or else the Americans won’t buy it. And it can’t sacrifice wages or working conditions too much, or else the Ethiopians won’t work there. And in fact, part of its competition-optimization process is finding the best ways to attract workers and customers that it can, as long as it doesn’t cost them too much money. So this is very promising.
But it’s important to remember exactly how fragile this beneficial equilibrium is.
Suppose the coffee plantations discover a toxic pesticide that will increase their yield but make their customers sick. But their customers don’t know about the pesticide, and the government hasn’t caught up to regulating it yet. Now there’s a tiny uncoupling between “selling to Americans” and “satisfying Americans’ values”, and so of course Americans’ values get thrown under the bus.
Or suppose that there’s a baby boom in Ethiopia and suddenly there are five workers competing for each job. Now the company can afford to lower wages and implement cruel working conditions down to whatever the physical limits are. As soon as there’s an uncoupling between “getting Ethiopians to work here” and “satisfying Ethiopian values”, it doesn’t look too good for Ethiopian values either.
Or suppose someone invents a robot that can pick coffee better and cheaper than a human. The company fires all its laborers and throws them onto the street to die. As soon as the utility of the Ethiopians is no longer necessary for profit, all pressure to maintain it disappears.
"