Well, it's certainly not determined by some objective standard, or a god. Every culture, and even every individual has different moral views. When I say "do no harm" that's a relative statement, relative to the context and parties involved. What's morally apporiate changes depending on the moral contract between parties. I can call my drinking buddy a "fucking idiot" for making a mistake, and there's no harm done. But if I call my grandmother a "fucking idiot", it would harm her a great deal. What's harmful in one context may be fine in another. Morality is like an instinct that humans have evolved to allow us to detect when something may be considered harmful to ourselves, our partners, our community and our planet. It's not always an infallible sense, but it's often pretty good and useful to pay attention to.

Back to your question: who determines what's harmful? Our innate sense of morality has evolved to show us what's harmful and what's not. The more each of us focuses on listening to and improving our own sense of morality and harm, the better we'll be at making decisions that avoid harm as a society and as a species. Ultimately, I'm a humanist, and one of my favorite quotes about morality is GK Chesterton's response when asked to write an article answering the question, "What's wrong with the world?". His response, "Dear sir, I am."

One reason I don't concern myself too much with the morality of others, is that the only person's morality I am responsible for determining is my own.