You don't have to sacrifice personal happiness, but yes, if you're in a position where you can do good and if you then choose not to do good I do consider that somewhat unethical.
Working hard for people like us consists of sitting on a chair and listening to music while we type for 6 hours a day. Maybe another 4 hours or so of chatting and planning and sketching out ideas. While the rest of the world is born in a position where the choice is between backbreaking work for 12 hours a day or being unable to feed the family.
Yes, we have the freedom to lounge about even though a billion people are underfed or living in otherwise miserable conditions, but who can in good conscience exercise that right?