In either case, you need serious research and numbers to figure out what the right path is. 80,000 hours (http://80000hours.org; I am not affiliated) is a charity that's set themselves up to research and give career guidance for people who want to maximize their positive impact. They will probably give a better answer than StackOverflow.
Have you tried? What's that based on? I know lots of people doing great direct work.
In my experience, money is NOT a good substitute for concerted, focused, long-term attention in a problem domain.
GlobalGiving - http://www.globalgiving.org/aboutus/jobs/software-engineer-f...
DonorsChoose.org - http://www.donorschoose.org/blog/category/job-opportunities/
Kiva - http://www.kiva.org/jobs
charity:water - http://www.charitywater.org/about/jobs.php
Volunteer Match - http://www.volunteermatch.org/careers/
And thousands more at http://www.idealist.org/
My advice would be to stay in academia -- with a PhD and 4 years of experience they are overqualified -- but search out inter-disciplinary programs, especially in computing. Collaborating with computer scientists (or becoming one) is a sure bet.
Perhaps a few mathematicians might join the fight against innumeracy among programmers?
Please define "good".
To me, good is where you find something interesting and worth doing.
Financial speculation makes the market more liquid and reduce the spread. You say you find that interesting. I would then call that "good".
If it matches your definition of good, you may want to engage in that.
If it does not match your definition of good, look at international finance, then international macroeconomics - you will see many people are actively looking to find ways to mitigate the effects of speculation.
Also, there are some interesting side problems where your mathematical skills could be applied (should monetary policy be used in a discretionary way?) for similar problems (if you define a stable economy as a "good" thing)
Go there, learn about it, make some money while you're at it. Leave if you don't like it - it's not a sect.
i changed fields, but for this guy on SE, who enjoys algorithms and complex data, there are a wealth of openings right now to study all sorts of data in fields to "do good", including genomic and life sciences, weather, and much much more.
to anyone considering a Ph.D. in a field, i suggest you look beyond the material and instead look at the day to day of the field you think you want to get into (e.g. academic research) and train for it, and decide if you want to do it. if you don't, consider still getting the Ph.D. and exploring other things to do with it. i'll be honest, i don't think anyone would have given me the time of day in a new field without my Ph.D., so even if you change fields radically you'll still get doors opened to you!
At times (for me) it has even been helpful to get professional career-counseling to discover careers (or even avocations) whose shape and impact have changed in preceding decade since I last looked at it. There are useful psychological batteries for assessing the fit between fit/vocation that take such assessment far beyond navel-gazing/tea-leaf-reading.
Another approach would be to involve yourself with an organization (vocation or volunteer) with a project that appeals to your vision for a better world - and discover what roles beyond traditional mathematician you find you can make a contribution to. Hope that's some help.