Luck is also a huge factor. I know cases of International Olympiad gold Medalists, who didn't make it as portfolio managers. Do you really think you are smarter?
If you are mathematically inclined software engineer, I would avoid finance unless you are immediately hired into the quantitative role in the front office. In Silicon Valley you will get similar or better salary, more freedom, more respect and better culture. I worked on the both sides.
