I don't think anyone outside Google knows the exact details, but GMail clearly calculates a global reputation score for each sender that is influenced by what percentage of recipients mark its messages as spam. Then there's some additional weighting on top of that based on your personal actions.
Additionally most reputable email senders have a "feedback loop" set up with Google, Hotmail, and Yahoo where clicking that Report Spam button actually passes your email address back to the sender's email system. For example if you click Report Spam on one of our email newsletters in GMail, we will flag your record in our database and not send you any more messages even if you specifically sign up for a newsletter in the future. (Please don't test this.)