In order to get to the Sun rather than just speed past it, you have to decrease your velocity dramatically.
When you're in a stable orbit, you are actually spinning around the sun at a huge pace. To gain enough velocity to leave the solar system, you have to increase that pace by an amount that is less than the pace you already have.
As a terrible analogy, it takes less energy to overtake a car that is travelling in front of you at a higher speed than it does to slow yourself to a complete stop.
