In space (and any frictionless medium), you can't "redirect" an existing velocity vector with thrusters. You can only add a velocity vector. This means that if your desired direction is perpendicular to your current direction, your current speed is no good to you at all. If you're heading due "north" at 10mph, and you want to be heading due "east" at 10mph instead, you have to 1) fully negate your "north" velocity, and 2) come up with 10mph of "east" velocity from scratch.

Now, the trajectory of an object in solar orbit is exactly at right angles to the direction it needs to go in to hit the sun. No part of this velocity is helpful for getting to the sun - in fact it actively prevents it! The only vector that takes you directly into the sun is one with no sideways component - if you imagine yourself falling right in, any sideways nudge will cause you to miss it by a hair and go flinging off into a highly elliptical orbit. If you just ignore this and just thrust directly at the sun, hoping to overpower everything by brute force, then like a ballerina pulling her arms in, the more you try to get close to the sun with your thrusters, the faster your orbit will go; the closer you manage to get, the further out you'll be flung when you inevitably miss.

All this ignores that the sun is not a point, but quite a large ball - you can get away with some small horizontal velocity. A highly elliptical orbit will still do what you want if its lowest point is below the surface.

Once you're in orbit (say around the sun), you have to cancel the orbital velocity to fall into the object you're orbiting around. If you point at the sun and accelerate 1 km/s directly at it, you're still moving 30 km/s "sideways". All you'd end up doing is making the orbit more elliptical-shaped.

At least that's how I see it, but I am far from being an authority on this topic.

(I mean this in all seriousness)

You should play Kerbal Space Program. It will very quickly give you an excellent intuition for basic orbital mechanics.

I'm no rocket scientist but perhaps that would cause the object to shift from the Earth's kinda-circular orbit into a highly-elliptical orbit, its existing sideways velocity (relative to the sun) causing it to be flung past the sun at more of a straight line and hence way out into the solar system instead.

You can but then you're using a lot of propellant rather than gravity to reach your target.