For me personally, the biggest downside is that I don’t feel good about myself when I take things I know I don’t have the right to take. YMMV.
And frankly, I don't feel one bit bad, copying AND providing copies free of charge. I've paid enough to content, media, and game companies, and screwed over on rentals that were sold as sales.
But you do you.
Once I put in enough work to both have some spare money in the bank and to empathize with the people who made the software and music I could so easily "just download", I had this transitionary period where, if I possessed a copy of something available commercially without having paid for it, and found that I enjoyed it enough to keep using it, I would buy a legitimate copy. For music, if I really liked it, I'd buy, for example, the deluxe vinyl edition - hopefully kicking some extra money over.
With software (including games, though it's rare that I play games) I'm now at the point where I won't give it a second thought, and will just pay for it. I bought an iPad app on sale, years before I had an iPad, knowing that one of these days I'd pick one up - it worked out.
I understand and share your ethical dilemma. The goal is to support the creators of the work. If you do that, then there is no reason to feel guilty because you enjoy their work in a different way.