It really is exactly this. My default mindset is "everybody's busy with their own lives, so they probably don't have time so I won't even try to invite them to X." Change your assumptions a little bit to instead assume people want to do things. If they say no, so be it. But I've found that people want to be invited out to do more things than they are, so send the invites.

I started swimming with a community team two years ago, and about 4 months in I invited them to also lift weights with me. Now there are about 8 of us that are together 5 mornings per week. Took a chance and invited them on a trip, and now 5 of us are going on a week long trip together.

Find a group of people doing something you like. If it's a tech meetup, community organization, hobby group, whatever. What it is doesn't matter. What matters is that you find people with whom you share _an_ interest. Then take a chance there and say "hey, want to meet up for lunch next week?" Or just say "hey, I'm going to see X next weekend, want to come?"