There are more email addresses lingering inside your email account than you probably realize. I just did a search for "LinkedIn sorry" to find all the apologies I sent out. 38 of them. All to complete and total strangers. When it first started happening I did some digging and found the connection for a lot of these people is we were on a common mailing list.
Not to mention the more obvious ones like ex-girlfriends, ex-bosses (including one I am not on good terms with), companies that I applied to and did not end up working for, etc. This is my gmail account, it has every email I've ever sent in the past 9 years. There's a lot of stuff in there.
EDIT: I don't know if LinkedIn still does this, but at the time LinkedIn would send reminder emails for any ignored invites. Which just compounded the problem.