Say your ISP is Comcast... If Comcast knows your are connected to some VPS via VPN, it's likely that anything coming out of that VPN is yours. And if Comcast (or some subsidiary or partner) is also the upstream provider for that VPS, they could pretty easily make some correlations.