There's a difference between someone who knows you don't have a good relationship with your family and then really just being in-your-face about it (that's intentionally making you feel bad about your family), vs someone who doesn't know.

Yes you feel bad in both cases, but you probably feel much worse when it's an intentional thing.