I'm a white guy who was born into a poor, solo parent family situation. I still had privilege due to my colour. Example: no-one ever looked at me and wondered whether I was going to steal something from their shop just on the basis of my skin colour. When I went to school, no-one discouraged me from high-level achievement or assumed that I should do sports based on my colour (and where I come from, this definitely happens to people)
I'm a little peeved at people dismissing their privilege just because their life wasn't perfect. That's not what privilege means.