1) Kids from poor / lower-working class families - compared to their peers "higher up" in the socioeconomic chain - simply do not know the importance of a good school. Further, they do not even know the jobs that will follow, or even require such an education.
Even regular college is a long-shot for many, and to them, college == college.
If you're a upper-middle class kid, with parents working as bankers and consultants, then I can promise you that they'll push you to aim high (academically). Hell, they'll probably plan your (academic) life long before you can walk or talk.
In fact, having educated parents seems to play an important role on the chances of your success. If you're poor, chances are that your parents don't even have a HS diploma.
2) Getting into a prestigious school isn't just about getting good grades - you need to show skills and drive far outside that scope. Lots of poor kids can not afford that luxury, like volunteering, getting GOOD at extracurricular activities (sports, instruments, etc.), etc. Some families _depend_ on their kids working after school.
When I grew up, I thought bankers where the people/tellers you actually saw behind the counter. I had never heard of consultant. Silicon Valley SW devs? Had no idea.
None of my parents went to college - they worked menial factory jobs, and we lived out in rural nowhere. 10% of my HS class went to college, and in that case, a no-name rural state college. No big companies _ever_ came to career fairs, so we had zero exposure to them.
If by chance you were exposed to the world of Silicon Valley tech, New York banking, etc., you were told it was too late - that train had left the station before you even enrolled college.
For many of us, the absolute pinnacle of (professional) life was to become a local gov. worker, engineer at the local utility company, a store owner, farmer, or similar. Maybe a doctor if you were deemed very, very smart.
So even if you see some success stories here and there, they are so incredibly rare. Show me one poor (minority or not) kid on scholarship at HYPS, and I'll show you hundred others going absolutely nowhere.