Peers make a huge difference. Before university, I split my high school between two schools - one that was near the top academically, and one that was quite poor. The latter did have some smart students intellectually, but almost none did well academically because it wasn't valued by their peers.

Then I went to a very average state university for undergrad, and a top school for graduate studies. The difference wasn't that high in terms of teaching (the average school actually had much better teachers, but offset it by low expectations). The real difference was in the peers.

You like engineering? You like coding? Want to do some cool side project? Very hard to find someone like you in that average university.

Then when I started working, I started tutoring some middle school kids. The kids seemed totally capable mentally, and I was trying to figure out how they can't retain simple facts like number of months in a year. Until finally it hit me. They don't have problems learning things. It's just that no one in their orbit (peers or parents) care if they know these things. When I was a kid, I'd be an idiot amongst my fellow students if I didn't know it. So I did. Everyone did.

But if you're around people who think it's OK not to know how many days are in a year, chances are you won't know it, no matter how intelligent you are.