Your ability to graduate college signals to future employers that you have some combination of intelligence and work ethics. The more prestigious the institution, the stronger this signal is.

At the top end of the scale, it's not even about course difficulty but admission requirements. A lot more people would be able to graduate from Harvard if given a place than can actually get into Harvard, legacy admissions who are still able to graduate being the perfect proof of this. There's research that shows that Harvard courses aren't that much more difficult than those at other, far less prestigious universities, but the admission requirements are far stricter.

This phenomenon is a self-perpetuating feedback loop. If your average college graduate is slightly smarter than a non-graduate, employers will prefer graduates over non-graduates, as it's an easy thing to check. People notice how much of an advantage the college graduates have, so in the next generation, even more people choose college over other paths. This makes the gap between graduates and non-graduates even wider, which makes employers favor them more, which makes life harder for non-graduates, which causes even more people to go to college, and so it goes.

At the top end, other things start to matter, it's advantageous for smart kids to mix and network with rich kids and vice versa, hence legacy admissions, but this is what matters for most graduates.