This is obviously wrong. If their methodology says that Caltech is in the bottom 2% of US colleges, then one concludes that their methodology is worthless, or at least that what it's predicting is not very closely related to the quality of the education provided. I suspect that many Caltech undergraduates decide to pursue grad school and academia, which is not an especially lucrative career, and which is probably not highly corrolated with political leftism or "reefer madness".
I can also tell you that my friends at Rice who were looking to make a lot of money after they graduated, by and large succeeded.
In summary: Bullshit.
"The analysis goes against my gut feeling, therefore it's wrong."
Did you bother to read the article?
The Economist’s first-ever college rankings are based on a simple, if debatable, premise: the economic value of a university is equal to the gap between how much money its graduates earn, and how much they might have made had they studied elsewhere.
That's what they did, and those were the results. They even say it's debatable. Using a different analysis or metric, the results would be different.
>In summary: Bullshit.
As opposed to the anecdotes regarding your Alma Mater and rich friends.
If one school tends to attract students interested in making a lot of money, and another school tends to attract students interested more in academics and graduate school, can one really conclude that the former adds more value? Perhaps, but certainly not without taking into account the different student interests and inclinations towards money-making.
They do control for field of study, but I don't think this is sufficient. A school that attracts a lot of STEM majors who go to graduate school instead of industry is pretty much going to get destroyed in this study.
Yes, I did. In suggesting that their methodology is "debatable", they are indeed hedging their bets. But they are still advancing the theory that, after looking at the data, one could plausibly accept their model.
My "anecdotes" consist of four years as an undergraduate at a university (Rice) rated in the bottom 0.5%. I know damn well that Rice prepared the vast majority of students to be extraordinarily successful in the career paths they chose. I also know that my education was tremendously valuable for its own sake, and that the same was true for my classmates.
And as an academic now teaching at the University of South Carolina, I can only wish that we offered our students what was offered at Rice.
It is true, as a Rice graduate I am biased. But nevertheless I do not believe that I am stretching when assert with confidence that, according to any metric which measures anything worth measuring, Rice lies somewhere in the top 99.5% of universities in the US.
As a Caltech grad, I'll give my two cents. They are afflicted with the worse version of the same problem MIT has - "working for Harvard men". Who besides Adam d'Angelo has made it really independently big in the tech world from Caltech? They are turning out somewhat eggheadish very good Google engineers but without the reorientation towards entrepreneurship that MIT at least has with Sloan, the Media Lab, etc. Over the last 10 years, student life has become progressively more proscribed as the administration seeks to make campus a tamer version of Princeton. Caltech's purpose was to produce scientific leaders of the mold of Oppenheimer, Mark Wrighton, et al. - people who could both do research at a high level and manage institutions while providing a useful counterpart to the institutional drift that comprise NSF and NIH review panels. It's sad that the college that prides itself so much on the elite and supposedly self-selected nature of its student body and touts the supposed freedom and creativity offered doesn't trust its students. It may be that the faculty collectively take them less seriously than in the past, I don't know.
Nevertheless, as a graduate, I hope you regard it as self-evident that, even if your alma mater has slipped, it still deserves a place in the top 98%?
"The bar is set extremely high for universities like Caltech, which are selective, close to prosperous cities and teach mainly lucrative subjects. If their students didn’t go on to extremely high-paying careers, the college would probably be doing something gravely wrong."
And as far as being close to a prosperous city -- that is because Caltech is near the world hub of the entertainment industry. A Caltech education is mostly irrelevant to finding employment there.
Think of it as a supplementary rankings system, rather than a definitive one.
The biggest problem with the methodology, to me, is it doesn't consider average cost of attendance to students at all.
The question they're trying to answer is: once a smart and successful person chooses a college, how much value does that college add? Looking at that same expected earnings sorted table, the actual earnings is quite varied. MIT graduates earn 8k more than expected, and Rice earned 10k less. That's quite a difference!
There's many reasons this dataset is flawed, but not for any of the reasons you mentioned:
1. Financial aid applicants are not a representative sample of students. 2. Highly selective schools have a very high bar to exceed. 3. Students may prefer other opportunities that pay less, like civil service, or politics. 4. Earnings are only tracked for ten years (standard loan repayment).
I also question how earnings are calculated. For example, because I max out both a 403b and a 457b retirement account, my earnings are taxed as if I earned half of what I actually do. Does my double savings opportunity affect the calculation?
In particular, I believe that your (3) describes what I see as the biggest flaw with their methodology; that (at least for elite colleges) what they are largely measuring the proportion of students that choose to go into high-paying professions.
I actually don't think that 3 has much explanatory power. Rather, I think the bulk of the variance is beyond 10 years. Yale's strongest suit is likely professional programs like law and medicine. These are high paying professions, but they require additional years of education beyond a 4 year bachelor's, and that subtracts earning years from the 10 year cutoff.
I don't know much about Rice, but I suspect it's particular problem is being a highly selective school in a low cost of living area. If an average Rice CS student randomly selected into other schools with comparable SAT scores, they'd be going to a much more expensive region, and their controlling for location may not sufficiently model how a CS degree differs from community college nursing.
It's not "ranked by alumni earning".
we ran the scorecard’s earnings data through a multiple regression analysis
So they ran some machine learning algorithm, the Over/Under column just shows the errors in their final model. If their choice of variable was perfect, then every school would have an over/under of $0, some metrics are just not linear and not easy to get (school reputation, networking power?)
In the end this is not a very valuable metric, who cares if they think a cal tech graduate should make $82,126, it still makes $74,000. What is more interesting to me would be how much money that graduate spent on school to get there.
Mostly someone who wants to measure the effectiveness of their undergraduate curriculum. As a student, there's a strong case to be made for ignoring this data and seeking the most selective alma mater possible. But if the law surrounding Duke v Griggs ever gets overturned and IQ / SAT scores are allowed in screening job applicants, you definitely want to know how much of a school's function is the actual learning and placing bit.
That said, if the predominant value of education is in the pursuit of money, then I think using the Brookings Institute model (which The Economist references) that includes two-year and vocational education entities is a much more pragmatic and practical approach. That is, if money is the goal, only focus on money as the outcome. Conflating money with prestige is, to me, foolish. A University is a prestige degree, insofar as there's a pursuit of knowledge, in theory, to produce a well-rounded, educated person...in theory.
I say this as a graduate of two generally highly ranked Universities, who occasionally gets the feeling that I'd be making more money if I'd taken my education budget, got trained in HVAC service/repair/sales, and started my own company.
Look at CalTech for example. It does well in the Brookings model because it attracts a lot of people who want to be engineers. It does poorly in The Economist's model because its engineers are not making as much in 10 years as would be expected given a variety of factors that they control for. (Now that said, CalTech may be hurt because they graduate a lot of people who take side tracks through grad school and presumably would make more 15-20 years out. But you have to work with the data you have, and all they had was 10 years.)
Now if I'm a student planning to go into engineering, the Brookings model merely confirms that I'm making an excellent choice. The Economist's model suggests that if I'm capable of going to CalTech, maybe that is not the best school for me.
From the research I've absorbed over the years, students that can afford to go to prestige universities often come from well-to-do, college educated families - these are high indicators of future success. For a much larger swath of the population that is trying to advance their stake in life, they are first-generation students, and often not well-served by taking on debt and attending a large University environment. Thus, many Universities are, practically speaking, very bad at ROI for a large contingent of students who simply want to make more money - vocational training? An excellent prospect.
What remains to be seen is how much more earning potential there is in professions that require vocational training to be licensed or successful (i.e. welding, machining, air traffic control, plumbing, electrical contracting) as the generation currently working retires / dies from the workforce, thereby spiking demand, and in fields where a "traditional" University degree are, well, useless.
That said, one of the shortcomings of their metric is that they only had data on students who took out federal loans. How predictive this result is for others is open to question.
Median is also possibly not what an incoming Caltech (or MIT, or Harvard) student should optimize for though given that many of them will have relatively safe downsides. Mean for Caltech might be quite high (I don't know for sure). But given that I overlapped with multiple people who have certainly made large sums of money since graduation (about 10 years ago), I'm thinking it might look favorable for Caltech with a small student body.
I don't think that part's necessarily true. If a school focuses on an area with high median salaries, the model will take that into account in the predicted salaries, so the school will have to have even higher actual salaries than typical for the field (and its input demographics, SAT scores, etc.) to get a positive value-add. See Caltech for an example of a STEM-focused school that does badly by this measure: from its SAT scores, demographics, and heavy concentration of STEM majors, the regression analysis predicts that it should produce graduates with a median salary of $82k. But the actual median is $74k, so its value-add is taken to be -$8k.
Some of the schools that do well are in areas with poor salaries, but score highly because they do better than you'd expect (or than the model would expect, anyway) for that area and student demographics. Otis College of Art and Design has a predicted salary of $29k from the regression analysis, but actual median is $42k, so implied value-add +$13k.
You also presume it is not possible to measure academic success empirically.
Also, a very similar argument to what you're making explains why I don't think that the gender wage gap is an big issue.
CalTech is a common example being cited here. It is basically impossible to deny realistically that going to CalTech is a great way to make a lot of money. But it is in the bottom percentages of these rankings, likely because so many CalTech alumns go into academia.
The metric is supposedly showing which colleges can potentially increase your earnings the most, but it just isn't doing that. It's a great idea, it just hasn't been implemented fully.
I think these one size fits all rankings are all flawed by their inherent nature.
You can select which variables are important and how much, and it generates a top/bottom 50 list.
This was inspired by this article from NPR (http://www.npr.org/sections/ed/2015/09/21/441417608/the-new-...)
> Its upper tiers are dominated by colleges that emphasise engineering (such as Worcester Polytechnic) and attract students with high SAT scores (like Stanford). The lower extreme is populated by religious and art-focused colleges, particularly those in the south and Midwest. This number represents the benchmark against which we subsequently compare each college’s alumni earnings to produce the rankings.
Yep yep. Otis College of Art and Design "overperforming" for $42k salary is ridiculous. The title Junior Graphic Designer pays around $40k* in Los Angeles, where Otis is located. Therefore, the average for graduates is as-expected. *Source: Glassdoor.
----- Geography (both city and state) are variables included in our model. Colleges in the Bay Area are not rewarded for the high salaries available there--they have to surpass a higher "bar" of expected earnings. That is why our eighth-ranked college is in West Virginia, and the ninth in Laredo, Texas. ------
Now, I don't know how they compensate for this. Cost of living at the place where the student ends up? Or where they study?
If you study in a high cost of living area but move to Podunk, Iowa, your return will be far lower than expected, if they're only controlling for CoL at the college's geographical location.
If they control for CoL at the student's new home, then okay.
For example: Why should CalTech get hurt for being near L.A.? They're basically arguing that you're better off going to a school in the middle of nowhere because "hey, for being in such a crappy location, you did pretty well!". In an absolute sense you are better off going to CalTech, it's just that they might not leverage their advantage as well as some other schools.
Not that they even show the last point -- it seems unlikely that the true model is linear (I'm guessing they used linear regression). For example, if the true model is closer to a sigmoid, then schools at the high end suddenly get unfairly penalized and schools near the low end get unfairly boosted.
Finally, the statistical indicators are equally misleading. I can obtain an R^2 of 1.0 just by including indicators I[is COLLEGE_NAME] for each college. While that might not give you significance, the point is that getting good prediction is meaningless.
I think what they really want is to restrict to predictors about the students. So, given that you're a straight A student with a 2400 SAT score, what would you expect to make coming out of each school? This at least tells me something about the added value to me of going to a certain school. (This approach is still prone to bias, but in the opposite direction -- there's a chance that the straight A student with a 2400 SAT score going to community college may have been smart but unmotivated, which might correlate with lower salary.)
Edit: Here's another concern. They're claiming to have a model for "expected" earnings:
earnings = A * (college covariates) + b + error
but they can't distinguish between model error (i.e., error because their model is misspecified) vs. the school variation that they are trying to capture.
Well, no, not exactly. It's a subtle distinction, but what it's actually ranking is how well that school exceeds expectations, not best outcomes. This is not necessarily a list that will give a student the best school to go to, but rather (what it says on the tin) a scorecard for how well those schools are doing, given their resources.
Here's another way to see my concern. Suppose you had a classifier that achieves 1.0 R^2; then since it perfectly predicts each school's expected value, it'll assign each school a score of 0. I'm pretty suspicious of an approach where the results get worse with better predictive power.
Even if you want to do "exceeds expectations", I think you shouldn't include variables that are school specific, only variables that are student specific. In other words, for my expected outcomes, which school is best?
This depends on your view of the importance of undergraduate education, and what worse is. From my point of view, undergraduate education is an institutional obligation used to fund or justify faculty's personal objectives: research.
The reason that the model counts location is simple: universities tend to place candidates locally. I'm pretty sure the recruiters attending fairs at Stanford and Berkeley have higher starting wages than the ones at University of Kansas, and that a lot of that difference is simply regional cost of living. If you don't factor that in, you risk a bad school in an expensive place ranking higher than a good school in a cheap place.
If I'm understanding correctly, that result would indicate a world where the college you attend has no effect on your earning power. ie. choose any college you want, because you'll earn the same amount regardless of which one you choose.
This would only apply to colleges that people in your demographic group actually attended though. If the dataset doesn't contain any information about people like you who went to Harvard, then maybe Harvard would indeed increase your earning potential if there was a way for you to actually go there.
Take the top-3 state schools in Virginia (by most other rankings, that's UVA, W&M, and VT). All three are nationally recognized. And all three are competitive that you need very good grades and a solid set of extra-curricular activities to be accepted.
UVA ranks the lowest of the three, yet has the highest actual earnings. VT is ranked significantly higher with the middle earnings value.
What does a student do with this? Cost of attendance at the three is similar. Should they attend VT with it's higher ranking, despite lower average earnings?
Similar comments can be made about UNC-CH, UW-Seattle, UT-Austin.
Also lacking from this analysis seems to be the loan burden borne by students. Georgetown and Villanova both rank very high in this list. But, both are insanely expensive to attend. Even with high actual earnings, it could take a decade or more for many students to pay off a potential six-figure loan.
Maybe I need to dig into the source of the expected earning figures. I'm definitely biased to some degree, being a graduate of one of the state schools with a poor ranking - I'm sitting here wondering where else I could have gone to get a better value.
1. Graduates of some colleges enjoy much more economic success than their characteristics at time of admission would suggest. Colleges with high value-added in terms of alumni earnings are often focused on training for high-paying careers in technical subjects. ...
2. Four college quality factors are strongly associated with higher earnings for alumni:
Curriculum value: The amount earned by people in the workforce who hold degrees in a field of study offered by the college, averaged across all the degrees the college awards;
STEM orientation: The share of graduates prepared to work in STEM occupations;
Completion rates: The percentage of students finishing their award within at least 1.5 times the normal time (three years for a two-year college, six years for a four-year college);
Faculty salaries: The average monthly compensation of all teaching staff
3. Value-added measures are fairly reliable over time. ...
(Personally, I think the most valuable things gained in college or in any education don't happen to have much monetary value.)
[1] http://www.brookings.edu/research/reports2/2015/10/29-earnin...
Well, someone does, because they're charging for it.
Good point; I should have said, I don't think they are things that earn you money.
There's one thing that jumps out at me when I see the actual rankings:
At least three of the eight schools that received a perfect 100 score -- W&L, Babson, and Bentley -- have a very, very high number of students who go to work for lucrative family businesses immediately upon graduation. In the case of W&L (which provides a great liberal arts education mind you) it's mostly southern good ol' boy/girl types. At Babson / Bentley (which provide great undergrad business educations), a huge chunk of the student body are the scions of the economic elite in developing countries, who are getting schooled up so they can be ready to be put in charge of something at a young age.
My hunch is that the Economist's "expected earnings" methodology wasn't granular enough to take these idiosyncratic attributes into account, and that its r-squared of .85 might not be a rigorous number.
But having classmates who look like that seems to be a really good economic choice! Which is a factor that is not apparent in normal college rankings.
or, given the economist's core demographic (rich borderless global elites) this is exactly what they optimized for.
So the dataset excludes students who do not apply for loans? (i.e., this analysis penalizes schools who admit the folks most likely to make lots of money, and the schools that have the lowest expected student contribution.)
>"...based on a simple, if debatable, premise: the economic value of a university is equal to the gap between how much money its graduates earn, and how much they might have made had they studied elsewhere."
If we are only to look at financial incentives, a more reasonable analysis would be to compare the expected future earnings _distribution_ as opposed to just a central tendency statistic like the median.
It penalizes (or at a minimum, undersamples in a nonrepresentative way) schools with their own, non-loan, aid programs.
This probably very badly hurts schools like Caltech, since it means that there (unless Caltech no longer has the essentially full-coverage need-based aid they had when I went -- unsuccessfully, I graduated elsewhere) they are only counting people who are making the unwise choice of taking loans when they have no need given their current resources.
That this will skew the data badly should be obvious.
Take all the schools I could get into, look up their actual median earnings, and go to the one which is highest (let's ignore issues of cost of attendance).
If it turns out that the one which is highest isn't ranked high on this particular list, I don't see why that would suggest I still shouldn't pick that school. Imagine these are my options, for instance:
(A) A school with expected earnings of $90k and actual earnings of $75k.
(B) A school with expected earnings of $50k and actual earnings of $55k.
(B) will rank vastly higher than (A) in this study, but I'd pick (A) over (B) every time if I just wanted money.
In short, I don't actually see the value add here from this list. How am I supposed to act on these rankings? How are these rankings supposed to change any idea I might have about which school I should attend?
It seems if you want to know which school to attend based on earnings we already have much more reliable data for that: actual earnings data.
(A) A school that admits 1000 people with IQ of 200 who go on to earn an average of $100,000 and 100 people with IQ of 50 who go on to earn an average of $1,000
(B) A school that admits 100 people with IQ of 200 who go on to earn an average of $1,000,000 and 1000 people with IQ of 50 who go on to earn an average of $10,000
(A) has median earnings of $100,000 while (B) has median earnings of $10,000, so by your criteria (A) is much better. However, (B) has much better outcomes for both groups of students.
Another interesting study would be to see whether a child's college attendance correlates with a better or worse standard of living from their parents'. Maybe look at Father's salary at age 30 and child's salary at age 30, and see if the attended college is significant. The question being: If you're born rich, you tend to stay rich, and if you're born poor, you tent to stay poor. Does college choice matter?
I much prefer the Times Higher Education model, especially their reputation survey. They survey 10,000 tenured and published academics worldwide, using what looks like well-designed methodology [1]. These are people in a position to have expert knowledge about the qualities of various universities. Yes it's imperfect and those people have bias too, but I can't think of a better model:
https://www.timeshighereducation.com/world-university-rankin...
One excellent alternative is Washington Monthly's rankings. Their approach:
We rate schools based on their contribution to the public good in three broad categories: Social Mobility (recruiting and graduating low-income students), Research (producing cutting-edge scholarship and PhDs), and Service (encouraging students to give something back to their country). More here:
http://www.washingtonmonthly.com/college_guide/rankings-2015...
----
[1] https://www.timeshighereducation.com/world-reputation-rankin...
I understand why they made those assumptions but they're almost certainly not true, which isn't exactly ideal. Then again I can't quickly think of a better way to do it without more granular data. Maybe use a mixed effects model with multiple years of data from each school....
https://en.wikipedia.org/wiki/List_of_Yale_University_people
...and just sorted it by # of entries, you'd have a far superior list.
It stresses how school ranking are part of the system reducing economic mobility.
1. http://www.amazon.com/gp/product/0691155623/ref=as_li_tl?ie=...
from the article, apparently they excluded universities when the scorecard dataset was missing one or more factors; it would be useful if the tool at least showed the university anyway as a placeholder and indicated what data was missing.
