If there existed a stand-alone test that could measure intelligence I would agree with you. But current standardized tests like the SAT and GRE measure a combination of intelligence, time spent studying, and money spent on educational resources like test prep tutors + materials.
Taking into account environmental variables is just an attempt to statistically control for the third variable and make the test more accurate at measuring intelligence.
I definitely think NYC is overcompensating by tossing testing entirely, and that's bad. But relying on raw scores is no less arbitrary or political than composite scores that attempt to control for confounding variables.
With respect to considering multiple factors, most elite colleges and universities already do that. For example, in the admissions processes I've been directly involved with the test scores had to meet certain thresholds depending on the program the candidate was applying to, but beyond that the reviewers focused on other factors. I mean if someone who didn't go to an elite high school got a stellar score then one of the reviewers would probably note it in the report, but otherwise it just wasn't a major factor in deciding who to admit.
