I then went on to work for multiple firms that placed a premium on candidates from Ivy League/Top Tier (Stanford/Duke etc) candidates.
This taught me that:
- Their are pros and cons to any selection criteria.
- There are smart people everywhere. One of the smartest people I ever worked for spent several years in prison for drug dealing. He was on par with many of the Managing Directors I've worked for
- There was a study where they asked big bank recruiters which school consistently produced people who were excellent employees 2-3 years out from hiring and the answer was Penn State (not my alma mater)
- There used to be "manager's choice" hires where managers had 1 slot in a training program where they could select whoever they wanted. Sometimes that was terrible. Sometimes that person was top of their training program.
- Smart people are just as capable as creating problems as less intelligent people. Smart people, in some ways, are better at creating problems. Especially if the incentives reward them for creating those problems.
If we move to using just a small number of AI models to help do things like hiring, we will amplify biases and possibly completely lock out portions of the population. We need to be very careful when using AI systems to evaluate people in general -- not because they might be biased (which they might be), but because even a small bias, if used by virtually everyone, can be damning.
Definitely open to opposing or critical views
I'm not saying AI is not biased, but this study does not prove that.
[0] https://arxiv.org/pdf/2605.27371
From the paper:
> Fig. 1. The pymetrics process. > Stage 1: Applicants apply to positions. > Stage 2: Applicants are directed to the pymetrics platform to play assessment games. > Stage 3: pymetrics algorithms use applicant gameplay features to recommend 58.2% of applicants per position on average. > Stage 4: Employers decide which applicants to interview or hire, typically rejecting applicants that were not recommended by pymetrics.
> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.
It sounds like this study was using real-world applicants, and the other study they're comparing against was using synthetic applicants.
Consider the chance of being accepted as being composed of signal+bias+noise. Noise is random. Signal is a per-applicant value, and what's meant to be measured. Bias is a per-group value, and an artifact of the measuring process.
If acceptance/rejection is independent between positions applied for (as in the synthetic applicant study), that suggests that it's random or composed entirely of noise; ie there is no signal; ie the applicants are all equally qualified.
If acceptance/rejection is correlated, that means there is some nonzero amount of (signal+bias). But real-world applicants are not all identical, so there should be some amount of signal. So you can't just assume zero signal in order to infer that there must be bias.
We find applicants are more likely to be rejected from every position they apply to than would be predicted by the baseline of each position making statistically independent decisions.
Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer. Like online dating, most employers are looking for some baseline indicators that you are going to be successful and stable.
High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations
That's a pretty common sense legislation to me.
"Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards."
That seems like a nonsensical way to measure racial discrimination. What could justify it?
They find "disparate impact" of pymetrics across racial groups, but it doesn't seem like they controlled for anything.
>If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring.
I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, was considered at the end.
https://www.yahoo.com/news/us/articles/california-judge-upho...
AI works by learning patterns. So it will become bias by just learning from factors like education history, schools attended, employment history, ZIP codes, or geographic location. Those 3 factors alone are an easy proxy for race.
And if you add names into the equation (if the AI was trained without removing applicant names), the model can become even more bias.
I guess this one just compounds.
I see nothing that shows any system was making a decision on race. How is the race being presented to the AI?
All this is showing from what I can see, is that certain groups of people were more often denied a next step in the process - but why?
Was the AI going by spelling and grammar? Were there names that were different but the rest of the resume was exactly the same? Were there pictures?
There were mentions that the rate of each group may be more prominent in the data when you split apart different types of jobs instead of all jobs in aggregate.. One could read that like it's inferred; that more warehouse jobs are offered to a race and less admin jobs.. but that same would happen if AI was more focused on perfect grammar for one job and it was not as much of a factor for a warehouse job.
Also if the people applying for the various jobs were self selecting, acceptance percentages this would skew things based upon which ones were applied / not applied to right?
There are so many ways you could draw conclusions like this from data, however correlation is not causation, yet this seems to say it is.
I feel this is an important thing to watch, but Stanford may not be the place to trust with 'Policy Recommendations' as it's very unclear there is any proof that 'AI Hiring Tools Yield Racial Bias and Systemic Rejection' from this study and paper.
PS - now that I see the HN title did not have the word "can" in it, and the title of the article is actually "Tools Can Yield" - maybe that is less accusing and more noting.
Only 40% self report gender/race
no resume data, no education information, degrees, schools, GPA, major, work experience, skills/certifications
Zero job qualifications
> We follow 3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors. Each job application was assessed by an AI hiring tool built by a single third-party vendor.
3.4 million people applying to just 150 employers... Who are all using just 1 platform. WTF. This is where the discrimination is happening. Why the f do 3.4 million people feel forced to apply to just 150 employers and why the f do all these 150 employers feel forced to use just one platform. WTF.
> 30% of Black applicants apply to at least one position that demonstrates adverse impact against Black applicants.
The whole thing reads like a tautology.
I would be surprised if the results were different.
I tried it before, and discrimination is there, I would get one resume rejected quickly and few days later the same company would invite another resume for a screening call. I tried this before and after AI hype, results weren’t that different btw, and that was tested in US and Canada employers only.
(I assume they're just using a big LLM for this, it doesnt say, it just says "AI" when they say "AI like that they usually mean LLM".. A custom trained hiring ML system would be better)
Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.
> There is no such thing as anti-white racism.
If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is. Racism is pervasive, insidious and systemic.
A good example in the hiring space is what's called the "second syllable name problem". Traditionally Afrcian names often stress the second syllable (eg Jamal, Lakisha, Malik, Lashonda). Studies have shown that such names have higher rejection rates in job applications [1]. So if you're wondering about the four-fifths rule, it's because it exposes this kind of bias. It's not proof of bias. It simply means further investigation is required.
The problem with AI hiring tools is the logic is opaque. You have no idea why an AI system is rejecting or selecting candidates and you may find it's doing something illegal. Some companies want to hide behind this opaqueness, arguing that if no explicit decision was made then there is no bias. But that's not how system racism works.
There are many such signals that correlate with race that if they affect selection rate, it could be a problem. Did you go to an HBCU? Was your high school in a minority-majority area? What about your previous employers?
This kind of bias doesn't have to be intentional.
[1]: https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...
Too many of these studies only focus on percentages and the end result is unqualified candidates getting hired from minority groups at the expense of qualified ones.
Happy to share some sample reports if anyone is interested!
