It could be "it's a new product, we randomly assign credit limits to see how it affects behavior".
It could be "it's a community property state and we're overexposed to this household if we give the second card the same limit as the first".
It could, realistically, be almost anything except for evil bankers deciding to use illegal criteria to underwrite that has a side effect of limiting the amount that can be charged to the account each month (you know, how they actually make money).
Oh, random CSRs don't get a pithy explanation of a multivariate nonlinear underwriting decision to poorly convey to customers? That's kind of precedented!
You can be implicitly biased without being aware of it. This is true for both humans and algorithms.
Yes, this is exactly what everyone thinks happened.
Nobody thinks people at Goldman-Sachs wrote
if (applicant.gender == "F")
limit /= 20
But regardless of how the thing happened, if millions of people are treated significantly differently for no reason other than their plumbing, that's a major problem. People have been talking about this "accidental proxy for gender" for years now; there's absolutely no excuse for no doing a basic sanity check to make sure that this kind of thing isn't happening.
edit: typo
Here you have a bank and a company known for secrecy.
But you acknowledge this;
> random CSRs don't get a pithy explanation
What the throwers of hissy fits are pointing to is that, by building blackboxes, you can get whatever result you want (which doesn't mean all results are expected) with plausible deniability.
As a side note, Idon't really understand the thread's appeal to credit scores here, considering that the TransUnion rating system is supposed to be the inferior one that Apple is looking to replace.
But most of all, I'm shocked by the number of people here outraged by Apple's behavior who will not even be considering switching off their iPhone (Twitter OP included, hell, he even posted a screenshot of his recurring TransUnion payment still being served via Apple Pay). I actually happen to agree with OP, but this blaise attitude towards real customer complaints has kept me from using Apple products for years.
There certainly is. Ignoring the anecdotal data of people replying to him who saw the same outcome, credit scores for women skew lower than for men.
One of many sources: https://www.federalreserve.gov/econres/notes/feds-notes/gend...
Which is crazy, because for most of my guy friends (myself included), saying that their significant others/women make significantly better financial decisions is putting it far too lightly. I basically wasn't making decisions at all (besides savings) until my s/o set me right.
This discrimination is tragic and disgusting.
1) according to him, the bureau they were using for underwriting data indicated she had higher credit score
2) "conditioning on credit score which is correlated with gender" is not conditioning on gender. The algorithms that derive credit score from a set of credit data do not condition on gender.
FTA:
“My belief isn’t there was some nefarious person wanting to discriminate. But that doesn’t matter. How do you know there isn’t an issue with the machine-learning algo when no one can explain how this decision was made?”
The anecdotal evidence, that many women are claiming that they received much lower credit limits than their partners, despite similar or superior credit conditions, would seem to indicate otherwise.
I don't believe anyone is saying that they're directly doing it. Just that their algorithms have that outcome.
Of course not, but why spoil the outrage mobs victimhood narrative du jour?
Just to be clear: I am not dismissing the possibility that the algorithm (meaning the training data, really) is gender biased, it just isn't clear from what I have seen in the tweet storm that this is necessarily the case.
E.g. I had an ex girlfriend who had a slightly lower income and slightly worse credit score (well, German Schufa score) than me, and yet she got offered like twice as credit when she applied for a card. I am guessing (just guessing) that this was due to her having paid off e.g. a car loan in the past and being generally more consumerist than me, while I never had taken out any major loans.
The axis doesn’t really matter. Maybe it’s gender, maybe it’s not. In any case, it should be about money, credit score, trustworthiness. Apparently, it’s not. Being a women could be one indicator. It might be "being black" for other people.
There seems to be something fishy going on and Apple doesn’t even know why. Hence, there is a bias.
That seems a little silly, though, since Goldman Sachs is the bank in charge of the card and approval process. Just like Amazon doesn't know anything about the processes that are used to determine credit worthiness for their cards (that falls to Chase and Synchrony), Apple is merely licensing services from GS.
On top of that, GS representatives that work in a call center aren't going to know anything about the details or inner workings of the process either. With the amount of potential abuse for such information, it feels like it would be irresponsible for those people to have access to that info.
The bigger issue is that this couple had a valid complaint and had to turn to Twitter to resolve it instead of GS having some formal method of requesting the info.
None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned.
When the process is set up like that, I think it's fair to assume the worst case scenarios and put the burden on the company to prove otherwise.
Adverse inference is a sensible way to combat secret decisions like this.
We all scoffed at this for a long time. ML makes it real, apparently.
I don't care what the context is. Whether its credit ratings, job applications, or college admissions.
The entire article is about questioning the black box, with regulatory force. Which is a good thing that is is going to be investigated.
>When the process is set up like that, I think it's fair to assume the worst case scenarios
Disagree. Consider it yes, but not assume it as a foregone conclusion.
> and put the burden on the company to prove otherwise.
Agree.
I started using credit cards a few years before my wife, and despite us both having excellent credit the first time she started applying for cards she was getting limits around $1K or $2K while at that point I was around 10x that. But after a few years of getting more cards and requesting limit increases, both of us now have roughly the same (fairly ridiculous) limits across all of our cards. Admittedly I don't have any cards issued by Goldman Sachs bank, but I can't imagine their algorithm would be much different than Amex, Citi, Chase, etc.
It would seem that his conclusion is warranted absent evidence to the contrary, the difference is too large to explain in ways that make any sense.
I think even if two people have their property in common (and if the algorithm even knows about that), it is still not unreasonable to believe that there is a higher probability of the one with higher income paying off his or her loans.
I had the same thought as you did when I first read it, for those of us not in US, it seems strange and we assume those Credit Score and Algorithm works as intended, like in your example of past loans and payment. And from experience they tend to be consistent and can be easily explained.
In this case however it seems something is very wrong.
I am still thinking and not sure if it is Apple to be blamed. It is easy to just point a finger to Apple, but in reality our financial and insurance system works pretty much the same way, changing these algorithm will require lots of work. Luckily GS is new to all these consumer business and changes are much easier as compared to other banks.
She actually got a higher limit than that but spent the majority of it. The complaint was that she had paid off the entire balance that she had spent already but the limit wasn't reset and wouldn't reset upon payment but would reset at the end of the billing cycle so she only had a $50 limit until that point.
My wife's credit history is longer and historically her score higher because I used no revolving credit until recently, and not using credit cards counts against you. Also married filing jointly...
For the Apple Card, I was given her limit several times over.
I did notice that the expected limit is shown before the hard pull on credit. This means they've got a pretty good idea before getting the latest credit report.
> GS ... probably put a ton of diligence into their risk model.
To your point about the risk model, I generally think the credit score the bureaus give us is wrong, while I think the decision GS made on the card limits is probably plausible ... if there's some chance or probability we might split up.
For instance, we work in different states, so data patterns might look like we are already separated? I also don't know if she put her income or household income. My income has been several times hers since long before we got together.
Rather than gender bias, I would imagine that given the probability of divorces at a certain age, executive level, and income bracket, that would put a thumb on the scale for ... what if you were not married filing jointly? Who makes more? What cash payment can they carry without going broke? Weighted that way, we should have different limits, and it's not a gender thing. This is the kind of correlation humans might not come up with, but ML probably would.
If this is purely actuarial, the decision might be correct, while not feeling moral.
To me it seems odd that married couples, filing jointly, get separate credit limits from each other at all. Aren't you one economic entity?
I find this likely too, the first time I applied I got denied and the credit score in the email was considerably lower (-100 points) that what is shown on Credit Karma.
Disclaimer: Work in financial services in risk management, interface with regulators. Opinions are my own.
He also adds "It gets even worse. Even when she pays off her ridiculously low limit in full, the card won’t approve any spending until the next billing period. Women apparently aren’t good credit risks even when they pay off the fucking balance in advance and in full."
Are we really to imagine the Apple/Goldman algorithm has some "if(gender.female){ cc_payment_terms = :discrimination }" sort of code in it?
FWIW I'm a male and have a mid-700s credit score and was denied Apple card approval. I am fairly certain there was no sex discrimination involved in the denial.
The whole idea behind decisions like these is that they have to be explainable and ML especially does not lend itself well for that.
I’ve been having the same issue since Monday 10/14. I paid my balance in full, money was taken from my Chase account right away and cleared next day, but my Apple Card available balance hasn’t updated. I’ve also gotten the same response from support. It could take several days. Funny thing I paid my wife’s card off a few weeks ago and it updated within seconds, same bank account and everything......
Not intentionally, but it's possible for black box AI algorithms to engineer such a model feature.
This is not an isolated incident.
I know nothing about this particular credit card. I'm not as up on credit stuff as I once was (and the world has changed a lot since then). But when I was a homemaker, I had a credit card in my name with a much higher limit than my husband had on any of his cards. That's not exactly the norm.
There are various factors that go into this. He's not wrong to suggest that there is a very big problem with employees having no idea what went wrong. I'm less confident that it is reasonable to infer gender is the entire explanation.
If you read the thread, he was specifically told the Apple Card uses Transunion, which is why they checked with Transunion, and found her score was higher on that report: https://twitter.com/dhh/status/1192945415538106369?s=21
Stop crying, stop bitching, grow up.
I worry about this a lot with the growing importance of algorithms and machine learning. You can't just not actively program the thing to discriminate and assume that is enough. You have to specifically program it to not discriminate.
I see this a lot, and the background assumption seems to be "in the real world, minority group A is actually riskier, dumber, or objectively worse in some other way, so in order to comply with anti-discrimination, we have to introduce special cases."
Maybe instead we should start with the assumption that women are NOT riskier, dumber, or objectively worse, and fix the likely bug, instead?
As world depends more and more on algorithms, we need to have more security around them. Especially as algorithms are used a lot as cost cutting, so reaching competent human to appeal error is becoming harder and harder.
If we don’t put more scrutiny around tech, we may end up living in kafkaesque world.
It’s got to be inferences like you said. And those are so much harder to spot.
> This is such a shallow, disappointing take. If we relegate all responsibility for discrimination to the individuals discriminated against, nothing is going to change! Individual action against structural problems is INSUFFICIENT.
Today it might be gender and race. That makes a lot of sense, because the alternative is to further entrench what are basically inheritances.
But aren't we just going to rattle on through and have the algorithms discover (whether we actually realise it or not) that, say, someone diagnosed with X is less creditworthy than someone diagnosed with Y, or that someone bullied in school is less creditworthy, or whatever else?
The whole point of ML is to extract this sort of information from a dataset.
Is it even possible or meaningful to create an unbiased model? Doesn't a model's profit imply bias, whether we currently consider it morally correct or not?
I'd be interested in an argument to convince me otherwise. My view at the moment is basically 'we spend all of this time building models, and then we have to stop using them because they're socially negative/immoral, but for a brief period shareholder value was maximised'?
Western society has mostly accepted the idea that—outside of some specific cases—we should avoid building systems and processes that systematically discriminate against people on the basis of a selection of characteristics which have historically attracted it. The exact application of this concept, the interpretation of it, and the boundaries of discrimination or protected characteristics will continue to be subject to gray areas and refinement. The rules are not perfect.
But I do think a better solution to the problem (“we have implemented a whizz-bang new technology which is inherently subject to bias”) is to either fix or discard that technology, rather than discarding the concepts of equalities regulation and civil rights.
My argument is that if we take the standpoint that we don't just want the metric to be whatever is short-term economically optimal for the designer of the model, we should just stop/ban it now, because we already know that we're going to have to kill it once we actually understand it properly.
It's only allowed now because we haven't figured out the bad things that are happening.
Inventing more and more categories that businesses are not permitted to discriminate upon is precisely the wrong approach - if we're talking about huge companies and not the bakery down the road, we need something that's more like 'you need a very good reason to exclude someone', rather than 'you can exclude someone for whatever reason they like, unless they're a member of the set of continually extending list of protected categories'.
Is there anything more to this than typical twitter mock indignation and outrage?
Now, if his wife was enormously wealthy before they met and this is the outcome, I suppose that would raise some eyebrows!
Intentional or not, it's possible they could have used something that proxied for gender.
This is still bad for Apple and for consumers. The "ALGORITHM" that can't be questioned, inspected, explained, or overruled is a massive failure. Whether its criminal justice, credit scores, or behavioral predictions, ceding authority to some "AI" overlord can't end in just or fair outcomes. (Scare quotes around "AI", because the black box may just be a chain of if/then statements that conveniently proxy in the worst of our institutional biases. Or it could be sophisticated ML... proxying those same things.)
I don't think we're done hearing about this. I'll be very interested in what Apple has to say about it, or what—if anything—is discovered. And it'll doubleplusungood if gender is an explicit input into the "ALGORITHM".
DHH is absolutely right to push back against all the respondents that offer plausible explanations. They're all missing the point. The point is that Apple should be providing a concrete explanation for their credit decisions. All credit providers should.
But on topic, she got the considerably better interest rate.
Every credit card company probably maintains a list of "big fish" - i.e. famous people - and grants them all a huge credit line.
It's a non-story.
You can't determine A, simple as that. The only way would be if Apple/GS comes out and says something like "he capped out the household credit limit", which Apple/GS would probably only tell him anyways.
B is a different issue, one with lots of room to actually converse over. But it's one which the author seems to pivot to after some reasonable arguments are presented as to why her credit limit was literally $57, making his argument for A very weak.
If I was as rich as DHH, who makes millions per year, I would cancel my credit cards and refuse to do business with companies like Goldman Sachs entirely.
I wouldn't give a crap about 2% cashback and I don't know why he does.
And despite the fact that he's probably wrong about the entire complaint, the least he could do is cancel his cards in protest. Instead, he seems to have accepted the "bribe" (his word) quite willingly.