Twitch is hacked, and its source code leaked (opens in new tab)

I also worked for Twitch and can confirm what you're saying is true. These repo's any staff member had access to - including non-engineering staff.

Revenue for the longest time was as simple as navigating to a streamers dashboard as staff, but they did finally gate that away from staff who don't need to see that info, however I am sure there are other ways to obtain revenue reporting info.

I am assuming all data - including personal - has been compromised but so far, the data leaked is data that most staff would have access to in some way or another. Some may find that shocking, but this was not a "high level hack"

So much for information compartmentalization. Does the typical engineer need access to payment details for their daily work?

Why would an intern at Twitch have access to data in production?

Saying that no 'secrets' were leaked is effectively burying the lede.

I worked for a multi billion company and even 6 month contractors had access to basically everything with little effort.

No one in IT should have access to business data. That's simply best practice. Worst case would be a database engineer who has access to backups or some prod data for troubleshooting, and even that should be under tight control with good access accounting.

Could have been a hack of a twitch engineer's laptop or something like that.

Sounds like someone in Twitch Security needs to take a course on Least Privileged Access then

Given the torrent is labeled "twitch-leaks-part-one" I'm curious too as to what they have. The torrent breaks out into a lot of compressed volumes, so it's clear this wasn't just a backup file, but a curated collection of files. I'm very curious if we will see any other amazon related leaks come from it.

Either way, I can only imagine the chaos inside as they try to figure out what has transpired here.

>It's possible that this first leak was just to establish trust so they can random or auction password hashes later.

Password hashes are relatively useless though? Once the leak is announced I imagine most of the big targets will rotate their credentials. Then the next thing you need to do is spend possibly thousands in CPU time bruteforcing bcrypt hashes. Then I'm not sure what you can even do with those.

I'm not criminally creative but I imagine you could make more by abusing trust with payment processors or fraudulent invoices.

Maybe that Twitch is competent in the password department so they decided against it? But thinking about it, although it's unclear if two-factor secrets are included in the leak, but maybe the two-factor secrets may be usable to someone who has already the password of a victim. Unless it's the dongle-type one (WebAuthn/FIDO), the secret is common to both the server and the user, so two-factor bypass is almost certain in this case.

Doesn't seem likely to me. If the attacker has password hashes then they would want to keep this attack quiet so that the buyer of the hashes would have time to compute the passwords. If Twitch gets wind of this happening then a simple password reset would foil any efforts.

Twitch has not been known to be transparent about anything.

Gosh - I've worked at shops where we handled multi-terabyte images and we'd regularly stream large chunks of that while debugging tools. I've also worked at places where data was king and 125GB of stuff might be a reasonable dispatch of data to help someone debug.

The volume of data is irrelevant - source code is usually teensy tiny and of far more value to companies than, say, three months of livestream chat logs.

I'm not certain what security hardware you're thinking of - but I'm pretty sure I hate it already since it doesn't effectively guard anything while making everyone's lives difficult. For effective corporate security you need 1) data use policies and 2) access control lists - both of those are generally more effectively implemented at an entirely software level.

Trying to protect against leaking developers/employees is like trying to protect against lone gunman terrorists: useless. And, if you try anyway, it is likely to cause more annoyance to everyone involved than actual protection (think TSA).

If the bulk of it is a git repo, it's probably expected that every engineer will download it regularly.

> Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.

I am trying to recall, but I am pretty sure when I worked in Microsoft Office that a build would pull down many tens of gigabytes of data.

125GB in one day from the build system wouldn't be uncommon!

There was a fad for tools that accomplished this in enterprise networks, with much clearer rules for who needs to access what (it was called "data loss prevention", or DLP) and those tools for the most part don't work. This is a harder problem than it looks like.

> It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.

Remember that Twitch handles streams. Good luck implementing this without having all sorts of false alarms everywhere.

Plus, you don't have to exfiltrate 125GB in one go.

I feel like once you have it pulled downm, it would be as simple as an upload to s3 (which wouldn't trigger any flags), then making the bucket public whenever you want. Hell, S3 used to (still does?) support being part of a torrent swarm...

Why would that help? They just have to accumulate work over a period of time and then 'lose' their laptop.

That's 6.25GB/day over a 20 day working month. More time, less data per work day, harder to detect.

ML engineers / data scientists are regularly moving terabytes of data around at Amazon.

Indeed , how could this happen, really curious.

So let's say someone with access to all GitHub repos gave the password to someone else, maybe then it was downloaded from another machine?

Or someone stole the credentials and downloaded from another machine?

Or someone got access to such a machine?

It's it not possible to prevent these cases?

How long does such a download take?

Shouldn't that be discreet devices? Or do they make a really high pitched whine with a big flashing light when they start transferring data?

It's funny because for me each letter of FAANG is an hypergrowth B2C company...

Conflating AWS security with twitch security is probably the wrong way to think about it.

Within Amazon those are almost going to be two entirely separate companies, with very different security focuses.

The idea that Amazon is monolithic and uniform wasn't true when I left there in 2006, and I'm certain it is less so now.

And that isn't just that its related to the merger, but that fundamentally its different business orgs with different focus.

I always had the impression that Twitch were operating in a largely independent fashion. For instance, it had been an open secret for years that one of their executives had been sexually harassing female streamers. Only a year ago he was finally fired. If Amazon had a firmer grip on Twitch, I'm sure they would have stepped in much earlier.

If you go back to the Adobe software breach circa 2013, a large part of their issues were the bolt on connections between acquisitions. It's honestly the most common thing I see in the startup world.

> From what I heard about Twitch-interns over the years, it seems the company is more a third-rate-s*hole that grew too big too fast and accumulated a huge amount of technical debt and fatal security flaws.

I mean this as a genuine question, but is there any company that didn't end up like this after an exponential growth phase? I'm not saying it's okay, but this feels par for the course. I've now been at two start ups during that hockey stick growth time and both went through this as well.

I'd be curious if anyone here has worked at a large, fast growing tech company where they didn't accumulate a ton of technical debt during growth. If so, what did the company do to prevent that?

> Does anyone know if Twitch employees have two factor auth?

Yes, IIRC everyone at Amazon has a hardware security key (which is more secure than the standard mobile app TOTP most of us use everywhere online).

Every Twitch Developer has 2FA even 3rd party developers are required to have 2FA I also think, but don't know, that this applies to Twitch Broadcaster Partners as well in order to have their tax information in the system.

Luckily iirc from a conversation with a senior Twitch engineer the Tax information backend has been migrated to Amazon. So hopefully that did not leak... Because that would be full legal name and addresses of a ton of streamers that likely have stalkers.

Who decides what speech is radical enough to compromise the privacy of users?

And if speech is "radical" meaning to the point of illegality, shouldn't the legal system decide, rather than the court of public opinion?

>Because you expect Amazon to put security priority over new features and profit?

I don't know what you think Amazon stands for, but Amazon runs the largest cloud hosting service in the world - AWS, which not only runs a large number of other large companies but governments as well. I know, first hand, that their datacenter security protocols are state of the art.

Amazon has a much larger surface attack area so if they were playing fast and loose with security, chances are we would know already.

EC2, Amazon's cash cow, competes with nearly identical offerings from Microsoft and Google, and is not a place where additional features are often all that valuable to customers. Any sort of breach like this on EC2 would seriously hurt Amazon's bottom line and they know it.

On one hand I understand why you'd ban that kind of content, on the other it's essentially public information now... what's the point.

They = you. It's fine to be honest, you're not exactly making it unobvious.

> I've never worked at "web scale" so I'd probably learn a lot.

As someone who has worked at both large and small companies, you'd probably be disappointed.

A lot of it is probably hacked together -- like, embarrassingly hacked together lol

Well, you know what they say, "Self help is the best help."

I hear Netflix has a good tech blog ;)

Yep, we saw it happen live.

Any bits you recall from the chat?

Ironic? Why?

aaand new ad bypass dropped 4 hours ago :)

>You can check this by loading a stream with MPV

I watch all of my twitch using mplayer. "magic incantations" when generating access token is what produces ad free .m3u8. For example early methods involved setting origin and/or referrer headers to internal Amazon systems.

Please don't rub the services, it causes unnecessary friction, and wear & tear.

?

Yup, and AWS Code*

Sir (Madame?), I ask you one simple question:

Was Twitch built in 10 years, or over just a few?

Steam was built since I was in FUCKING high school. Im old now, well over 30.

Apples, and blueberries.

Bluebarry, Drewbarry, tomato, ToMaHtoH.

Fuck their stupid ass streaming code, it’s a giant crud app, only their devops team can take credit for scaling, everyone else is not worth a shit, sorry, thats life, I gotta Leetcode too, and ur code isn’t worth me reading it, leaked or not).

inefficiency.

I work in a small nation state.

That doesn't stop CV-hungry engineers from finding ways to overcomplicate it.

(I do agree with you on this topic in general)

Netflix & Youtube in shambles.

One shouldn't aspire to be a Zuckerberg/Dorsey.

I’m so misread, Twitch is a lot of luck, so is all of these companies. Show me the the source code for luck. I don’t give a fuck if you leaked a video streaming crud app code lol.