1) those directly involved with the incident, or employees of the same company. They have too much to lose by circumventing the PR machine.
2) people at similar companies who operate similar systems with similar scale and risks. Those people know how hard this is and aren’t likely to publicly flog someone doing their same job based on uninformed speculation. They know their own systems are Byzantine and don’t look like what random onlookers think it would look like.
So that leaves the rest, who offer insights based on how stuff works at a small scale, or better yet, pronouncements rooted in “first principles.”
We should always be doing (the thing we want to do)
Somme examples that always get me in trouble (or at least big heated conversations)
1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release".
2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time.
There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"
Unless Netflix eng decides to release a public postmorterm, we can only speculate. In my time organizing small-time live streams, we always had up to 3 parallel "backup" streams (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt they could simply summon any of these providers in, but I guess Akamai / Cloudflare would have been up for it.
A company I used to work for ran a few Super Bowl ads. The level of traffic you get during a Super Bowl ad is immense, and it all comes at you in 30 seconds, before going back to a steady-state value just as quickly. The scale pattern is like nothing else I've ever seen.
Super Bowl ads famously seven million dollars. These are things we simply can't repeat year over year, even if we believed it'd generate the same bump in recognition each time.
Also, "No experience in" really? You have no idea if that's really the case
Rolling Stone reported 120m for Tyson and Paul on Netflix [1].
These are very different numbers. 120m is Super Bowl territory. Could Hotstar handle 3-4 of those cricket matches at the same time without issue?
[0] https://www.the-independent.com/sport/cricket/india-pakistan...
[1] https://www.rollingstone.com/culture/culture-news/jake-paul-...
It might just have been easier to start from scratch, maybe using an external partner experienced in live streaming, but the chances of that decision happening in a tech-heavy company such as Netflix that seems to pride itself on being an industry leader are close to zero.
depending on whom you ask, the bitrate used by the stream is significantly lower than what is considered acceptable from free livestreaming services, that albeit stream to much, much smaller audience.
without splitting hairs, livestreaming was never their forte, and going live with degradation elsewhere is not a great look for our distributed computing champ.
1. Netflix is a 300B company, this isn't a resources issue.
2. This isn't the first time they have done live streaming at this scale either. They already have prior failure experience, you expect the 2nd time to be better, if not perfect.
3. There were plenty of time between first massive live streaming to second. Meaning plenty of time to learn and iterate.
Peak traffic is very expensive to run, because you're building capacity that will be empty/unsused when the event ends. Who'd pay for that? That's why it's tricky and that's why Akamai charges these insane prices for live streaming.
A "public" secret in that network layer is usually not redundant in your datacenter even if it's promised. To have redundant network you'd need to double your investment and it'll seat idle of at 50% max capacity. For 2hr downtime per year when you restart the high-capacity routers it's not cost efficient for most clients.
What was the previous fail?
I spoke to multiple Netflix senior technicians about this.
They said that's the whole shtick.
Live has changed over the years from large satellite dishes beaming to a geosat and back down to the broadcast center($$$$$), to microwave to a more local broadcast center($$$$), to running dedicated fiber long haul back to a broadcast center($$$), to having a kit with multiple cell providers pushing a signal back to a broadcast center($$), to having a direct internet connection to a server accepting a live http stream($).
I'd be curious to know what their live plan was and what their redundant plan was.
This isn’t NFLX’s first rodeo in live streaming. Have seen a handful of events pop up in their apps.
There is no excuse. All of the resources and talent at their disposal, and they looked absolutely amateurish. Poor optics.
I would be amazed if they are able to secure another exclusive contract like this in the future.
I guess we now know the limits of what "at scale" is for Netflix's live-streaming solution. They shouldn't be failing at scale on a huge stage like this.
I look forward to reading the post mortem about this.
Every major network can broadcast the Super Bowl without issue.
And while Netflix claims it streamed to 280 million, that’s if every single subscriber viewed it.
Actual numbers put it in the 120 million range. Which is in line with the Super Bowl.
Maybe Netflix needs to ask CBS or ABC how to broadcast
That’s a very different area to transmission of live to end users.
Sure thing, but also, how much resources do you think Netflix threw on this event? If organizations like FOSSDEM and CCC can do live events (although with way smaller viewership) across the globe without major hiccups on (relatively) tiny budgets and smaller infrastructure overall, how could Netflix not?
The Masters app is the only thing that comes close imo.
Cable TV + DVR + high speed internet for torrenting is still an unmatched entertainment setup. Streaming landscape is a mess.
It's too bad the cable companies abused their position and lost any market goodwill. Copper connection direct to every home in America is a huge advantage to have fumbled.
Reliable and redundant multicast streaming is pretty much a solved problem, but it does require everyone along the way to participate. Not a problem if you're an ISP offering TV, definitely a problem if you're Netflix trying to convince every single provider to set it up for some one-off boxing match.
It was really bad. My Dad has always been a fan of boxing so I came over to watch the whole thing with him.
He has his giant inflatable screen and a projector that we hooked up in the front lawn to watch it, But everything kept buffering. We figured it was the Wi-Fi so he packed everything up and went inside only to find the same thing happening on ethernet.
He was really looking forward to watching it on the projector and Netflix disappointed him.
What did your Dad think about the 'boxing'?
This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...
They never tried to do a live reunion again. I suppose they should have to get the experience. Because they are hitting the same problems with a much bigger stake event.
In particular, they have been revising their compensation structure to issue RSUs, add in a bunch of annoying review process, add in a bunch of leveling and titles, begin hiring down market (e.g. non-sr employees), etc.
In addition to doing this, shuffling headcount, budgets, and title quotas around has in general made the company a lot more bureaucratic.
I think, as streaming matured as a solution space, this (what is equivalent to cost-cutting) was inevitable.
If Netflix was running the same team/culture as it was 10 years ago, I'd like to say that they would have been able to pull of streaming.
They stream plenty of pre recorded video, often collocated. Live streaming seems like something they aren’t yet good at.
I always assumed youtube was top dog for performance and stability. I can’t remember the last time I had issues with them and don’t they handle basically more traffic than any other video service?
It has been pretty useless. At the moment seems to be working only when running in non-live mode several minutes behind.
So if there are 1 million trying to stream it, that means they would lose $15 million. So.. they might only give a partial refund.
But people should push for an automatic refund instead of a class action.
Most people pay Netflix to watch movies and tv shows, not sports. If I hadn't checked Hacker News today, I wouldn't even know they streamed sports, let alone that they had issues with it. Even now that I do, it doesn't affect how I see their core offering, which is their library of on-demand content.
Netflix's infrastructure is clearly built for static content, not live events, so it's no shock they aren't as polished in this area. Streaming anything live over the internet is a tough technical challenge compared to traditional cable.
Maybe if we're not counting Youtube as 'streaming', but in my mind no one holds a candle to YT quality in (live)streaming.
> Some Cricket graphs of our #Netflix cache for the #PaulVsTyson fight. It has a 40 Gbps connection and it held steady almost 100% saturated the entire time.
YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated to stream simultaneously to hundreds of millions live without any issues. This was Netflix's chance to do this and they have largely failed at this.
There are no excuses or juniors to blame this time. Quite the inexperience from the 'senior' engineers at Netflix not being able to handle the scale of live-streaming which they may lose contracts for this given the downtime across the world over this high impact event.
Very embarrassing for a multi-billion dollar publicly traded company.
In my country every time there's a big football match the people who try to watch it on the internet face issues.
Not trying to downplay their complexity, but last I heard Netflix is splitting the shows in small data chunks and just serves them as static files.
Live streaming is a different beast
8m vs 60m. And not in 4K. Not a great choice for comparison.
Live is a lot harder than on demand especially when you can't estimate demand (which I'm sure this was hard to do). People are definitely not understanding that. Then there is that Netflix is well regarded for their engineering not quite to the point of snobbery.
What is actually interesting to me is that they went for an event like this which is very hard to predict as one of their first major forays into live, instead of something that's a lot easier to predict like a baseball game / NFL game.
I have to wonder if part of the NFL allowing Netflix to do the Christmas games was them proving out they could handle live streams at least a month before. The NFL seems to be quite particular (in a good way) about the quality of the delivery of their content so I wouldn't put it past them.
If anyone was waiting for the main card to tune in, I recommend tuning in now.
Also, no buffering issues on my end. Have to wonder if it's a regional issue.
I'd think this isn't too crazy to stress test. If you have 300 million users signed up then you're stress test should be 300 million simultaneous streams in HD for 4 hours. I just don't see how Netflix screws this up.
Maybe it was a management incompetence thing? Manager says something like "We only need to support 20 million simultaneous streams" and engineers implement to that spec even if the 20 million number is wildly incorrect.
Anyway, network cable is the only way to go!
A combination of hubris and groupthink.
- Months ago, the "higher ups" at Netflix struck a deal to stream the fight on Netflix. The exec that signed the deal was probably over the moon because it would get Netflix into a brand new space and bring in large audience numbers. Along the way the individuals were probably told that Netflix doesn't do livestreaming but they ignored it and assumed their talented Engineers could pull it off.
- Once the deal was signed then it became the Engineer's problem. They now had to figure out how to shift their infrastructure to a whole new set of assumptions around live events that you don't really have to think about when streaming static content.
- Engineering probably did their absolute best to pull this off but they had two main disadvantages, first off they don't have any of the institutional knowledge about live streaming and they don't really know how to predict demand for something like this. In the end they probably beefed up livestreaming as much as they could but still didn't go far enough because again, no one there really knows how something like this will pan out.
- Evening started off fine but crap hit the fan later in the show as more people tuned in for the main card. Engineering probably did their best to mitigate this but again, since they don't have the institutional knowledge of live events, they were shooting in the dark hoping their fixes would stick.
Yes Netflix as a whole screwed this one up but I'm tempted to give them more grace than usual here. First off the deal that they struck was probably one they couldn't ignore and as for Engineering, I think those guys did the freaking best they could given their situation and lack of institutional knowledge. This is just a classic case of biting off more than one can chew, even if you're an SV heavyweight.
These failures reflect very poorly on Netflix leadership. But we all know that leadership is never held accountable for their failures. Whoever is responsible for this should at least come forward and put out an apology while owning up to their mistakes.
[0] https://time.com/6272470/love-is-blind-live-reunion-netflix/
It wasn't their first live event. A previous live event had similar issues.
If you can't provide the service you shouldn't sell it?
They failed. Full stop. There is no valid technical reason they couldn’t have had a smooth experience. There are numerous people with experience building these systems they could have hired and listened to. It isn’t a novel problem.
Here are the other companies that are peers that livestream just fine, ignoring traditional broadcasters:
- Google (YouTube live), millions of concurrent viewers
- Amazon (Thursday Night Football, Twitch), millions of concurrent viewers
- Apple (MLS)
NBC live streamed the Olympics in the US for tens of millions.
So Netflix had 2 factors outside of their control
- unknown viewership
- unknown peak capacities outside their own networks
Both are solvable, but if you serve "saved" content you optimize for different use case than live streaming.
Live events are difficult.
I'll also add on, that the other things you've listed are generally multiple simultaneous events; when 100M people are watching the same thing at the same time, they all need a lot more bitrate at the same time when there's a smoke effect as Tyson is walking into the ring; so it gets mushy for everyone. IMHO, someone on the event production staff should have an eye for what effects won't compress well and try to steer away from those, but that might not be realistic.
I did get an audio dropout at that point that didn't self correct, which is definitely a should have done better.
I also had a couple of frames of block color content here and there in the penultimate bout. I've seen this kind of stuff on lots of hockey broadcasts (streams or ota), and I wish it wouldn't happen... I didn't notice anything like that in the main event though.
Experience would likely be worse if there were significant bandwidth constraints between Netflix and your player, of course. I'd love to see a report from Netflix about what they noticed / what they did to try to avoid those, but there's a lot outside Netflix's control there.
- 120m viewers [1]
- Entire Netflix CDN Traffic grew 4x when the live stream started [2]
[1] https://www.rollingstone.com/culture/culture-news/jake-paul-...
It's not full stop. There are reasons why they failed, and for many it's useful and entertaining to dissect them. This is not "making excuses" and does not get in the way of you, apparently, prioritizing making a moral judgment.
I’m pretty confident that when the post mortem is done the issues are going to be way closer to the broadcast truck than the user.
> But the real indicator of how much Sunday’s screw-up ends up hurting Netflix will be the success or failure of its next live program—and the next one, and the one after that, and so on. There’s no longer any room for error. Because, like the newly minted spouses of Love Is Blind, a streaming service can never stop working to justify its subscribers’ love. Now, Netflix has a lot of broken trust to rebuild.
And you’ll never guess which Presidential candidate they both support!
edit: literally a nginx gateway timed out screen if you view the response from the cdn... wow
In a protocol where a oft-repeated request goes through multiple intermediaries, usually every intermediate will be able to cache the response for common queries (Eg: DNS).
In theory, ISPs would be able to do the same with the HTTP. Although I am not aware of anyone doing such (since it will rightfully raise concerns of privacy and tampering).
Now TLS (or other encryption) will break this abstraction. Every user, even if they request a live stream, receives a differently encrypted response.
But live stream of a popular boxing match has nothing to do with the "confidentiality" of encryption protocol, only integrity.
Do we have a protocol which allows downstream intermediates eg ISPs to cache content of the stream based on demand, while a digital signature / other attestation being still cryptographically verified by the client?
I don't see it much mentioned the last few years, but the research groups have ongoing publications. There's an old 2006 Van Jacobson video that is a nice intro.
I assume this came down to some technical manager saying they didn't have the human and server resources for the project to work smoothly and a VP or something saying "well, just do the best you can.. surely it will be at least a little better than last time we tried something live, right?"
I think there should be a $20 million class action lawsuit, which should be settled as automatic refunds for everyone who streamed the fight. And two executives should get fired.
At least.. that's how it would be if there was any justice in the world. But we now know there isn't -- as evidenced by the fact that Jake Paul's head is still firmly attached to his body.
I have done live streaming for around 100k concurrent users. I didn't setup infrastructure because it was CloudFront CDN.
Why it is hard for Netflix. They have already figured out CDN part. So it should not be a problem even if it is 1M or 100M. because their CDN infrastructure is already handling the load.
I have only work with HLS live streaming where playlist is constantly changing compared to VOD. Live video chunks work same as VOD. CloudFront also has a feature request collapsing that greatly help live streaming.
So, my question is if Netflix has already figured out CDN, why their live infrastructure failing?
Note: I am not saying my 100k is same scaling as their 100M. I am curious about which part is the bottleneck.
100k concurrents is a completely different game compared to 10 million or 100 million. 100k concurrents might translate to 200Gbps globally for 1080p, whereas for that same quality, you might be talking 20T for 10 million streams. 100k concurrents is also a size such that you could theoretically handle it on a small single-digit number of servers, if not for latency.
> CloudFront also has a feature request collapsing that greatly help live streaming.
I don't know how much request coalescing Netflix does in practice (or how good their implementation is). They haven't needed it historically, since for SVOD, they could rely on cache preplacement off-peak. But for live, you essentially need a pull-through cache for the sake of origin offload. If you're not careful, your origin can be quickly overwhelmed. Or your backbone if you've historically relied too heavily on your caches' effectiveness, or likewise your peering for that same reason.
200Gbps is a small enough volume that you don't really need to provision for that explicitly; 20Tbps or 200Tbps may need months if not years of lead time to land the physical hardware augments, sign additional contracts for space and power, work with partners, etc.
In fact, optimizing for later can hurt the former.
Would be interesting to read any postmortems on this failure. Maybe someone will be kind enough to share the technical details for the curious crowd.
I'm sure they will get it figured out.
That's the plain-text message I see when I tried to refresh the stream.
Follow-up:
My location: East SF Bay.
Now even the Netflix frontpage (post login, https://www.netflix.com/browse ) shows the same message.
The same message even in a private window when trying to visit https://www.netflix.com/browse
The first round of the fight just finished, and the issues seem to be resolved, hopefully for good. All this to say what others have noted already, this experience does not evoke a lot of confidence in Netflix's live-streaming infrastructure.
I remember when ESPN started streaming years back, it was awful. Now I almost never have problems with their live events, primarily their NHL streams.
Anyway, I wouldn’t be surprised if they were prioritizing mobile traffic because it’s more forgiving of shitty bitrate.
All your attacking power comes from your legs and hips, so if his legs weren’t stable he didn’t have much attacking power.
I think he gave it everything he had in rounds 1 and 2. Unfortunately, I just don’t think it was ever going to be enough against a moderately trained 27 year old.
There is a reason that cable doesn’t stream unicast and uses multicast and QAM on a wire. We’ve just about hit the point where this kind of scale unicast streaming is feasible for a live event without introducing a lot of latency. Some edge networks (especially without local cache nodes) just simply would not have enough capacity, whether in the core or peering edge, to do the trick.
Isn't live streaming at scale already solved problem by cable companies? I never seen ESPN going down during a critical event
To understand how to do things correctly look at something like pornhub who handle more scale than Netflix without crying about it.
The other day I was having this discussion with somebody who was saying distributed counter logic is hard and I was telling them that you don't even need it if Netflix didn't go completely mental on the microservices and complexity.
I'm tired of all this junk entertainment which only serves to give people second-hand emotions that they can't feel for themselves in real life. It's like, some people can't get sex so they watch porn. People can't fight so they watch boxing. People can't win in real life so they play video games or watch superhero movies.
Many people these days have to live vicariously through random people/entities; watch others live the life they wished they had and then they idolize these people who get to have everything... As if these people were an intimate projection of themselves... When, in fact, they couldn't be more different. It's like rooting for your opponent and thinking you're on the same team; when, in fact, they don't even know that you exist and they couldn't be more different from you.
You're no Marvel superhero no matter how many comic books you own. The heroes you follow have nothing to do with you. Choose different heroes who are more like you. Or better; do something about your life and give yourself a reason to idolize yourself.
And then all these sessions lag, or orphan taking up space, so many reconnections at various points in the stream.
System getting hammered. Can't wait for this writeup.
I'd expect the NFL games to have a largely American audience, but today's boxing event attracted a global audience.
Unfortunately, except for the women's match, the fights were pretty lame...4 of the 6 male boxers were out of shape. Paul and Tyson were struggling to stay awake and if you were to tell me that Paul was just as old as Tyson I would have believed it.
https://www.theverge.com/2024/11/16/24298338/netflix-mike-tt...
Assuming Netflix used its extensive edge cache network to distribute the streams to the ISPs. The software on the caching servers would have been updated to be capable of dealing with receiving and distributing live streamed content, even if maybe the hardware was not optimal for that (throughput vs latency is a classic networking tradeoff).
Now inside the ISPs network again everyting would probably be optimized for the 99.99% usecase of the Netflix infra: delivering large bulk data that is not time sensitive. This means very large buffers to shift big gobs of packets in bulk.
As everything along the path is trying to fill up those buffers before shipping to the next router on the path, some endpoints aware this is a live stream start cancelling and asking for more recent frames ...
Hilarity ensues
I don’t know if it’s still the case, but in the past some devices worked better than others during peak times because they used different bandwidth providers. This was the battle between Comcast and Cogent and Netflix.
For me it was buffering and low resolution, on the current AppleTV model, hardwired, with a 1Gbps connection from AT&T. Some streaming devices may have handled whatever issues Netflix was having better than others, but this was clearly a bigger problem than just the streaming device.
I think that every time I wait for Paramount+ to restart after its gone black in picture on picture, and yet, I’n still on Paramount+ and not Netflix, so maybe that advantage isn’t real.
I specifically found the Netflix suite for Spring very lacking, and found message oriented architectures on something like NATS a lot easier to work with.
Since then I am very used to it because our institutional web sites traditionally crash when there is a deadline (typically the taxes or school inscriptions).
As for that one, my son is studying in Europe (I am also in Europe), he called me desperate at 5 am or so to check if he is the only one with the problem (I am the 24/7 family support for anything plugged in). After having liberally insulted Netflix he realized he confirmed with his grandparents that he will be helping them at 10 :)
I remember a lot of trade magazines in the late 1990's during the dot com boom talked about how important it would be.
https://en.wikipedia.org/wiki/IP_multicast
I never hear about it anymore. Is that because everyone wants to watch something different at their own time? Or is it actually working just fine now in the background? I see under the "Deployment" section it mentions IPTV in hotel rooms.
Some buffering issues for us, but I bet views are off the charts. Huge for Netflix, bad for espn, paramount, etc etc
This is bad for Netflix imo.
I had some technical experience with live video streaming over 15 years ago. It was a nightmare back then. I guess live video is still difficult in 2024. But congrats to Jake Paul and boxing fans. It was a great event. And breaking the internet just adds more hype for the next one.
I’m in the Pacific Northwest. I wonder if we got lucky on this or just some areas got unlucky.
All of the conditions was perfect for Netflix, and it seems that the platform entirely flopped.
Is this what chaos engineering is all about that Netflix was marketing heavily to engineers? Was the livestream supposed to go down as Netflix removed servers randomly?
Oddly having watched PPV events via the high seas for years, it feels normal...
I didn’t watch it live (boxing has lot its allure for me) but vicariously lived through it via social feed on Bluesky/Mastadon.
Billions of dollars at their disposal and they just can’t get it right. Probably laid off the highly paid engineers and teams that made their shit work.
More likely overpaying a bunch of posers without the chops, a victim of their own arrogance.
The fight itself was lame which worked in their favor. No one really cared about not being able to see every second of the "action". It's not like it was an NBA game that came down to the last second.
In boxing you are old by 32 or maybe 35 year old for heavy weight, and everything goes down very very fast.
End of rant.
A see in the comments multiple people talking about how "cable" companies who have migrated to IPTV has solved this problem.
I'd disagree.
I'm on IPTV and any major sporting event (World Series, Super Bowl, etc) is horrible buffering when I try to watch on my 4K IPTV (streaming) channel. I always have to downgrade to the HD channel and I still occasionally experience buffering.
So Netflix isn't alone in this matter.
Leading up to the fight, there were many staged interactions meant to rile up the audience and generate hype and drive subscription revenue, and subsequently make ad spots a premium ($$$).
Unfortunately, American television/entertainment is so fake. I can’t get even be bothered to engage or invest time into it anymore.
Hopefully Netflix can share more about what they learned, I love learning about this stuff.
I had joked I would probably cancel Netflix after the fight.. since I realized other platforms seemed to have more content both old and new.
Then the video started stuttering.
Netflix is clearly not designed nor prepared for scalable multi-region live-streaming, no matter the amount of 'senior' engineers they throw at the problem.
Well, yes. Who would think Netflix was designed for that? They do VOD. They're only trying to move into this now.
It's almost like this platform has been taken over by JavaScripters completely.
Woke up at 4am (EU here), to tune for the main event. Bought Netflix just for this. The women fight went good, no buffering, 4K.
As it approached the time for Paul vs Tyson, it started to first drop to 140p, and then constantly buffer. Restarted my chromecast a few times, tried from laptop, and finally caught a stream on my mobile phone via mobile network rather than my wifi.
The TV Netflix kept blaming my internet which kept coming back as “fast”.
Ended up watching the utterly disappointing, senior abuse, live stream on my mobile phone with 360p quality.
Gonna cancel Netflix and never pay for it it again, nor watch hyped up boxing matches.
I have Spectrum (600 Mbps) for ISP and Verizon for mobile.
Internet live streaming is harder than cable tv sattelite live streaming over "dumb" TV boxes cable. They should not have used internet for this honestly. A TV signal can go to millions live.
Update: Switched to the app on my phone and so far so good.
You're referring to Hooli's streaming of UFC fight that goes awry and Gavin Belson totally loses it, lol. Great scene and totally relevant to what's happening with Netflix rn.
These secondary streams might be serving a couple thousand users at best.
Initial estimates are in the hundreds of millions for Netflix. Kind of a couple of orders of magnitude difference there.
I guess in the year when Trump is being reelected this is hardly a surprise.
i thought tyson was in eldercare.
Our engineers are fucking morons. And this guy was the dumbest of the bunch. If you think Netflix hires top tier talent, you don't know Netflix.
Apparently he was smart enough to get away from the Fortune 500 company he worked at, reporting to yourself, and "got a pay raise too."
> Our engineers are fucking morons. And this guy was the dumbest of the bunch.
See above.
> If you think Netflix hires top tier talent, you don't know Netflix.
Maybe you don't know the talent within your own organization. Which is entirely understandable given your proclamation:
Our engineers are fucking morons.
IOW, either the guy left to get out from under an EM who says he is a "fucking moron" or he actually is a "fucking moron" and you failed as a manager to elevate his skills/performance to a satisfactory level.
I don't think I'd want to work for you.
Have you considered that maybe you're being overly harsh about your co-workers? Maybe take the fact that one of them was hired by a top paying employer as a sign that you should improve your own ability to judge skill?
I interviewed at Netflix a few years ago; with several of their engineers. One thing I cannot say is that they are morons.
their interview process is top notch too and while I was ultimately rejected, I used their format as the base template for how I started hiring at my company.
If your "dumbest engineer" got a job and a hefty raise going to Netflix, it means he was very capable engineer who was playing the part of moron at this Fortune 500 company because he was reporting to a manager who was calling him and the entire team morons and he didn't feel the need to go above and beyond for that manager.
Also, highly likely that it was the manager that was the moron and not everyone around him.
They obviously have some really good engineers, but many low-tier ones as well. No idea how long they stay there, though.
I'm watching the fight now and have experienced the buffering issues. Bit embarrassing for a company that fundamentally only does a single thing, which is this. Also, yeah, 900k TC and whatnot but irl you get this. Mediocre.
It's more likely that you are bad at managing, growing and motivating your team.
Even if it was true, to refer to your team in this way makes you look like you are not ready for management.
Your duty is to get the most out of the team, and that mindset won't help you.
There's no reason to doubt what you say, probably people identify with the mistreated one. Why?
How are you involved in the hiring process?
> Our engineers are fucking morons. And this guy was the dumbest of the bunch.
Very indicative of a toxic culture you seem to have been pulled in to and likely have contributed to by this point given your language and broad generalizations.
Describing a wide group of people you're also responsible for as "fucking morons" says more about you than them.
Why do you call your engineers morons? Is it a lack of intelligence, a lack of wisdom, a lack of experience, inability to meet deadlines, reading comprehension, or something else?
I wonder if Netflix is just hiring for different criteria (e.g. you want people who will make thoughtful decisions while they want people who have memorized all the leetcode problems).
I have questions..
I think this is a result of most software "engineering" having become a self-licking ice cream cone. Besides mere scaling, the techniques and infrastructure should be mostly squared away.
Yes, it's all complicated, but I don't think we should excuse ourselves when we objectively fail at what we do. I'm not saying that Netflix developers are bad people, but that it doesn't matter how hard of a job it is; it was their job and what they did was inadequate to say the least.
Jonathan Blow is right.