- If I watched a movie and gave it a thumbs down, don't recommend it
- If I watched a movie < 1 month ago, don't recommend it
- If I browsed over a movie 50 times, read the info, and still didn't play it, stop recommending it.
- If I watched the last episode, remove the "new episodes" banner.
WTF
I just went to the frontpage of my Netflix and...:
- "My list" recommends 3 series where I have alread watched the last episode
- "Only on Netflix" recommends another two series, I have already wathed.
- A section displayed is "Watch together for older kids" - I dont have kids, never watched any kids stuff on my account.
- "Documentaries" contains six suggestions, 3 of which I have already watched on netflix".
- I am suggested several shows, which are good - but I have already watched outside Netflix earlier - and there is no way to tell Netflix (none that I know of, anyway)
9 out of 10 times when I go to Netflix, I intend to continue watching a series - but Netflix makes me scroll past SEVEN sections of recommendations to get to "Continue watching.." before showing me the series I have watched 1-2 episodes of most days for the past two weeks.
Maybe they are just too busy making sure all new series are woke-i-fied to care about how this simple stuff works?
I think the only on Netflix part works as part of positive reinforcement - it always shows me stuff it knows I watched all the way through (hence I must have liked it) mixed with things I haven't watched or haven't watched in a long time.
Thus when I see that section it is reminding me they have stuff I liked that I can only get there so please don't ever change account, and here is some more of that stuff only we got - try it out!
It seems some shows have been trending for years.
Or maybe it's not as simple as you think? "Baffled" is a strong word. It's the same "baffled" that Amazon can't find all the fake reviews. The same "baffled" that Facebook can't delete all your photos, everywhere, when you close an account.
Maybe things at scale are more challenging? These are some of the most valuable companies in the world. I'm sure they're happy to throw buckets of money at you if you can solve these "simple" problems for them.
They are clearly not using the ratings based automatic recommendations any more. It's not even relevant given the limited and generally low-quality content available. It's just about keeping enough people paying.
The Netflix prize also put Netflix on the map in terms of being a company that solves hard problems. We are still talking about it today and you'd better believe it has inspired talented people to work at Netflix; they could easily blow $1M on recruiting people and have much less to show for it. It's why Netflix is part of "FAANG".
Reed Hastings genius is that he led Netflix through a number of transitions between fundamentally different businesses: he built a strong brand with the DVD-based business without permission from the studios, transferred that brand to streaming when the studios saw it as "free money". By the time the studios understood what it was worth Netflix decided it was cheaper to buy than rent. (just as a consequence of having more customers)
The new frontier is that they use your engagement data not just to "suggest" the next movie but to design movies that will keep you engaged.
It's a little bit scary with these services that are "all you can eat" games for $10 a month because you're giving up "voting with your dollar" but creating a trail of engagement that will be fed back into satisfying your narcissism. Taking screenshots and videos of games seems fun and harmless at first but somebody knows I had a big crush on Nikola and Chiara from Valkyria Chronicles 4.
For those of us old enough to remember, it's not much different than going to Blockbuster. All of the new releases were along the walls with lots of copies to support the high demand. That's where everyone started when entering the store. If you found what you wanted, you grabbed a copy and left. In the middle of the store, the shelves were full of stuff you'd never heard of with one, maybe two, copies available. Both of those copies were covered in dust. You'd see people doing the physical version of endlessly scrolling to ultimately settle on "something" just to not be scrolling any more.
Really, the only difference now is at least you don't have drive somewhere to do the scrolling. I'd also say that there's at least the advantage of being able to do it in your PJs, but Blockbuster (any video rental place really) was the first public place that I noticed it became acceptable to not have to get dressed to visit.
With Netflix producing its own content now, and with the cost of acquiring content rights much higher than it used to be (all major streaming platforms want to offer great content), I'm wondering how much the business imperative impacts the recommendations we get -> eg. Netflix giving priority to its own content over licensed shows/movies.
Yes, Netflix’ engine is the reason I left Netflix…
That seems rather odd to me. I can believe it was a final straw after other reasons like running out of content you particularly want (absolutely or in comparison with other services), but not it begin "the" reason.
I don't particularly pay attention to the recommendations on either Netflix or Amazon, instead picking up things I might like to try from external sources (friends & family, discussions or records in various media, having liked something or some part of it looking into what else the performers/writers/directors/other have done it are involved in now, sometimes the does own external advertising).
I feel that the recommendation systems are more optimised for people who use TV/movies as background noise rather than actively watching. That would explain re-recommending long running series that they have already watched, amongst other things people have mentioned in this discussion.
Maybe my behaviour is a vestige from the life of piracy back when content was less readily available otherwise (somehow region locked, or simply not available on local channels yet, etc, so I often couldn't get things I cared about more legitimately for many months, if ever, and back in the scheduled TV days things were often in at inconvenient times). I seek out what I want rather than waiting for it to be handed to me by the service(s).
At least in my locale a good way to be reminded of this is searching for any movie you'd like to watch but isn't on the front page of netflix, typically they won't have it.
Presumably poorly rated movies are a lot cheaper to license; and when you're licensing 2/10 rated movies, user satisfaction is a lot higher if you don't show the ratings.
Shows with 10 seasons you hated and disliked after 5 minutes will still be in our "continue watching" or suggested.
This is a hugely important point. I'm completely uninterested in most netflix originals, but I understand why they're going to continue to recommend them to me.
I still find it a marvel that despite the majority of content introduced being not for me, I have been able to watch something unseen approximately every night and have given up on so little. In that sense, it is better than TV – even if the impression I get from the new content I scroll past is that it appears to be following all the same trends that made TV less appealing to me.
The most irritating consequence of this content problem for me is that nothing remains in the same place. Is continue watching going to be one, two or even three down button presses tonight? The reward is occasionally something gets suggested that is worth watching that would have been found anyway within a few minutes of searching.
I would imagine what they’re doing makes sense for the majority, but it would be wonderfully nice if there were some kind of alternate "advanced" mode for people who understand the lack of content where all this suggested and popular stuff went away and the search filter improved.
If I watched a movie and gave it a thumbs down, don't recommend it
I sometimes click thumbs down by accident, or upon rewatching change my opinion.
If I watched a movie < 1 month ago, don't recommend it
I like to rewatch movies, sometimes more than once in a month.
If I browsed over a movie 50 times, read the info, and still didn't play it, stop recommending it.
I have a terrible habit of browsing Netflix and watching trailers right before falling asleep, and I'm sure I've done this to the same titles over and over again.
If I watched the last episode, remove the "new episodes" banner.
I'm not in front of Netflix right now; what happens if new episodes are added while you're watching the series?
It's also trivial to put all of the things you watched recently into their own subcategory in case you want to watch them again, which is in fact something that Netflix already does. It's called "Watch It Again". There's no reason to pollute recommendations for that.
> I sometimes click thumbs down by accident
The recommendation engine should be obeying your explicit actions, not trying to subvert them. Accidentally clicking thumbs down is an outlier action that is trivial for you to rectify on your own as soon as it happens.
> or upon rewatching change my opinion.
Intentionally rewatching a movie that you expressly disliked is an outlier position.
> I like to rewatch movies, sometimes more than once in a month.
Netflix already has a personal queue+favorites list called "My List" that you can add things to. If it has been less than a month since you last watched something, the reason you're watching it again so soon is because it's on your mind already and you don't need the recommendation engine for that.
I really dislike it. I wanted it to be more like netflix so I don't get suck in.
Could be a location issue since I am not in the US.
Would you agree that these rules will quickly become unwieldy and a pain to maintain? And that these will be personalized to your taste but not to someone else's?
Wouldn't it be great if you didn't have to maintain those rules and if the system were tailored to every user? Congratulations, you've just realized you'd like to use ML/AI.
I don't actually want recommendations, I want a catalog with exclusions.
- Removing a movie based on "not payed X times" would remove all popular movies for all users in a multiple of X steps.
-I agree with the new episodes banner.
yeah some movies might be more likely to be rewatchable quicker by a larger amount of the population, and some people might be more likely to be able to rewatch movies they like more quickly than a month - and those people might have preferences that indicate their liking to rewatch more often.
Thus it might be nice to do AI on this.
Wait [ ] months before recommending again a show
I already watched.
The former group is just simple filters as you point out - which informs the ranking of the second group.
However, what they do might actually work for them, i.e.
(1) The HN is probably not representative of their audience as a whole (2) What they do now might be RoE accretive for them, but not so great even for a wider section of their customers
Could still be suboptimal and probably is...
How could it work with Netflix without their explicit support? For the movie database, I assume there's some data source somewhere that lists all the movies and series Netflix currently has in any given region. As for ingesting watching history, it could parse Content Interaction History exported from Netflix via their GDPR Subject Access Request flow. Sure, they have up to 30 days to process such request, but I'd happily accept a recommendation system I have to manually update every month, over the disaster Netflix has been offering to its users.
Netflix got more than 1 million dollar in free advertising from it, and are still getting brand value out of it today. They implemented some of the algorithms, and probably got a 10X ROI through retention alone.
As mentioned in the Quora answer, they were also able to recruit top talent - and that's much harder to put an ROI figure on.
He's extremely intelligent and passionate about this space, and every time we spoke I felt like I was learning something new. You can listen to him give an in depth talk about the Netflix problem and solution here [1].
Think of it like Google's famous obsession with speed. Did returning search results 17ms faster really matter? It's hard to say for sure, but I suspect it did.
That said, I agree personally. I don't like Netflix' UI. I suspect you could hand code a browsing/ranking UI of similar value, from a casual users' perspective.
>> large enough catalog
I think this is a case where Netflix didn't end up where they expected the. I think they expected to have a vast catalogue... a "spotify of movies." It just didn't go that way.
You could also reverse the question. Does netflix have a big enough dataset to make a great recommendation system? I think this might be the more pertinent question. Google & FB have their vast ad-centric datasets. I suspect these could be used to make a recommendation engine that's a lot better.
They haven't really done this for youtube though. The priority is to match ads to users. For this, they're willing to push the envelope on how they use user data. For youtube recommendations, it doesn't seem that youtube gets access to much data from outside of youtube.
Then comes this flashy $1 million prize. Tons of universities had teams. So it really helped their recruitment.
It also likely contributed to the idea of creating Kaggle which has itself greatly contributed to data-science education by giving everyone an open forum in which to compete.
Then there were other signficant projects around this time like ImageNet which became a competition too. That open dataset led to tons of research and applications.
> Netflix is an online DVD rental company that lets people choose movies to be sent to their homes, and makes recommendations based on the movies that customers have previously rented.
It was a different and exciting time back then! I never finished that book but hope to some day... :)
[0] https://www.oreilly.com/library/view/programming-collective-...
I would have expected that a blog post would discuss how this was structured. Netflix contracted with innocentive.com, which is a website for solvers, and contracting to that website expanded Netflix's reach to a greater available pool of solvers. As far as I recall, all the allowed solvers for the netflix challenge _had_ to go through innocentive. I'm not sure if they would have been able to get the same level of improvement if they had not contracted with a set of potential solver teams like that.
The original challenge listing for Netflix is no longer listed at innocentive.com, but an industrious person may be able to find it on archive.org or somewhere similar.
It's entirely possible that as a new solver at the time I fell for innocentive's PR. The netflix challenge was actually the first I ever signed-up to work.
I think the data is still kicking around somewhere in torrent land. I also still have my own copy somewhere, I think.
A Wired article from 2010 [1] suggests it lead to some legal liability risks for Netflix.
Most notably, it taught me that it was incredibly hard to make significant progress past the most simplest and naive approach. That approach was "Take average rating a user gives, take the average rating a movie gets, multiply". (Ratings normalized to be between 0 and 1).
Just using this method would give us 95% of the accuracy of our final method. I think I calculated, and compared to the prize winning result, our method got ~90% as accurate a result.
A few percent can make a difference, especially in competitive areas; but the biggest win is just getting something in where there was nothing before. It's a bit like optimizing code.
The benefits of such a competition are pretty nebulous, and there's no way to convince an ardent skeptic. OTOH, many business decisions are like this and skepticism isn't a viable frame in many cases.
Netflix got visibility with investors and potential employees. Netflix's recommendation engine became famous, even though it doesn't seem impressive as a user. The exercise created a structured way of thinking about their recommendation algorithm. They cemented its importance. Even though they didn't implement the winning solution, they did get a useful benchmark. This was potentially very useful in further decisions in R&Ding the recommendation engine in-house.
All that for $1m?
Kaggle has been around for a long time now. If it works, I would expect them to be pumping out tons of interesting results from winners but I don't think I've heard many stories like that. It seems to be mostly useful for recruiting purposes?
But I highly encourage you to read the winners' solutions. They are full of clever data insight, augmentations, regularizations, feature engineering, and preprocessing and postprocessing tricks.
But above all, compared to the academic literature, it's shocking how much time and creativity they spend on validation. Maybe I'm reading the wrong papers, but the flashy new neural architectures rarely even mention their validation setup; Kaggle winners sometimes devote half of their explanation to it. It's part of their secret sauce.
Two personal favorites:
(1) https://www.kaggle.com/c/severstal-steel-defect-detection/di.... The "random defect blackout" was a really clever data augmentation.
(2) https://www.kaggle.com/c/ieee-fraud-detection/discussion/111.... Particularly how they reduced overfitting with adverserial validation. They trained a separate model to distinguish between train and test sets, and then dropped features that ranked highly in feature importance on that model. That's probably a well-known technique in some circles, but I had never seen anything like it before.
> But above all, compared to the academic literature, it's shocking how much time and creativity they spend on validation. Maybe I'm reading the wrong papers, but the flashy new neural architectures rarely even mention their validation setup; Kaggle winners sometimes devote half of their explanation to it
I agree, but in the end it is a competition, and the solution that scores the most is not always the solution that is "the most interesting" (or practical, or best in real world cases)
Though the details you mention are interesting, and can definitely apply at real-life solutions.
The goal of the Netflix prize wasn't to come up with the best algorithm - it was to make the Netflix brand exciting and legitimate to engineers. At the time, Netflix wasn't super high-tech and I'm sure it was hard for them to get the top talent they needed. It seems silly in retrospect now, but I'm certain the reason this was approved was because they wanted the free advertising this would provide within graduate classes and academia in general.
As a serious question, why do people include Netflix in the acronym FAANG, which I see on HN all the time? Is there something special about Netflix? Netflix is around the #14 tech company, so it's strange to see Netflix in there instead of Microsoft. Or is the use of FAANG divorced from its literal meaning?
"Put money to work in the companies that represent the future," he said. "Put money to work in companies that are totally dominant in their markets, and put money to work in stocks that have serious momentum."
I will admit that it was interesting to see what algorithms were poised to be cutting edge in media recommendation. The result was rather disappointing to me.
Netflix STILL isn't that exciting from anything but a compensation standpoint. The problems at netflix are about programming, while the technical challenges are droll at best.
And they don’t ask because users don’t provide useful answers.
But users don’t provide useful answers, because rating things doesn’t do anyone any good.
I’m of the belief that if you can make ratings useful (catalogue all movies, including not on Netflix; give useful ways to view/update your lists; have direct relationships to recommendations), you would have dramatically better recommendations for dramatically less effort/complexity.
I don’t think you’ll ever get to “good” recommendations based on usage. The data is fundamentally garbage.
Of course, the other side is that Netflix isn’t interested in recommending things I like; their goal is to recommend things I’ll put up with. They just need 1 show worth watching and subscribing for every now and then, and N shows to keep me mildly amused to stop me from dropping it between good ones
It was my understanding that there was significant business value in improving the accuracy of personalized movie recommendations. Recall that this was at a time where the majority of the business was DVDs sent via mail. A poor choice of movie created significant risk to customer satisfaction and hence retention.
A few suggestions:
1. Review channels by genre
2. Trailer TV - let me leave a “comedy” trailer channel running that shows the trailer and movie rating and details at the bottom, let me easily skip to the next trailer (or let it play out)
So I think any truly personalized content channel would get exhausted quickly.
What I’d like to have is just a channel of curated or semi curated movie content that I can leave running or forward through to watch.
I recently stayed in a hotel with 6 channels of hbo. It’s kind of refreshing to have “hbo comedy” with random stuff like Beverly Hills cop and billy Madison on at 2pm in the afternoon.
Netflix doesn’t have enough content to do this, so they keep recommending the same crap originals to me over and over, knowing that I don’t watch them.
I can appreciate that you may not like said content, but they certainly aren’t lacking.
EDIT: I should add in terms of customer satisfaction, not revenue. I am sure forcing their originals down people's throats is great for their revenues.
why? what does he mean? netflix had a killer advantage with the old rating system and then dumped it why?
The contest started in 2006 and was awarded in 2009.
If he started two years later and there was not a trace of the Prize work at the company, that would be an indicator that the competition was not important. If he started and could still see knock-on effects from the competition, that's an indicator that it was important.
Plus, he didn't just start at Netflix. He "took over the small team that was working and maintaining the rating prediction algorithm that included the first year Progress Prize solution."
Yeah, that sounds like he has some authority on the matter.
Its very possible Netflix realized they needed to course correct the UX and as a result the winners algorithm was solving for a problem that no longer applied because it was using assumptions (rating system & no existence of different profiles) that were no longer relevant.
He did work on productionize the 2007 winner, after all.
> Ten years ago, when I was leading Algorithms at Netflix
So... I think he does know a thing or two about specifically this issue.
On a funny note, Jordan Peterson used to be a well known Quora answer writer prior to his current career as an internet celebrity. Source: https://www.quora.com/profile/Jordan-B-Peterson
The later winning entries were too compute-intensive to implement, or not enough of an advantage over the existing engine to justify more compute.
So I would say it was a real competition, not just PR, even though a particular solution wasn't used.
(Factoid: It was the the last team to start the migration to Cassandra.)