Friday, June 8, 2007

A Guide to Similarity %

We're seeing more and more avatars on movie pages -- and that's fun: the reviews have somehow changed for us, from something like "content" into something more human, more interesting. You put a photo on your reviews and they become as much about you as they do the movie. And people now seem a lot more interested in clicking on those photos to see what else the reviewer likes and has reviewed.

But for me, the key is that Similarity %. But there is no scale, it's just a relative value. Is 50% similar good? What does it mean? So here are some comments on Sim%.

How is it calculated? Netflix uses algorithms comparable to those employed in the "Cinematch" engine which recommends movies -- but turns it around. Now computers take all the movies you've connected with -- rented is the most weighted, but also rated or even just put in your queue -- to get a signal about your taste. Then we compare those movies to the same set from each reviewer and generate a number. But its not an absolute value. Sometimes there is little direct overlap of titles, but there is overlap in "similar" titles, or more importantly, an overlap of genres. You might not have seen (or rated) the same set of movies I did, but we are interested in the same kind of movies, and this would make us similar. We do this very quickly to get a general sense of similarity.

What is a "good" match? Like I said, it's all relative -- if you and I are 60% similar, I may not know precisely what that means, but it suggests we're more similar than someone that i'm 55% similar to. The wisdom around here is that if you are 70% similar to someone, that's pretty darn similar. 80% is dead on. My very best friends -- with whom i would see ANYTHING they liked most of the time -- i'm in the high 80s with. And I'm not 90% similar to anyone I know. (Although I sometimes find reviewers who share that much taste with me). Below 50% and i tend to check carefully if i agree with their Favorite movies...

With your Friends list, we add a few more passes through the algorithm, to get an even subtler taste similarity, where we push up the emphasis on how you and I rate movies, and how common that kind of rating for a movie is (if you and I love a movie that the whole world loves, that doesn't really make us all that similar, but if you and I love a movie that everyone hates, well then, that's worth noting. So we do.)

One note: With Friends, the Sim% is asymmetric--that is, I can be more similar to you than you are to me. This is because if you have seen 10 movies and I have seen 100, including all 10 of yours, due to some intricacies in the formula, it shows a (small) difference between us--you with 10 movies will be MORE similar to me than I am to you (since I've seen so many you haven't, because there is such disproportion between our viewing histories). The presumption is that if you've only seen 10 and I've seen 100, i may have a far wider interest range than you. If you watch (or rate) 90 more, and there is still good overlap in interest, that eliminates the difference pretty much, but there is a lot of uncertainty with your smaller dataset. (We actually don't like this asymmetry very much, and are exploring that part of the equation even as we speak.) I know I was disappointed to learn that my very best (most similar) friend--who was 89% similar to me--didn't hold me in a comparable position, and I was only 80% similar to him. That was a bit of a let down. (I'm rating more movies and the difference is shrinking.)

Like the recommendation engine at Netflix, we continually improve these mathematical formulas (see the Netflix Prize). The only (somewhat cryptic) thing i'd add is that we're only scratching the surface for how many cool things we can do once we have calculated this Sim%, and you will be seeing more use of the tool throughout the year. Here's my question of the week: besides being able to find and save other people who are very similar to you, and sorting reviews based on (among other things) how similar the reviewer is to you, what ways can you imagine applying the Sim%?

Do you find it useful? Interesting?


  1. Giving the rentals more weight is probably the only way to go, since this is a rental site, but it skews the % numbers in a major way. It doesn't count all the DVD movies we already consider great enough to own! I'm not renting say, Apocalypse Now when I already have LD/VHS/DVD versions stacked unwatched in my closet. This also applies to reviews. Go look at many classic "best of all time" movies here and many have low star reviews as only kids have written reviews for them. Of course a Spiderman fan will not understand Casablanca or Wings of Desire. I thought for a long while that members shouldn't be able to write reviews until they had actually rented the DVD, but that wouldn't work either. So, in the end I don't think there is any kind of reasonable answer that will satisfy all members.

    It would be nice if the search engine worked, tho...

    (Go ahead, type "Final". The movie doesn't appear until the middle of page 2.)

  2. My problem with the sim % is that for a number of people I get a blank (maybe 20-25%) - I get their icon and user code/name, but no % score. Using Firefox & XP.

  3. Is there some way to find the Netflix users who are MOST SIMILAR TO YOU? That might be cool.

  4. I enjoyed this explanation of how the percentages are calculated, and while I won't venture to make any serious suggestions as to how it might be improved (this type of algorithmic theory is not exactly my strong suit), I do have a question: What is the rationale behind favoring movies that have been rented vs. movies that have only been rated? Like eviltimes said, I own a ton of movies and will therefore never rent those. I also watch some movies in the theater before they're out on DVD. If anything, I would think my opinions of those movies should be considered more strongly, as I probably either like them enough to own them or I was excited enough to shell out $9.50 each to see them in the theater.

  5. I forgot to mention in my other post that I love what you're doing with the percentages, and I'm looking forward to anything new you guys choose to do with them. I especially enjoy seeing the percent similarity on the reviews, as it gives me a better idea of how much stock I should put in someone's opinion.

  6. Similar 2U is very useful. I was wondering if there was any plans to be able to add or request to be a member's friend with a high simialr2U%. Now that would be a cool feature.

  7. A question I've had about ratings for some time: Is "one star" more/less negative than "not interested"? I could infer that "stars" would presume you had watched the movie (or part of it) and "NI" you had not.

    (I've also thought that another choice ("so-so" or "OK") might be helpful. Many a "date movie" I've seen falls into that category...)

    Finally, I've rated many movies as "NI" so they wouldn't display and make it easy to find movies I want to see.

  8. I would like to see more emphasis on ratings than on rentals. I've rented some movies that I didn't even end up liking enough to finish watching. The flip side is that have rated quite a few movies that I love, but I won't rent them since I already own them.

    Please, please please get rid of the "2U" garbage!

  9. michael, from netflixJune 10, 2007 at 7:18 PM

    To be candid, rented is given slightly more weight, but not enough to be dramatic. The most important factors are to get a sense of the kinds of movies you like, how deep your experience in subgenres, and how unusual some of your genre or movie tastes might be. Everything goes in the mix, and I totally get your concerns about rented vs rated (and you guys articulate the issue quite well, including your thoughts about "owned" being a particular strong signal), but at the end of the day, with lots of tweaking the algorithm, the rented/rated difference isn't that strong, and i probably overstated the weight in my original post.

  10. Why are people allowed to rate movies that do not exist? "The Brazilian Job", for example, has not even begun filming yet it has 3 reviews (all akin to "I can't wait to see it!")and a 4.3 star rating based on 471 votes.

    Seems like that might throw off the accuracy a bit. Makes me wonder how many other ratings were skewed in advance by the extreme fanboy types.

  11. One really cool thing you could do with the sim% is to somehow allow it to be exported to blogs, myspace/facebook profiles, etc. I'm imagining a "how similar are we" link where an anonymous user can click a link on the blog/profile/whatever be taken to a netflix page where they are given a dozen or so movies to rate, and are then given a similarity number.

    An even cooler thing would be to set a "similarity cookie", and then provide some html/js code that would automatically allow you to see your similarity% of friends that have that code. this would only work for netflix subscribers of course.

    eg: Joe Blow has a blog where he posts some netflix supplied code into the sidebar. I'm a netflix subscriber and I go to his blog. I instantly see a netflix similarity %. If I'm not a netflix subscriber... well, see idea 1 above.

  12. I think that it's a brilliant feature. I'm a little curious about how the ratings similarity is weighted. If I've given 5 stars to a film, someone who's given it 3 stars is surely more similar to someone who's given it one star. I'd like to know how differently those figure. Talk to me like I'm a tenth grader because I'm not a math person.

    Thanks and keep up the great work.

  13. I just wish that the reviews that showed up on a movie's at-a-glance page were the 3 most helpful and the 3 from the people with the highest similar 2 U ratings for me. It seems that that would give me plenty to make a decision about the movie.

  14. I think being able to find other Netflix users that are "most similar to you" would be quite useful, even if you didn't have to sign them up as a friend, to see what new movies they've seen and liked/disliked.

    Along the ratings line, though, several friends who are also Netflix members, and myself, think it would be helpful to be able to rate with 1/2 stars. It was interesting that Netflix had guessed that I would rate a particular movie 3.5 stars, and that's exactly what I would have given it (impressive), but I could only give it 3 or 4. This creates a slightly incorrect appearance that you may have liked the movie "less" or "more" than the average, too, if the customer average was also 3.5 stars, for example.

  15. Suggestions:
    1. Give reviewers rankings ala Amazon so we can judge their credibility, experience, etc.
    2. Per reviewr, show the number of movies reviewed by simple # and those actually written about.

    3. Per reviewer, show film genres by total #s and by percentage.
    4. Also show an average users genre breakdown, for women, men.

    5. Show reviewers stats, M/F, age, etc, education, etc. It would be interesting to see how this correlates to review correlations.

    5. Have more meta links for movies. As in, if I like Saving Private Ryan, maybe I like War Films, Films with Actor X, the Cinematographer, Director, WWII films. For example, it would be nice if there were the following clickable options along side a S.P.R. page:
    Spielberg (Director)

    WWII (Genre) > Warfare type(Navy, Air Force, Army, espionage, etc.) > Country (Russian, French, German, etc.)

    WWII (Genre) > Country (Russian, French, German, etc.)

    Such that I could find all French WWII films, or i could first find espiongae films, then filter out for French only.

    For example, some potential searches:

    What are the top rated by reviewers like me, French > Comedies > 1960s.

    6. It would be nice to see correlations solely based on films viewed, independent of rating.
    For example, based on the least popular (based on # of reviews/rentals) films I've seen, show me which reviewers are most like me. That is, for the 10 least popular films I've seen, which reviewers have seen the most of those films.

    7. Allow me to sort movies by ranking of reviewers like me, and by # of reviews/rentals. This would allow me to find quality obscure movies.


  16. I am very centered around anime and animation. So anyone who is simalar to that, I'm interested in.

  17. I have a Netflix friend that I'm 99% similar to. We did not work to achieve this status, that's just what it has been for the past 4-5 months. It has varied slightly, the lowest being 97% similar. We have not seen all the same movies either. Netflix friends is a huge selling point for me. The ability to trawl through the queue of friends is gold, especially when we share the same tastes. I have 4 Netflix friends. Of course it only works if your friends participate. (I mostly use Firefox, and everything works except WatchNow, and I don't mind using IE for that)

  18. Netflix's similarity engine works pretty darn good. The key is to participate with others who also participate. If one of your friends is not showing any similarity, check his ratings, he probably has only rated a few movies and not much in queue. Queue up some movies, try to rate as many movies as you can find that you've seen, write some reviews, create lists and expect the same from your friends. I have rated 2731 films, written 23 reviews, have 6 lists, have over 350 in my queue and have 4 Netflix friends. And I have been a member for a sparse 6 months. Two of my friends barely participate and I usually keep their profiles collapsed. One of my friends is 99% similar in taste to me and he has over 500 movies in his queue, has rated about 2400 and has 2 reviews and 2 lists. Netflix friends, ratings/similarity is for those who love film and wish to discuss and compare their opinions. If you think having 10 in queue and 3 ratings is going to show a result, you really don't understand how it works.

  19. I don’t think I find similarity as useful or interesting now as I once did. I’m finding as I get more deeply into member reviews, that similarity is little help when it comes to predicting how much I will like or agree with their review. There have been reviews from people falling in the 20 and 30 that I’ve loved as dead-on, yet find others in the high 60’s and 70’s that don’t seem sympathetic to me at all. Then sometimes I follow a really good review of someone with a really high sim rating, one that’s well written and in sync with much of my own thoughts and find that a large part of their other reviews are not so compelling. Maybe accounting for the often huge disparity between the participation of members would help make the similarity rating more reliable.

    I have to agree with notaturnip… I hate pre-release reviews. They shouldn’t be allowed.

    And banxton has a wonderful idea. One that may also help solve the problem of good reviews getting buried under all the bad reviews that just made it in early. The average review-rater doesn’t view and rate reviews any deeper than the first page or two, but even so they click on the choices at the top increasing their prominence, where in fact there may be far better reviews more deserving of notice further down. I thought maybe a couple randomized reviews along side the ‘most helpful’ and ‘most recent’ might help mix things up a bit over time… but banstons idea would help with that just as nicely, and potentially increase the usefulness of the entire member review section in the process. Especially as the reliability of the Sim rating is improved.