Thursday, November 8, 2007

Flagging Reviews

I just saw this comment on Hacking Netflix:
What I REALLY want to know (And don't think anyone at Netflix would answer me on) is what "Flagging a Review" actually does. When I flag reviews, I do it because they are either not a review, they posted a url in the review, etc. Am I wasting my time by flagging them, or does Netflix really go through and look at flagged reviews to find the junk?

Does anyone know, or is it a big company secret? Now that I've got that email address, I might use it, but I wonder if flagging does the same thing.
First of all, why don't you think anyone at Netflix would answer this? Have I made it appear this is a big company secret? I thought i had explained this in the past, but i'll take a new stab at it here:

The flags are methods for you, the Netflix members, to self-police the reviews on the site. They are automated. We spot check them, but we have tuned (and continue to refine) the automated process triggered by flagging a review.

The short answer is that if a review gets a quantity of flags from different members, we move the review off the movie display page. It is not removed from the site, and will remain among the reviewers other reviews on their Profile pages. The algorithm that does this removing is the thing we keep refining. It would be too simple to say it takes X flags to get something pulled. It depends on a few factors, but particularly the helpful/unhelpful votes. If a movie has a quantity of helpful votes, for instance, it takes more flags to remove it. (We don't want the flags being used to remove reviews that are simply controversial, or irksome to a small group.) We also want to give new reviews a little bit of time to get seen, so on popular titles that period might be less than for smaller titles... but either way, it is automated, and really is working. Over the past few months we've changed the thresholds and we continue to find cases where folks are misusing the system and thus we revise the algorithm to account for this. The thresholds and algos are also different for "this is not a review" and "objectionable content" (etc) and we know some people who don't like a review just click on everything hoping this will accelerate the removal (it doesn't). I presume we will continue with internal tweaks to improve it over time.

There. Secret is out of the bag.


  1. This comment has been removed by the author.

  2. I was going to say that you shouldn't assume everyone at Hacking Netflix also reads here, but I see that it was a regular from here, heh. I will say that I am well read up on your comments and as proven I forget or mis-remember them at times.

    Though when she said "And don't think anyone at Netflix would answer me on", I don't think she meant that you don't answer questions. I took it more as "I don't think netflix would tell us because they don't want the abusers to know exactly how it works."

    Anyway, thanks for the extra details on how it works. Cool stuff. Glad to hear that its working.

    I have flagged several non-reviews I saw scroll by on the community page. Just yesterday I saw one that that said "Don't send me any more movies like this one. I prefer movies that are..." I found that hilarious. As if they have no control over what movies Netflix sends them.

  3. Consider that the person who posted that to Hacking Netflix might not have any idea that this blog exists (one reason some of us whined about hidding the blog link at the bottom of the page). Since Netflix no longer has offers email support, there's nowhere to just drop a line and ask a question, so it makes sense that this person might assume no one will answer, and that they have no idea there's a blog here which already answered his question.

  4. @Baff. McW is a fella, the alluring photo notwithstanding.

    @Anon 1. He's a regular contributor to, and so one assumes reader of, this blog.

  5. Wow, I feel special now :) Not only do I now know that Netflix does watch flagged reviews, I also know that Netflix checks out Hacking Netflix occasionally. Ha! You've been caught!

    Thank you for answering my question Michael. Where I was coming from was that I have seen many people on Hacking Netflix say (In other blog posts) that the Netflix customer service can sometimes be less knowledgeable about technical things such as "Flagging Reviews".

    The reason I thought that no one was checking Flagged Reviews at Netflix is because if reviews are moderated in the first place, how do so many reviews that say "My DVD arrived scratched and unwatchable, twice in a row!", get through the pipeline? The fact that I see so many of those reviews made me think that flagging and moderation wasn't effective. See where I was coming from?

    I agree with anonymous guy, now that the email support is gone, I can only imagine trying to explain what a Flagged review is to someone on the phone at Netflix customer service (I have not had to call them for anything yet, so I may have unfair preconceived ideas about what would happen if I did call and ask a technical question).

    I'm glad I at least got the answer to my burning question now. Oh, and to whomever mentioned it, yes I am a Male, and yes I use the same username everywhere because I am lazy.

    Now, I've gotta go flag some more reviews :P

  6. re comment from mcw re NF customer service (I have not had to call them for anything yet, so I may have unfair preconceived ideas about what would happen if I did call and ask a technical question).

    I've called about 3 times and encourage you to. Each time I've spoken to reps who seem very knowledgable, have excellent communication skills and really want to help. From experiences I've had calling some other big company help desks I was dreading my first call to NF but service was great and the wait was no more than a few minutes. It shows that NF (unlike many big companies) cares about their customers. My calls were not about technical questions but based on the experiences I had, I'll bet if the rep didn't know the answer they would try to find out.

    All the reps sounded happy with their job, too!

  7. You are certainly correct, Michael, in that you have more than answered this question already. I pop in here once every two days and knew pretty much how the flags work.

  8. I have a few more questions about flagging…

    Mainly I’m wondering about two scenarios…

    First I’m sure I’m voting on reviews more than once, and probably flagging more than once as well sometimes. I know they only count once, the extra flagging would be unintentional because I may run across a single review more than once without remembering if I’ve flagged it before. (This is easy to do now that we run across reviews both on each dvd’s information page plus under members reviews pages too. Plus, who necessarily remembers which reviews were voted on/flagged months or years ago?)

    Second, is the scenario where there are differing opinions about what constitutes a spoiler, on a popular film you could end up flagging hundreds of reviews as spoilers.

    Would doing so raise an alarm with Netflix, perhaps flagging me as a flag abuser?

    In an earlier comment my example was the movie “Zodiac” where a substantial number of reviewers assume that everyone who will watch the film knows the outcome of the real life case, therefore they believe that revealing the outcome of the movie is ok. I however would say that even older people don’t commonly know much about the case, including the outcome, and let alone the details. Not to mention younger audience born well after the fact who have no clue what-so-ever. Again, if I am strict with spoiler flagging in this case that could be a fairly large number of reviews flagged in one go.

    Would there be any problem flagging reviews in these two cases?

    Would we be taking the risk of marking ourselves as flag abusers?

    What really constitutes flagging abuse?

    And are there repercussions for abusing flags?

    Becky Glass

  9. Becky,

    1) Don't worry about accidentally (or purposely) flagging multiple times. Only one flag is read, the remainder are ignored, so it does no harm.

    2) Spoilers. Since it takes a quantity of flags from different members to get a flag added for spoiler, I believe that the case of a range of opinions about what constitutes a spoiler is okay -- i mean by this -- i think it's fine to err on the side of caution. If a number of people feel this particular review spoils something about the movie experience, i'd rather just skip it, even if some people (or many) don't think it's a problem. You have to assume the reader is sensitive and there will be plenty of reviews that don't get too close to the line. Yes, some fine reviews will be flagged, but they aren't removed, just labeled with a warning. I don't see this being abused on the site, and i've felt good about what's getting flagged. If it becomes a problem we'll revisit it, certainly.

    We have some flag abusers, but it really takes an effort to cause change due to flag abuse -- that is, a group of folks ganging up on a review. I hope our algorithms work pretty well to address this. No, there is no punishment (at the moment) when we do discover abuse. I know in one case our Customer Service team contacted a member who was abusing the system, but we also changed the algo to keep it from happening in the future...

    Like i said, if something becomes a problem we'll deal with it, but until then i'm seeing members using these well and properly in aggregate.

  10. @Becky - This won't work with flagging, but there is a way to find out if you've already voted a review as helpful/not helpful: vote on the review and then refresh the page. If the number of votes doesn't increment then you know you've voted before. If the number does increment, it's reasonably safe to assume that your vote was the one that counted (that vote could have been someone else's, but the likelihood of that happening is reasonably small).

    I don't know if that's really helpful or not since Michael just told us that excessive voting is ignored (and without penalty), but it is at least a good way to satisfy your curiosity :)

  11. Thanks for the clarifications. :)

  12. eric said "there is a way to find out if you've already voted a review as helpful/not helpful: vote on the review and then refresh the page. If the number of votes doesn't increment then you know you've voted before. If the number does increment, it's reasonably safe to assume that your vote was the one that counted"

    Does that work for you? I just tried it on a movie I know I never voted on and refreshed and it didn't change the number. There is a notation on the page:

    "Votes are counted and displayed within 24 hours"

    So either it's because I use Mozilla or it doesn't change for 24 hours.

    I wish we could change our vote after we click on it and before we leave the page. A couple of times I've accidently hit the wrong one but it wouldn't let me change it.

  13. Unless the system has changed recently, the votes only get tallied once a day, around midnight (Netflix time).

  14. So - now that we've got all that straight, and everyone got their potshots in - what about the poll over there Michael from Netflix?

    Are you guys working on a way to feed Netflix out to iGoogle, or will you make a post about that if you plan to? I think it'd be pretty cool if things were improved with the feeding situation, as I've found the RSS feeds to be unhelpful, because they often mix up the order of what they're feeding out.

    I don't know what you could do instead of RSS feeds, but I've been delighted with everything you've come up with up to this point. I'm sure you've got something really cool headed our way.

  15. I wish we could change our vote after we click on it and before we leave the page. A couple of times I've accidently hit the wrong one but it wouldn't let me change it.

    I have done this on a number of times as well. It would be so helpful to be able to change or edit our choice in case we make a mistake. I feel awful marking someone's review unhelpful when I really meant to mark helpful. It's a stupid mistake but it happens.

    I have a totally unrelated request. When our friends & faves reviews show up featured on the community homepage, there is no title listed. If I don't recognize the movie by cover image I have to do the mouseover thing in order to figure out what movie is being reviewed. If you guys could please add the title along with the review and box cover that would be so much better so that we can immediatly know what movie is being reviewed.

  16. I've even voted by mistake meaning to go to the next page! So yes, some kind 'void' system would be great, especially since you can't now rectify the situation with a counter vote the next day, as I've done a couple of times in the past.

  17. Off topic, but 'Ask Mike' seems to be defunct. Someone was hunting for 'movie privacy' the other day, without success. Wouldn't it perhaps make more sense to have 'Invite Friends' + 'Friends Home' + 'Movie Privacy' all show up on the lower community toolbar without having to open 'invite friends' first? Or maybe the space is being reserved for future goodies?

  18. "We don't want the flags being used to remove reviews that are simply controversial, or irksome to a small group"

    A while ago I came across a review of Schindler's List that said the movie was hilarious because "it never happened". I flagged it as offensive and it was later removed, but I guess it could just be considered "simply controversial" or "irksome"? So where is the line drawn? I'm just curious as whether to such decisions are left to the reviewers+algorithms or if a staffperson actually takes a look and makes some sort of executive decision.

  19. This comment has been removed by the author.

  20. What makes me thing "Flagging" still isn't working is that the unaired "Lost" Season 4 is rated 5 stars with 9 reviews right now. Why is Netflix allowing people to write reviews for all of these T.V. shows that aren't on yet? The season starts in 2008, so why let people abuse the rating and reviewing system? How are we to trust a 5 star review when Lost Season 4 already has it? Everyone, go flag those 9 reviews as not helpful and not reviews - let's see if they're doing their jobs.

  21. @Knaldskalle. My understanding is that it's all based on pre-determined algorithms, but that these are being monitored and fine-tuned for the time being to make sure that 'demotions' and 'removals' are both fair and rational.

    Speaking of which, pretty much anything to do with religious, political or social belief systems might arguably be considered 'controversial', but denying well documented historical facts like the Holocaust is just plain moronic. So, based on your and others' flagging of that review, it looks like the system is working pretty well.

  22. Well... after reading this rather snarky thread, I found it essential to respond. First of all, McWhammer, you will be happy to know that ALL reviews of Lost: Season 4 have been removed. Personally, I find this encouraging and disheartening at the same time. I am a HUGE fan of The Movies and enjoy the discussion and sharing of ideas that the cinema encourages. However, I have been unable to post my reviews for more than THREE MONTHS! This irritates me on a number of levels.

    Initially, I contacted customer service (which was extremely personable and willing to help) to no avail. "Wait a-couple-a weeks 'cause I see your account has been flagged for this issue and it is being 'worked' on" was the standard response. Four phone calls later (and as many posts to this blog addressing this issue) it became apparent to me that my little problem was "not important" in the eyes of the NF Gods.

    I then settled into a comfortable void of folks-that-love-film-and-refused-to-be-silenced and kept posting my reviews, knowing they would never be published on the NF site. Then... I was assaulted by the barrage dumb (followed by a never ending parade retarded) "reviews" on the community page. Useful gems, like "This disc was scratched, send another, please" and "This movie suks cuz they dont speak american or nothin" or the ever faithful "Why can't they make movies about murdering your family without so many swear words", disgusted me to no end. I watched as my "influence rank" plummeted and still continued to post reviews.

    Then came the The Golden Compass. Gigantic numbers of right wing-fascist-religious-conservative-zealot NF members were able utilize the review system to call for a boycott of the film. This is when I started to get angry.

    Now... the attention of this blog to the unfavorable comments about NF on another site by ONE poster has gone beyond the pale. Not only has my issue been completely ignored, but the ego of NF employees has (apparently) become more important than the concerns of a paying customer.

    I truly love NF and appreciate the incredible changes that facilitate my ability to access films that, otherwise, may have eluded me. It just seems self-congratulatory to scold someone for posting negative feedback elsewhere on the web when right here at customers are being marginalized every day.

    ugh........... frustrated.....

  23. That's totally insane! No reasonable explanation? A far less patient and restrained NF blogger was saying the same thing (on every thread!) a while back, but we just assumed that he'd been blacklisted for being totally obnoxious. Good luck.

  24. We are considering using another online rental, due to the fact that every other movie we receive is scratched or unwatchable. Someone needs to figure out this problem and weed through the bad DVD's because it seems that I am not the only customer with this problem.

  25. You may want to consider getting a new dvd player. I have about 10 players. Some of them don't like about half the dvds I stick in them, others will play just about anything no matter how scratched they are.

    Its not just how expensive the player is either. I have a $20 player that has played everything but a couple dvds with actual cracks in them.

    The dvd players in my computers tend to be the most finicky.

  26. And half the problem is fingermarks and crud from previous members. Unless they're cracked I find it rare that skipping or 'unplayable' DVDs won't function perfectly after a gentle wash with warm water and a little dishwashing liquid, and some pat-drying it with a soft towel (no rubbing and certainly no circular wiping, as per vinyl LP's). Just hold them using the center hole and the edges, Folks!

  27. @Baff - Do you notice any trends on which players work better than others? Have you found that one brand works better than another? I had an old 2nd-gen Pioneer that would play just about anything, but I finally replaced it due to lip-syncing issues (remember those days?). I now have a Sony 5-disc Progressive Scan model that probably chokes (to some degree) on 30-50% of the Netflix discs that I give it.

  28. @Michael - this discussion raises an interesting set of questions for you as well: What happens when we flag a disc as damaged/unplayable? Is action taken on it immediately, or does it take several complaints? What action is taken once a disc is flagged? Does someone look at it or test it? Is any attempt made to clean or repair the disc? Should we flag any disc that gives us problems, or is that designation really only intended for discs that are completely unplayable?

  29. Eric, when you say 'flag a disc', are you referring to the 'report disc problem' links? If so, I just sent NF a suggestion that they add a description box for situations like the one I had the other day, when I got an 'unplayable' notice immediately, but after washing the disc the movie itself played fine but the previews kept skipping. I wasn't sure whether or not to report it, and risk having it withdrawn from circulation unnecessarily. A single additional sentence under 'damaged disc' would make that their decision, not mine.

  30. michael, from netflixNovember 13, 2007 at 10:43 AM

    1) Let's not confuse the "flagging" of a review with reporting a problem disc - these are different worlds and handled in different ways. if you report a bad disc using the movie reviews system, no one in customer service will see it and your fellow netflix members will just remove the review (since it's not a review).

    2) If a movie has hundreds of "helpful" votes, it is going to be difficult to gather enough "flags" to get the review removed from the movie page. This should keep reviews on polarizing movies from getting moved away. On the other hand, moronic reviews - one would hope - would not end up with many helpful votes, and therefore the flags would more easily get them demoted. Just know we continue to tune as we spot problems.

    3) jaymikey - i cannot imagine why your reviews aren't showing up. The Customer Service folks are usually remarkably good, and they should be able to find all your reviews and see why they aren't being posted. We get thousands of new reviews every day, so the system seems to be working - and it sounds like your problem isn't one of getting flagged and removed so much as them never appearing, right? This is clearly as CS issue - ask them to get your info and if they can't solve it, forward it to the community team to explore. It could be a bug, or as simple as something in your reviews that keeps them from being posted. Sorry to hear about this.

  31. @Michael - I guess I should have chosen my words more carefully. By "flagging a disc as damaged/unplayable," I was referring to the process of reporting a disc problem to Netflix, not to the process of flagging a (non)review. Sorry for the confusion.

  32. @eric - Most of my players are different brands, so I haven't noticed any trends.

    I few weeks back I mistakenly flagged a disc as damaged. It turned out the problem was with my player. There is no way to unflag, so I called up Netflix. The person I talked to didn't have any way to unflag it either. She said it was no big deal though, they would attempt to play it and if it played fine they would put it back into circulation.

    I do think they should have an added field so we can explain how it doesn't play right. I've had some movies that play fine until the last few minutes. I usually throw a sticky note in to let them know what problem I encountered. Maybe thats better than entering it online anyway.

  33. "I usually throw a sticky note in to let them know what problem I encountered."

    Excellent compromise, as long as the system isn't totally automated. Do you happen to know if humans ever get to read your notes?

    And 10 DVD players!!. Isn't that annoying, trying machine after machine till one works?? My 5 year old Panasonic workhorse would play a slice of toast if I let it!

  34. I only put notes in with discs I have flagged as 'bad'. When they check the disc, they should see the note.

    My current main player has only had a problem with 2 discs in the last 6 months, and both of those played in my back-up player (playstation 3 that I usually only use for blu-ray movies).

  35. "I only put notes in with discs I have flagged as 'bad'. When they check the disc, they should see the note."

    Makes perfect sense. I'll be doing the same from now on, unless they ever add a 'describe the nature of the problem' box.

  36. Every so often I come across 1 star 'reviews' along the lines of "Fell asleep 2 minits into the movie, next thing i knew the credits were roling." The problem is, if you're really tired you can fall asleep during the best of movies, so is this really a valid review? Same applies to "we ejected this drek after 30 seconds and sent it right back". Give it l star, fair enough, but leave the reviewing to those that have actually taken the trouble to watch the darn thing, I say. Do you guys flag these as 'not a review', tag them 'unhelpful', or maybe even 'helpful' if you agree?

  37. I tend to mark them as unhelpful, though I can certainly see an argument for marking them as not a review.

  38. Richie, if it is not, by definition, a review, then it should be flagged as such. One cannot review a movie that one has not seen.
    A complaint is not a review.
    A moral or religeous judgement of a movie,or of those associated with it, is oten not a review.
    Political grumblings are not reviews.
    Hatred spewing is not reviewing.

    These are my own opinions, and I use them to flag reviews till the cows come home.
    I encourage everyone to have opinions, and to use them. That's what these tools are designed for, and what makes this site so much fun.

    gonna go eat some pumpkin pie now.

  39. @B & E. Thanks for the feedback, in case anyone checks back.

    And re the obvious non-reviews, like the latest one for 'Bagger Vance' ("This movie was badly damaged and stopped many times..."), I repeat an earlier suggestion that a Report Disc Problem box be added right next to Write A Review (maybe with an 'or' in between them?), since these people obviously aren't seeing the clear-as-day links at the top of their queue, hard as that is to understand.

    That would surely save all of us, especially the unfortunate - if aberrant - renters, a whole lot of grief.

  40. Netflix had a great part of their site called releasing this week...every sunday I would go onto the site and choose out of this page new movies to add to my queue....they got rid of this page this week...I called and was told that the subscribers didn't like the page...I'm a subscriber and was never asked...after prodding for a while...I was told to go to other sites for movies releasing, go back to netflix, enter it's name and then add it to my queue...I'm disappointed with this as it's an obvious way for them to save money without regards for the consumer and i'm even more disappointed that they are teeling folks it is a decision based on customer's clearly a lie...I would have said it was the most important page on the site as would many others...they should be embarassed...