Last week Slate ran a story about the "Hollywood Career-o-Matic," which claimed to use data from Rotten Tomatoes to chart the trajectories of Hollywood careers. Interactive feature: Just enter the name of an actor or director and it will instantly generate a graph showing that person's critical ups and downs.
For example, here's one for M. Night Shyamalan, with each dot representing the Tomatometer score for the features he has directed:
Slate concludes that, according to Rotten Tomatoes data, the Best Actor in movies is Daniel Auteuil, with John Ratzenberger the best American actor, since he's voiced a character in every Pixar movie. Best Actress: Arsinée Khanjian. Worst Actress: Jennifer Love Hewitt. Best Director: Mike Leigh. Worst Director: Dennis Dugan (veteran of Adam Sandler movies such as "Happy Gilmore," "I Now Pronounce You Chuck and Larry," "You Don't Mess with the Zohan" and "Grown Ups").
Yes, this is all so silly that the mind boggles, but let's start with the premise itself: What is the correlation between reviews and careers in Hollywood? Adam Sandler and Michael Bay wouldn't look much more impressive than Shyamalan if you looked only at reviews. And the Slate piece is riddled with misconceptions about the Tomatometer:
What makes the data from Rotten Tomatoes so brutal is that they depict not just one person's opinion of Shyamalan but the collective assessment of all our cultural critics. [...]
The site even aggregates archived reviews from films that were released in the 1990s and before. Whether you're looking at contemporary cinema or the classics, the Tomatometer can serve as shorthand for a film's critical reception, if not its box office success.
"All our cultural critics"? "Contemporary cinema or the classics"? Well, at least it gets this part right:
The Rotten Tomatoes website, created in 1999, aggregates reviews from hundreds of newspapers and websites across the country, converts each review into a thumbs-up ("fresh") or thumbs-down ("rotten") rating, and then combines those assessments into a single "Tomatometer" rating that gives the percentage of positive reviews.
And that's the source of another inequity. Who determines what constitutes a positive or negative review? That depends. If a critic uses some kind of ratings system (stars, a numerical system, letter grades) then anything above 50 percent would be considered "fresh." (And the average of all ratings must be 60 percent for the film to keep a "fresh" rating.) Which means that, on a four-star scale like the one Roger Ebert uses (much to his chagrin), 0 to 2 stars is "rotten" and 2.5 to 4 stars is "fresh."
At Metacritic, another review-aggregating site, at least an attempt is made to assign each review a numerical score between 0 and 100. So, on that same four-star scale, 2 stars would be 50 and 3 stars would be 75. The "Metascore" is simply the average of all those numbers. (Also, RT includes many unknown bloggers and online outlets, while MC "distills the opinions of the most respected critics writing online and in print...".)
But both Rotten Tomatoes and Metacritic have to assign values (fresh or rotten at RT; 0-100 at MC) to reviews that aren't accompanied by measurable ratings, and that's where things get even slipperier. Let's take "Super 8," for example. As I write this it has an 83 percent rating from 193 reviews (160 fresh and 33 rotten) on RT (and a 72 on MC). Selecting the All Critics tab (sorted by Most Recent, which is the default), the first three reviews are from BrandonFibbs.com, Mountain Xpress and Q Network Film Desk. Selecting the Top Critics tab gives you Tom Long of the Detroit News at top, followed by Lisa Kennedy of the Denver Post and James Berardinelli of ReelViews. (Dana Stevens of Slate is next, which reminds me: Why isn't Stephanie Zacharek, formerly of Salon, now of Movieline, also considered a "Top Critic"?)
Peter Howell of the Toronto Star gave "Super 8" 2.5 stars out of 4 and is quoted on the RT site: "Abrams fashions 'Super 8' in such a calculating manner, with every element weighed both for maximum nostalgia value and ironic hipster cred, that it has an artificial feel to it. It's like a birthday cake made of spun glass." This appears as a "fresh" review, though it may not sound like one. So does Rick Groen's 2.5-star Toronto Globe and Mail, represented by the quote: "It's the child actors, heroic indeed, who rescue 'Super 8' from the blockbuster grip of its adult makers."
The New York Times doesn't use a scale, but A.O. Scott's decidedly mixed review is considered "fresh": "Attempts the difficult feat of balancing self-consciousness about the olden days with wide-eyed, headlong, present-tense fun. For about an hour it succeeds marvelously. The modest letdown that follows exposes the limitations of Mr. Abrams's imagination."
You get the idea. A review that trashes the film gets the same rotten rating as one that finds it a near-miss. A rave is just as "fresh" as a lukewarm approval.
I find the Rotten Tomatoes site (and its predecessor, Movie Review Query Engine, or MRQE) useful in helping me to find reviews and information about individual titles in one spot. I can glance at a page and see who has reviewed a movie and then click over to read the entire review. (Google still can't sort search results for movie reviews that finely!) But none of this really has anything to do with monitoring or comparing Hollywood career trajectories. For that we still have palm readers and TMZ. The Tomatometer is a reliable way of measuring Tomatometer ratings. That's about it.
30 Comments
What I have found most off-putting is the suggestion that being in poorly reviewed movies is synonymous with being a bad actor/actress. The quality of a movie, beyond an actor's own performance, is largely out of his hands. This is even addressed in the article. "But all in all, they don't have nearly as much control over film quality as directors do." Yet, the article still declares Jennifer Love-Hewitt and Chuck Norris worst actress and actor respectively. While they might be good candidates for such distinctions, the Tomatometer is not a fair source of evidence.
I laughed out loud when I read this article last week. If they had published it on April Fools' Day, the methodology behind it would be more defensible. As it is, it's just sad that this was taken seriously by anyone.
I like the consensus bars, giving a basic idea of what critics have said about a certain film, along with the blurbs that also serve the same purpose. To be fair, every film Shaymalan has directed since "The Sixth Sense" with the exception of "Signs," which has a higher rating than "Unbreakable" has a decreasing tomatometer rating. Of course, this is still a superficial way to look at his films, particularly since "Unbrekable" is arguably even better than "The Sixth Sense" and I admittedly like elements of "The Village."
Mathematically I have a problem with upscaling a rating system to the 100% system. If the original system was say a star rating, Metacritic would assign zero stars as 0%, one star 25%, so one and so forth.
Now I would accept such conversions if perhaps Ebert gave a movie 3.252341231 stars, which would be 81.3%, if that is what Metacritic does. Well we don't have that, because no respectable film critic would fret out a rating system like that. Ebert can give half star ratings, which is the smallest fraction of a star he can give to any film.
So with that in mind, 1/2 of a star would be equivalent to 12.5% on the Metacritic website, well are we to believe here that if Ebert who gave Fantastic Mr. Fox 3.5 stars would have given the film a 87.5%? Why not a 89% or a 90%. Since any of those alternative scores would be equally valid in downscaled conversion to a 4 star rating system. Metacritic would have us believe that if Ebert gave a film 4 stars, it would be equivalent to 100%, nonsense I think I've seen some of his great movie reviews where he acknowledges some flaws in the movies he loves. Metacritic's system is dubious because if we follow their logic in converting various rating systems, in a mathematical sense their system is off by two significant digits.
When it comes to Ebert's rating system, Metacritic would or should say 80 + or - 12 %
I wouldn't bet on a dart player with accuracy that bad, so why should the movie going public have faith in Metacritic?
The Rotten-tomato system is somewhat better than Meta, in that they downscale the review systems, in a sense it would give as an idea as to the consensus of the critics, but even then they made a fatal mistake in calculating the mixed reviews. For instance how does Rotten Tomatoes rate a split between Siskel and Ebert At the Movies? I think they should at least put the mixed reviews in a third category.
There is also a rare problem when a critic reviews a film, but refuses to rate the film, or even say in the review that he liked or disliked the film. Those don't happen very often, but I do have to ask what exactly do the two sites plan to do with reviews of that nature?
To be fair, very few critics would actually say that more than a handful of the movies to which they've ever given maximum scores are "perfect." Every critic would say that the maximum score simply represents the maximum recommendation.
The way that RT handled Ebert's TV shows has simply been to give each of the critics his/her own "Rotten" or "Fresh" rating. Thus, there was never any problem when one gave thumb-up but the other gave thumbs-down.
I think that the philosophy behind the Fresh/Rotten distinction is about the degree of recommendation. If you personally felt an even 50-50 mix of good and bad about a movie, then you presumably wouldn't recommend it to anyone with tastes similar to yours (because the bad completely cancels the good, meaning that it's ultimately not worth your time), hence the "Rotten" label.
I'm not sure where you get this idea that reviews without ratings are a "rare problem." There are many critics who don't assign ratings as a matter of principle or because of the editorial policy of their employers. Such critics also typically don't explicitly indicate overall like/dislike. In these cases, RT and MC staff read the reviews and use a best guess. Do you not read many reviews at all?
First off: I think that with Ebert in particular, his 2.5 star reviews are always converted to "rotten" -- I checked for Green Lantern and for The Untouchables, and I am pretty sure I recall that this is done generally (to coincide with Ebert's "thumbs down" for 2.5 star reviews).
I agree with you that the Tomatometer rating doesn't tell you what the film's "true" response from critics is, etc., etc., and that it's silly as a measure. And of course the critical consensus on a film has little to do with how successful it will be. I think the tone of the Slate piece is very silly.
And yet, on a statistical average, Tomatometer ratings are going to be positively correlated with critical responses to movies. Is it really not true that M. Night Shyamalan's later films were much more poorly reviewed on the whole than his earlier ones? I think that the Tomatometer -- well, actually, I prefer Metacritic, but it's the same principle -- is a relatively useful guide for what films tend to have better reviews than others. It's silly to use that as a primary measure -- but then, most reviews tend to at least give something away about the film's plot, even the "non-spoilery" ones, and if I want to go in to see something cold and hope that it will be good, it makes more sense to see something with higher critical aggregate ratings. It's positively correlated with being well reviewed; and being well reviewed is positively correlated with being good. I can't tell if it's actually good without seeing the film, and I can't tell if it's actually well reviewed without reading the (again, potentially spoilery) reviews. It's a tool that's nearly useless except as a vague indicator -- but hey, vague indicators strike me as being pretty useful sometimes!
I think you're right about Ebert's reviews. He considers 3 stars and up a recommendation. As I say, I find these sites useful, but I'm more interested in who said what, and I don't pay any attention to them until after I've seen the movie myself (unless, like some of those Sandler titles, I have no intention of seeing them). One insightful review from a critic I've learned to respect is worth more than a hundred poorly supported opinions from less learned dilettantes.
I went through a heavy rottentomatoes phase from age 19 until about 22. I gave it way more credence then I should have, becoming upset at unfair tomatometer ratings for films I adored. I like to believe my capacity for critical thought has deepened since then. Now I do what you do jim and mainly go to rottentomatoes to read notable critics reviews or any blurb I find amusing, thought provoking, or contentious. Assigning a normalized rating for movies smacks of a grade school mentality where you can simply designate one movie as being "better" than other based on an arbitrary metric. Now I visit rottentomatoes to contrast my opinions and critiques of a film with that of other cinephiles. Most of the time I seek out the reviews that disagree with my opinion sharply. For instance, I loved tree of life but this review, http://tonymacklin.net/content.php?cID=382 by tony macklin made me laugh out loud.
The system, of course, is flawed, because the percentage rating is not an average of the star ratings (or imagined star ratings) of each individual.
A film can have a rating of 100% with no-one considering it above 60%.
I have nothing against assigning an aggregated rating from critics' reviews. They are what they are. I have nothing against rating a film either. We all do it in one way or another, all compare films in terms of what they say and how they affect us. I don't see it as an immature mentality but honesty.
Critics are more useful after you have seen a film, to offer a new spin on what you've seen. The star ratings don't influence me at all before I see a film. I go by trailers, my knowledge of their previous work and, at a push, whether a certain critic's views have chimed with mine in the past.
Ironically, box office success is probably a better measure when looking at 'career trajectory' (whatever that is).
Either you're still working or you're not. And a lot of the time, I'd imagine that comes down to how successful your previous films were.
I doubt very much that Generic Executive X at Paramount cares much for critical reception. (S)he wants asses in the seats.
Some studios will actually care about supporting noted auteurs rather than box office numbers. It's very rare, but it happens. That's what Warner Bros. said about Kubrick (in a press release, no less) around the time of Eyes Wide Shut.
Remember that the studios are still at least somewhat concerned about having "prestige." That's why they continue to spend lots of money on awards-bait and award campaigns, even though such endeavors are usually probably less profitable than producing blockbuster event movies all-year-round.
I think the tomatometer's pretty useful, personally. It's like a broad survey of people who at least care about movies, even if they don't all have particularly good taste. If you're not sure whether you want to spend time seeing a movie, a good or bad tomatometer score can give you a sense of how likely it is you'll enjoy it--"85% of people who give a damn about films liked it, so I probably will. Might as well give it a shot!" Or even "Everyone thinks this movie is great; I don't think I will, but I know people are going to be talking about it and I'd like to be prepared for that conversation."
Jim,
Rotten Tomatoes does actually assign a numerical score. It's on the movie's tomatometer and is weighted out of 5.0
I find the numerical score a much better representation of quality than the percentage.
In other words I believe that a film which is 80% positive but a 4.0/5.0 is better than one that is 95% positive but has a rating of 3.5/5.0
Too bad that most users of the site, yourself included, pay too much attention to the percentage and forget about the weighted score.
Now that you mention it, I see there in the "fine print" underneath the 83 percent for "Super 8": "Average Rating: 7.5/10." But, again, in a case like A.O. Scott's review, how do we know what rating RT has assigned it and why?
My understanding is that RT doesn't actually include reviews without ratings in its computation of average rating. So, in this respect, it's different from how Metacritic uses numbers. But someone can correct me on this if I'm wrong.
A minor correction to your description of Metacritic: the Metascore is not a simple average, but a weighted and normalized one. The weights are based on their perception of a critic or publication's quality and stature and the normalization "prevents clumping".
http://www.metacritic.com/about-metascores
I've always found Metacritic's scores and aggregation to be much more useful and easy to navigate than the free-for-all dichotomy of Rotten Tomatoes.
Thanks for that info. Obviously, some subjectivity is involved in who they choose to include, but I didn't know they also weighted the scores according to their assessment of a critic's or outlet's reputation. So, what we have is a numerically averaged subjective appraisal of subjective appraisals of subjective appraisals of movies.
When I saw the graph I thought of the movie The Hollywood Sign. There is a bit of dialogue about the stages of an actor's career - who is this guy, get me this guy, get me someone like this guy, who is this guy? Poorly remembered paraphrase, but loved it and enjoyed the movie.
I have a Rotten Tomatoes area on Entertainment homepage. Mainly I click on it when they are asking people about their favorite movies. RT seems to be mostly about buzz.
Different strokes.
First: my apologies for saying it was a 5 point scale. I did not double check and confused it with the user score.
As for what rating Rotten Tomatoes assigned to AO Scott's review. Well, that's easy. Sort by "Top Critics" and look under Scott's blurb. Rotten Tomatoes has assigned a score of 3.0/5.0 to Scott's review which, coincidentally(?), matches the '60' that Metacritic assigned to the review. Is that right? Perhaps not, but it does read like a very lukewarm recommendation.
As to why they assigned that score, well, your guess is as good as mine. I'm not a huge fan of the tomatometer but I felt it best to clear up some misconceptions.
Finally, I'd like to add that I'd much prefer watch a film with wildly mixed reviews like Antichrist or Enter The Void instead of a "safe" picture that has a high tomatometer but is ultimately about nothing. RT is a convenient way to read your favourite critics, but I fear that it's encouraging "groupthink" where the intelligent dissenter is ridiculed and dismissed. Shame.
Of course a well thought out review from a critic I trust (Michael Phillips, AO Scott, Roger Ebert, many of the staff writers at the AV Club, and the opposite of whatever Armond White says) will always yield a more complete picture of a film, but I personally find RT to a great way to get a quick snapshot of a movie. A 20% is a pretty clear indication that there are probably better options and an 80% indicates that a movie may deserve some further attention. As a first pass filter this method works great.
Statistically, I would guess that all the lukewarm reviews that mistakenly get recorded as “Rotten” would average out the near-miss reviews that mistakenly get recorded as “Fresh”, and, in the aggregate, would move the Tomatometer score towards the mediocre, where it belongs.
Beyond that first pass, though, it’s really just fun with numbers. It can be fun to see who has the best batting average against left handed pitchers in June in cities that contain the letter “E”, or to “plot the career” of a director/actor. This only actually becomes a problem when people are led to believe that these statistics have any sort of real world application or importance and give them weight beyond the trivia they are intended to be.
Hey Jim,
Matt Atchity here, Editor-in-Chief at RT. I think Slate did an interesting piece, although I don't think I would have couched it as saying which actors are the "best" or "worst." We try and be pretty clear that we're measuring a very broad spectrum of critical reaction to film, and I think the Tomatometer is a pretty good indicator of that.
Regarding our methodology, let me first say that a 2.5 star review from Roger will get marked as "rotten," based on direction from the man himself. In many cases, reviewers and critics are submitting their own reviews, and they mark them fresh or rotten themselves. When we're adding a review ourselves, we try and interpret whether or not a review is ultimately positive or negative, and many times, we'll reach out to a reviewer for a final ruling.
It seems like there are a lot of misconceptions about how we do what we do. If you have deeper questions about our methodology, feel free to shoot me an email, I'll be happy to answer them for you.
-Matt
Thanks much for your reply, Matt. Another commenter also pointed out that RT evaluates Roger's star ratings differently from others (because of Roger's own interpretation of his star scale -- which is that "recommended" movies are 3 stars and up). No method is going to be foolproof. I think the problem is when people misuse the data -- like those silly conformists who think they can use Tomatometer ratings to defend or justify their own likes/dislikes.
I dislike RT as it that seems to value consensus over objectivity.
I also find the RT reviews v v biased in either direction. It's either a 1 star or a 5 star. There are no shades of grey and that is why top, professional reviewers still have a place in my life.
I do like Metacritic as it gives me a nice overview of reviews on a single screen
In it's infancy RT was great, it aggregated only the best of the best but as time went on the critic's list grew an grew.
Now you have folks like:
Jeff Bayer, Scorecard review
Widget Walls, needcoffee.com
Eric D. Snider, ericdsnider.com
Couldn't the latter simply be a guy in his basement with too much time in his hands?
Oh and let's not forget the bunch are supposed to guide the others in their flocks like:
Steven D. Graydanus, Christianity Today
Sorry RT, you don't aggregate reviews anymore but merely people with an opinion.
Actually, Eric D. Snider is a fairly accomplished online critic, certainly more so than the majority of reviewers who are included in the Tomatometer. RT does have a minimum requirement about how many reviews a writer must be able to deliver annually in order to qualify for continued inclusion. Thus, since the included reviewers watch movies frequently and take the time effort to write, they are expressing at least somewhat informed and articulate opinions. They are not the opinions of just any person off of the street.
What exactly is wrong with "a guy in his basement with too much time in his hands"? The notion that a reviewer without professional credentials must have inferior film criticism skills is ludicrous. If Jim were to decide to post reviews in this blog on a semi-regular basis, he could be included in the Tomatometer as well. But he would not be labeled a "Top Critic," so to any unfamiliar RT user, Jim might seem to be no different and no better than any of the no-name bloggers that you're so quick to lambaste. Ebert's hand-picked hosts for his new TV show are a very interesting pairing with a very interesting dynamic. Christy Lemire is very accomplished in the mainstream press, as the longtime primary film critic for AP. But her perspective is very populist. Ignatiy Vishnevetsky has no particularly notable press career, but his perspective is in the realm of "sophisticated" criticism, much more in line with Jim than Lemire is. I won't say that one is a better critic than the other, but my point is that to automatically proclaim one to be better simply on the basis of being employed by a mainstream publication is untenable.
RT isn't about linking reviews from "the best of the best." It's about creating a broad survey of opinions from very diverse perspectives. People like Harry Knowles, who represent the "fanboy" crowd, have opinions that are just as valid (if not universally valuable) as those of the likes of Manohla Dargis and A. O. Scott. If RT doesn't represent the kinds of opinions that you want to see, then don't use it. Metacritic sounds more useful to you. There's no need criticize RT for having a different mission than MC.
"I dislike RT as it that seems to value consensus over objectivity."
First of all, I don't see why presenting a consensus is a problem. RT is a resource specifically designed to allow those who are curious about the critical consensus to access it. It's not saying that the consensus is the be-all and end-all statement on a movie's value.
Second of all, the notion about valuing objectivity is nonsensical. Admittedly, I'm not sure what you mean "objectivity," but in any case movie reviews aren't objectivity. They're about reporting subjective opinions. The only objectivity that comes to my mind is the pursuit of reporting such opinions in the most honest manner. Ratings and grades may seem "objective," but as tools of criticism they are very limited, which is why many critics choose not to assign them at all.
Also, I don't understand why you think that reviews on RT tend to be either one star or five stars. The reviews themselves almost always feature shades of gray, but the binary distinction between "Fresh" and "Rotten" is really no different from Siskel and Ebert's famous thumbs-up and thumbs-down. They do not represent any extremes of opinion; they are merely answers the very basic question, "Did you like the movie or not?"
Why isn't Stephanie Zacharek, formerly of Salon, now of Movieline, also considered a "Top Critic"? I think they read her reviews of The Dark Knight and Inception.
Welcome to the problems that every quantitative researcher (especially in the social sciences) faces, where a method must be chosen that is potentially flawed but ultimately determined to be the best method possible from the available data. I would like to make the argument though that every method would be somewhat flawed and that the Tomatometer arguably uses as good as any method for determining a quantified ranking for the quality of movies. Sure, if you just look at a few rankings then you are likely to get a misrepresentation of the quality of the movie. But if you look at a large sample size of rankings, you are increasingly likely to get a more accurate score for the quality of a film. You point out that some negative reviews come out positive. But there are also some critics who are unusually harsh on films that don't deserve the amount of negativity they get. The point is that with a high sample size, these different views add up to an aggregate score for the films that reflects a particular level of favorability for these films. When this favorability rating is taken as a relative score that is comparable to the tomatometer scores for all other films, it is meaningful.
I would argue that the major problem here is the way Salon has interpreted the data. It is incorrect to interpret the data as saying that John Ratzenberger is the best actor. What Salon should have said in quite literal terms is that John Ratzenberger is the actor who has, when taking all of his films together, been in the highest ranking films. It's not the fault of the Tomatometer people that others are misusing their data, trying to apply it for purposes for which it is simply not designed or intended.
Jim, this is something I realized a while ago. All they consider in their reviews is positive and negative. And considering some critics use no rating system whatsoever, comments have to be interpreted.
It all boils down to this. If the "Tomatometer" for a film is floating around say, 50-70%, it's sort of meaningless. I only think it helps a moviegoer is when it's at nearly 100% or 30% and under. In those cases, it's an okay indicator if a movie's really bad or really good. But if every critic happens to agree a movie is "not bad", than a mediocre movie will look like a great one, even if the "postive" review is half-hearted.
I know this topic is close to a month old, but if you get the chance you should read the reviews by James Kendrick for Q Network Film Desk. He may be a little less liberal with the four-star rating, but he is actually one of the better writers I've discovered thanks to Rotten Tomatoes.
Leave a comment