Do human “quality raters” influence which sites are impacted by Panda?

Matt's answer:

Today’s question comes from Dimwit. Dimwit asks, “If you have human quality raters evaluating the search engine result pages and influencing which sites may be impacted by Panda, how do you confirm that consumers are more satisfied with the results?” OK, so there’s a problem with this question, and it’s the word influencing. So we have evaluation raters who look at the quality of pages, using their own judgment, as well as guidelines that we give them on when things are navigational, when things are vital, which things are off-topic, which things are spam. All that sort of stuff. But those folks don’t influence our algorithm in any direct sense. So when an engineer has an idea. Suppose it’s an idea for a new algorithm, we’ll call it Panda, he’ll come up with an algorithm, and it will rank the results 1 through 10. And so you’ll have a side by side, left side and right side. So you actually have the results right there. That goes out to the evaluation team and these human quality raters. And as a blind taste test, they say, I prefer the left side of the search results or the right side of the search results– is that the right side, OK– and then we’ll get that feedback back. But that evaluation, where the search quality evaluators say I prefer this side or I prefer that side, does not directly affect the algorithm. It doesn’t affect Panda. So if somebody says, well, I don’t like this site in an evaluation, that doesn’t mean that that site will rank lower. We’re just using that as feedback to say the algorithm is doing what we expect based on our intuition and experience as ranking engineers. The people who look at the sites and look at the changes in the rankings in general agree that the page quality overall goes up, that the relevance goes up, all that sort of stuff. So it’s a fine question, but there is a flaw there, which is saying, OK, the raters are influencing the sites. And that’s not the case. The raters are saying, for a given algorithm, is this site better or worse? Is the set of search results better or worse? And then we use that in order to say did this algorithm change match our intuition, and do the sorts of things that we think are the best things for users. So it’s not the case that there’s a human quality evaluation rater person who is voting down a site, and, as a result, Panda will rank a site lower. I hope that clears up a misconception, and we might be able to make those human quality rater guidelines that we make available to people at Google available to the larger world. And I think that would be a good thing because then people would be able to read through it. It leaked a few years ago, and what someone said was– the biggest surprise is that there weren’t really that many surprises. All the guidelines that we provide are pretty much common sense and would match with, I think, what just about anybody would say about yeah, it does make sense that this is a navigational page or that this page is off topic. But it’s not the case that the ratings that we get back when we’re assessing how good an algorithm is will directly cause a site to go down or move up or anything like that. So that’s something you don’t need to worry about.

by Matt Cutts - Google's Head of Search Quality Team

Original video: