Expectations Create Negative Survey Bias

Research from Stanford School of Business Professor Itamar Simonson and coauthor Chezy Ofir at Hebrew University in Jerusalem, points out that telling customers they will be surveyed, or asking them about their expectations ahead of time creates significantly more negative feedback. A quote from the article:

The researchers found that people who expect to evaluate are decidedly more negative. They also discovered that merely asking people to state their expectations before they receive a service made people morenegative—even though their predispositions may have been quite positive. For example, people who are asked if they think they will like a movie before seeing it will be statistically more negative than people who were never asked that question.
Simonson and Ofir studied the responses of customers who had called for service at a major computer hardware and software company. The researchers divided the customers into four groups. Participants in the first group were told a technician would service their problems and that they would subsequently be asked about the service, such as whether the tech was on time, whether the employee was polite, and whether he or she solved the problem. A second group was not told there would be an evaluation, but the customers were asked to state their expectations, such as how long they thought it would take for a tech to arrive. A third group was told both: to state their expectations and to expect a survey. Members of a control group knew nothing but were later polled.
The result: People who expected to evaluate were significantly more negative than members of the control group. The same was true of the group asked to state their expectations ahead of time. Interestingly, the group that was the most dissatisfied was the one that was asked their expectations and also warned about a survey.

This has serious implications for customer satisfaction surveys, but also for product research groups. Showing product prototypes to customers in a research setting is a context in which participants will frequently both be asked about their expectations and expect a survey. The effect can be research that "finds" problems that aren't really problems:

The researchers warn that while marketers must stay on top of customer desires and complaints, they must also be aware of the effects the mere expectation of filling out a survey can have on how customers view their experience. "It may not be realistic," says Simonson. "They may be chronically more negative, pointing out problems that are not problems to the average consumer," he says. "You want people who are representative of the marketplace."

This suggests that if you have any opportunity for analysis that doesn't rely on surveys, but instead relies on behavior, the results are likely to be more accurate. Social media buzz, word of mouth, and collective intelligence applications based on behavior may all be more accurate than survey responses.