Harvard Study of Social Norms Deserves "F" Grade for Flawed Research Design
by H. Wesley Perkins, Ph.D. and Jeffrey W. Linkenbach, Ed.D.
When it comes to evaluating the recent controversial study and associated press commentary about social norms by Harvard University's Henry Wechsler, the validity of his study and commentary gets an "F" grade, but the publicity generated from it receives an "A plus." In Wechsler's latest attempt to challenge the social norms approach to promoting healthy behavior he claims to provide a "national evaluation of social norms marketing" on college campuses. Given the prestigious academic platform of a Harvard project, the public would logically assume that the study has merit-but it does not.
The purported evaluation of the social norms strategy by Wechsler
creates the illusion of an objective critique, but for years now
he has spoken disparagingly about the approach without being knowledgeable
about the research and theory supporting it. The social norms approach
works by intervening to challenge and correct students' misperceptions
of the college drinking scene. It does something radically different
than traditional scare tactics-it tells students the truth using
marketing campaigns and other strategies to give students accurate
the majority of their peers drinking moderately or not drinking at all.
Although alcohol abuse is a very serious problem, it is not the
behavior of the majority of students on any campus as revealed in
both survey self reports and in studies using breathalyzer tests
among representative samples of students on weekend late nights.
Yet if a high risk behavior is erroneously perceived to be "normal"
and acceptable, as documented in
more than forty studies, then some individuals will feel a social urge to conform to this illusion, and students already prone to risky behavior can engage in it freely believing they are just like everyone else.
In attempting to discredit social norms, Wechsler claims to show no positive effect at schools labeled as doing social norms marketing compared to other schools. He labels his study as "nationally representative" of schools, but one half of the 195 schools originally selected for the College Alcohol Study in 1993 never participated, subsequently dropped out, or had to be disqualified due to extremely low response rates. At the remaining 98 schools in the 2001 study, half of students responded on average resulting in an overall response rate of only 26% based on the originally representative sample of colleges drawn in 1993. Furthermore, at individual schools, small samples and low response rates lead to some large universities being represented by as few as 50 or 60 respondents. This news did not make the headlines.
The most fundamental and deceptively simple flaw of this study is the failure to distinguish between schools legitimately doing social norms marketing and those that are not. At the time of the study, one would have been hard pressed to identify more than 5% of schools nationwide promoting actual positive norms without also including conflicting scare tactics. Yet the Wechsler study classified almost 40% of schools in this category simply because a single administrator said they had done some sort of social norms campaign. Most readers of the report would be led to erroneously assume that the study successfully distinguished this particular prevention approach from others strategies when in fact it did not.
Although Wechsler states that his study "looked at social norms programs in every conceivable way," there is not one single question in his student survey that asks about exposure to actual social norms messages (i.e. data based messages about actual norms). It just asks generically about exposure to alcohol education material and drinking rates. On most campuses an affirmative to these questions would indicate recall of scare messages or Wechsler's familiar "binge" rates that have been widely publicized. We encourage the scientific community to examine the actual survey in this regard.
In claiming no scientific evidence for the efficacy of social norms, Wechsler exhibits an inexplicable ignorance of the scores of journal articles and book chapters providing data showing the strong connection between perceptions and personal behavior. Research reported elsewhere in detailed case studies and controlled experiments testing the social norms approach has produced substantial reductions in student substance abuse in schools and communities large and small across the country.
Wechsler's other now familiar but equally misleading criticism is that the alcohol beverage industry is funding the approach as if it was something cooked up by distributors to divert prevention efforts. Nothing could be more inaccurate. All of the primary case studies appearing throughout the last decade in support of this approach were implemented and evaluated with federal, state, and local school grants that had absolutely no ties to the beverage industry. Furthermore, providing accurate normative feedback about peers and communities to increase healthy behavior has been successfully applied (and reported in published work) to smoking prevention, recycling campaigns, seat belt use, and violence prevention-other projects clearly having no connection with beverage industry support.
When you have the Harvard name and a large grant for publicity, you can use some very questionable data and still make your work sound definitive in the media. Dr. Wechsler should agree to discuss his views in a panel forum where public debate is possible such as at a professional conference or national meeting where other prevention experts could directly challenge his claims and question his methods. Unfortunately, he has avoided these invitations. The public deserves to know the truth about how erroneous negative headlines are being generated from this study.
H. Wesley Perkins, Ph.D. is Professor of Sociology at Hobart and William Smith Colleges in Geneva, New York and can be reached at (315) 781-3437 or by email at email@example.com. He is editor of the new book, The Social Norms Approach to Preventing School and College Age Substance Abuse (Jossey-Bass, 2003). Jeffrey W. Linkenbach, Ed.D. is a research faculty member and Director of the statewide Montana Social Norms Project based at Montana State University in Bozeman, Montana and can be reached at (406) 994-3837 or by email at firstname.lastname@example.org.
*Originally distributed as an op-ed on the Higher Education Center for Alcohol and Other Drug Prevention listserv. Posted by permission of the authors.
filed under: [pending]