Late this summer, six Adams State University psychology students and three of their professors were among 270 co-authors on an article published in the journal Science regarding the Reproducibility Project: Psychology.The four-year-long study has produced the most comprehensive investigation ever done about the rate and predictors of reproducibility in a field of science. The study’s conclusions have ramifications that could impact scholarly research beyond the social sciences.
The popular – and academic – press frequently carry stories of new research findings from a range of disciplines. What may not be reported are less-than-stellar results, or negations of hypotheses. Further, "Results only have validity if they can be replicated. That’s what we tell students, but we don’t walk the talk," according to Dr. Kim Kelso, professor and chair of the psychology department.
Reproducibility means the results recur when the same data are analyzed again, or when new data are collected using the same methods. However, neither researchers nor academic journals are eager to do replications. "The pressure is to discover the next bigger, better thing," said Dr. Leslie Alvarez, professor of psychology.
With the goal of bringing more transparency to research methods and the publication process, Open Science Framework launched The Reproducibility Project: Psychology. Researchers conducted replications of 100 published findings of three prominent psychology journals. The study found that regardless of the analytic method or criteria used, fewer than half of the replications produced the same findings as the original study.
International collaboration
When Alvarez learned of the Reproducibility Project, she "realized what an amazing opportunity it was for us at a small school. Our students and faculty have been able to contribute to a historical effort in psychology. We are part of an international group of psychologists, and we did something under the leadership of the Open Science framework that has never been done before."
The inability to replicate original findings doesn’t mean they are invalid or were falsified, however. "Just because results can’t be replicated doesn’t mean they are negated," noted Dr. R. Nate Pipitone, assistant professor of psychology. Kelso added that a replication might find similar, but less robust trends. "There is a trend now in the social sciences of not publishing results that are not statistically significant, even though they may be meaningful," she said. Unknown variables or different physical characteristics of the studies may also vary and influence results. "From a teaching and learning perspective, one outcome of the project may be to encourage better methods descriptions to allow more accurate replication."
Hands-on research
The three Adams State faculty invited students to undertake replication of two studies for the project. Pipitone and Alvarez worked on "Detecting the Snake in the Grass: Attention to Fear-Relevant Stimuli by Adults and Young Children," with Tylar Martinez, Nicholas Spencer, Megan Tapia, and Kellylynn Zuni.
The study’s premise was that both adults and children should be able to detect evolutionarily relevant fear stimuli more readily than non-fear stimuli. To test that, the ASU team worked with parents and children at two local day care centers. Each person was shown two images with nine photos. One image included 8 snakes and one caterpillar, the other, the reverse. Researchers measured the time it took subjects to notice the single snake or caterpillar. Pipitone said their group found similar patterns as the original study, in that the snakes were identified sooner than caterpillars, but the results were not statistically significant.
"The Space Between Us: Stereotype Threat and Distance in Interracial Contexts" was the study reviewed by Kelso with Spencer, Ashlee (Bogle-DeHerrera) Welsh, and Emily Wright. They studied ASU students, 101 male subjects with an African-American male "confederate," who understood the experiment’s intent. Told they would be having a conversation either about racial profiling or love and relationships, the subjects entered a room where the confederate was already seated. The study looked at how far the subject placed his chair from the confederate’s, depending on which conversation they were expecting.
The original study, which used all white male subjects, found more distance in conjunction with the conversation about race. Although ASU’s work did not replicate those results, exploratory analyses on the three largest ethnic groups (White, Hispanic/Latino, and Black/African American) revealed participants sat further away when the topic was racial profiling.
The project also cultivated inter-disciplinary collaboration. George Sellman, asst. professor of mathematics/computer science, and student Lauren Karlskin created a web-based program for data collection in the "Snake in the Grass" study. The psychology students presented on this project at Adams State’s Student Scholar Days and the Rocky Mountain Psychological Association (RMPA).
Spencer, who’s interested in behavioral and evolutionary psychology, as well as school psychology, said participating in the project gave him valuable career preparation. "It was exhilarating and educational to be able to apply theories in a practical application. Replicating these studies put you in the shoes of people greater than yourself." He discussed the project with the original "Space Between Us" researchers at the Pennsylvania State University, Stanford University, and University of British Columbia.
"Now that I am in graduate school, I realize the impact replication has on my practice and profession as a school psychologist. In order to better serve kids, it is important to have research that is replicable across a variety of settings," said Tapia. "I think being able to collaborate as a team, along with faculty, in the community was one of the most rewarding experiences about this project. Without them, this project would not have been possible. "
The Adams State group participated with researchers from across the U.S. and several other nations, including such institutions as the Massachusetts Institute of Technology, University of Oxford, Harvard Medical School Department of Neurology, the Stockholm School of Economics, and many more.
Note: Emily Wright is now a caseworker for the Alamosa County Department of Human Services Child Welfare office; Ashlee Welsh is Enrichment Supervisor for Center Consolidated School District; Kellylynn Zuni is a Ph.D. candidate in clinical psychology at Palo Alto University; Megan Tapia is a graduate student in school psychology at University of Northern Colorado.
