Students and faculty in Adams State University’s Psychology Department, along with 261 co-authors, had their findings on the reproducibility of psychological science published in the journal Science. Launched nearly four years ago, the Reproducibility Project: Psychology has produced the most comprehensive investigation ever done about the rate and predictors of reproducibility in a field of science.
The international study has a total of 270 authors including the following from Adams State:
Dr. Leslie Alvarez, assoc. professor of psychology; Dr Kim Kelso, professor and chair of the Psychology Department; Dr. R. Nate Pipitone, asst. professor of psychology; current student Nicholas Spencer; and recent psychology graduates Tylar Martinez, Megan Tapia, Kellylynn Zuni, Ashlee (Bogle-DeHerrera) Welsh, and Emily Wright. In addition George Sellman, asst. professor of mathematics/computer science, and student Lauren Karlskin created a web-based program for data collection using the picture stimuli provided by the original author. The psychology students presented on this project at Adams State’s Student Scholar Days and the Rocky Mountain Psychological Association (RMPA).
“Once I learned about this project, I realized what an amazing opportunity it was for us at a small school. Our students and faculty have been able to contribute to a historical effort in psychology,” Alvarez said. “We are part of an international group of psychologists, and we did something under the leadership of the Open Science framework that has never been done before.”
The project conducted replications of 100 published findings of three prominent psychology journals. They found that regardless of the analytic method or criteria used, fewer than half of their replications produced the same findings as the original study. Mallory Kidwell, one of the project coordinators from the Center for Open Science, concluded, “The results provide suggestive evidence toward the challenges of reproducing research findings, including identifying predictors of reproducibility and practices to improve it.”
Alvarez added, “Replication is something we talk about all the time in STEM, but direct replications are seldom done. An amazing thing about this project was that we followed a rigorous methodology and were encouraged to communicate directly with our original author – which we did and was a great experience. Our students discovered first-hand what it was like to review a methods and procedure section with a fine-toothed comb and re-create it as authentically as possible.”
Science is unique from other ways of gaining knowledge by relying on reproducibility to gain confidence in ideas and evidence. Reproducibility means that the results recur when the same data are analyzed again, or when new data are collected using the same methods. As noted by Angela Attwood, team member from University of Bristol, “Scientific evidence does not rely on trusting the authority of the person that made the discovery. Rather, credibility accumulates through independent replication and elaboration of the ideas and evidence.”
The team emphasized that a failure to reproduce does not necessarily mean the original report was incorrect. Elizabeth Gilbert, team member from the University of Virginia, noted, “A replication team must have a complete understanding of the methodology used for the original research, and shifts in the context or conditions of the research could be unrecognized but important for observing the result.” Belen Fernandez-Castilla, team member from Universidad Complutense de Madrid, added, “Scientists investigate things that are not yet understood, and initial observations may not be robust.”
Yet a problem for psychology and other fields is that incentives for scientists are not consistently aligned with reproducibility. “What is good for science and what is good for scientists are not always the same thing. In the present culture, scientists’ key incentive is earning publications of their research, particularly in prestigious outlets,” said Ljiljana Lazarevic, team member from the University of Belgrade. Research with new, surprising findings is more likely to be published than research examining when, why, or how existing findings can be reproduced. As a consequence, it is in many scientists’ career interests to pursue innovative research, even at the cost of reproducibility of the findings.”
Fewer than half of the original findings were successfully replicated. This held true across multiple different criteria of success. The team noted three basic reasons this might occur:
- Even though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where, or how the replication was carried out might have influenced the results.
- The replication might have failed to detect the original result by chance.
- The original result might have been a false positive.
Johanna Cohoon, another Project Coordinator from the Center for Open Science, concluded, “the findings demonstrate that reproducing original results may be more difficult than is presently assumed, and interventions may be needed to improve reproducibility.” In keeping with the goals of openness and reproducibility, every replication project posted its methods on a public website, and later added their raw data and computer code for reproducing their analyses.
Alvarez also noted, “The Open Science framework is also working to bring more transparency to the field. Things are beginning to happen online that never existed before, such as a badge to indicate that authors have made their data set available and a badge to indicate that a project was pre-registered (meaning it was accepted for publication based on the quality of the literature review and proposed methods – NOT on the actual findings, which should diminish publication bias against null results).
Many organizations, funders, journals, and publishers are already working on improving reproducibility. For example, in 2014, the journal Psychological Science – one of the journals included in this study – implemented practices such as badges to acknowledge open sharing of materials and data to improve reproducibility. “Efforts include increasing transparency of original research materials, code, and data so that other teams can more accurately assess, replicate, and extend the original research, and pre-registration of research designs to increase the robustness of the inferences drawn from the statistical analyses applied to research results,” said Denny Borsboom, a team member from the University of Amsterdam who was also involved in creation of the Transparency and Openness Promotion (TOP) Guidelines published in Science in June.
Since the Reproducibility Project began in 2011, similar projects have emerged in other fields, such as the Reproducibility Project: Cancer Biology. And a discipline of metascience is emerging – scientific research about scientific research. These and the widespread efforts to improve research transparency and reproducibility are indications that, as suggested by team member Susann Fiedler from the Max Planck Institute for Research on Collective Goods, “Science is actively self-examining and self-correcting to maximize the quality and efficiency of the research process in the service of building knowledge for the public good.”
