Open Science Collaboration Research Articles

Bringing an Intersectional Lens to “Open” Science: An Analysis of Representation in the Reproducibility Project

Feminist psychologists have called for researchers to consider the social and historical context and the multidimensionality of participants in research studies. The Reproducibility Project documents the degree to which the findings from mainstream psychological studies are reproduced. Drawing on intersectionality theory, we question the value of reproducing findings while ignoring who is represented, intersecting social and group identities, sociohistorical context, and the power and privilege that likely influence participants’ responses in psychology experiments. To critically examine the Reproducibility Project in psychology, we analyzed the 100 replication reports produced between 2011 and 2014 (Open Science Collaboration, 2015). We developed an intersectional analytic framework to investigate (a) representation, (b) whether demographic and identity factors were considered through a multidimensional or intersectional lens, (c) explanations of non-replication, and (d) whether socio-cultural context was considered. Results show that reports predominantly include WEIRD samples (people from Western, educated, industrialized, rich, and democratic countries). Context and identity were rarely considered, even when study design relied on these factors, and intersectional identities and structures (considering power, structural issues, discrimination, and historical context) were absent from nearly all reports. Online slides for instructors who want to use this article for teaching are available on PWQ's website at http://journals.sagepub.com/doi/suppl/10.1177/03616843211035678

A Beginner's Guide to Conducting Reproducible Research

A Beginner's Guide to Conducting Reproducible Research

Psihološka nauka od "objavi ili nestani" do "veruj, ali proveri"

At the time when social psychologists believed they could be proud of their discipline, there was the devastating news that Diederik Stapel had committed a major scientific fraud. This event coincided with the start of the discussion on trust in psychological findings. It was soon followed by the report of a series of nine studies that failed to replicate the 'professor's study'. These replication results were astounding due to earlier reports of successful replications. Due to the crisis of confidence in the results of field research, the Open Science Collaboration subsequently replicated 100 correlation and experimental studies published in 2008 in Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition. Of the 97% of the original studies that had a positive effect, 36% were replicated. However, their findings have also been called into question by calculating the Bayesian factor. In addition to fraud, questionable research practices resulting from publication bias that results in false positives undermine confidence in the validity of psychological research findings. Perhaps the most costly mistake of false-positive findings is to erroneously reject the null hypothesis. However, that Stapel (2011) confirmed the null hypothesis, or that Bargh (1996) found that admission of participants did not affect walking speed, or that Dijksterhuis and van Knipenberg (1998) reported that participants received with the word 'professor' did not improve their performance on task, no one would be interested in their findings. Zero findings are only interesting if they contradict the main hypothesis derived from the theory or contradict a number of previous studies. The fact that good experimental research is usually conducted in order to test theories, researchers can never be sure whether they have chosen the optimal operationalization of a given construct. As researchers can never be sure that they have properly operationalized the theoretical constructs they are evaluating and whether they have been successful in controlling the third variables that may be responsible for their results, the theory can never be proven true.

Collecting big data with small screens: Group tests of children\u2019s cognition with touchscreen tablets are reliable and valid

Collecting experimental cognitive data with young children usually requires undertaking one-on-one assessments, which can be both expensive and time-consuming. In addition, there is increasing acknowledgement of the importance of collecting larger samples for improving statistical power Button et al. (Nature Reviews Neuroscience 14(5), 365–376, 2013), and reproducing exploratory findings Open Science Collaboration (Science, 349(6251), aac4716–aac4716 2015). One way both of these goals can be achieved more easily, even with a small team of researchers, is to utilize group testing. In this paper, we evaluate the results from a novel tablet application developed for the Resilience in Education and Development (RED) Study. The RED-app includes 12 cognitive tasks designed for groups of children aged 7 to 13 to independently complete during a 1-h school lesson. The quality of the data collected was high despite the lack of one-on-one engagement with participants. Most outcomes from the tablet showed moderate or high reliability, estimated using internal consistency metrics. Tablet-measured cognitive abilities also explained more than 50% of variance in teacher-rated academic achievement. Overall, the results suggest that tablet-based, group cognitive assessments of children are an efficient, reliable, and valid method of collecting the large datasets that modern psychology requires. We have open-sourced the scripts and materials used to make the application, so that they can be adapted and used by others.

Theory and the Replication Problem

Many refinements of statistical design have been offered to solve the replication problem identified by the Open Science Collaboration and Camerer and colleagues. There are, however, two distinct kinds of experimentation: Fisher design and theory designed. Therefore, there are two kinds of replication. Only for the Fisher design does replication reproduce conditions of prior experiments in order to compare new with prior results, and only there has a replication problem been demonstrated. In contrast, replications for theory-designed experiments test experimental results against theoretical predictions, and only for theory-designed experiments can replication be extended broadly across the scope of a theory. The authors analyze the logic of the two types of experiments as well as hybrids that mix qualities of both.

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

The Evolution of Behavior Analysis: Toward a Replication Crisis?

The Open Science Collaboration (Science, 349(6251), 1-8, 2015) produced a massive failure to replicate previous research in psychology-what has been called a "replication crisis in psychology." An important question for behavior scientists is: To what extent is behavior science vulnerable to this type of massive replication failure? That question is addressed by considering the features of a traditional approach to behavior science. Behavior science in its infancy was a natural science, inductive, within-subject approach that encouraged both direct and systematic replication. Each of these features of behavior science increased its resistance to three factors identified as responsible for the alleged replication crisis: (1) failures to replicate procedures, (2) low-power designs, and (3) publication bias toward positive results. As behavior science has evolved, the features of the traditional approach have become less ubiquitous. And if the science continues to evolve as it has, it will likely become more vulnerable to a massive replication failure like that reported by the Open Science Collaboration (Science, 349(6251), 1-8, 2015).

Assessing evidence for replication: A likelihood-based approach.

How to evaluate replications is a fundamental issue in experimental methodology. We develop a likelihood-based approach to assessing evidence for replication. In this approach, the design of the original study is used to derive an estimate of a theoretically interesting effect size. A likelihood ratio is then calculated to contrast the match of two models to the data from the replication attempt: (1) a model based on the derived theoretically interesting effect size, and (2) a null model. This approach provides new insights not available with existing methods of assessing replication. When applied to data from the Replication Project (Open Science Collaboration, 2015), the procedure indicates that a large portion of the replications failed to find evidence for a theoretically interesting effect.

Cognition in Context: Plasticity in Cognitive Performance in Response to Ongoing Environmental Variables

Cognition has evolved to allow organisms to process, use and store information in their natural environment. Yet, cognitive abilities are traditionally measured in controlled laboratory conditions to obtain consistent and accurate measurements. Consequently, little is known about the actual effect of natural environmental variation on cognitive performances. Being able to modify cognitive performance according to environmental conditions (e.g. plasticity of attentional performances according to current predator densities) could provide evolutionary advantages. In this mini-review, we compile evidence for what we call “cognitive performance plasticity” (i.e. flexible adjustment of cognitive performance in response to the current environment). We then discuss methodological approaches associated with measurement of cognitive performance plasticity and cognition in general. Finally, we discuss the implications of acknowledging plasticity in cognitive performance, including a better understanding of the reproducibility crisis observed in cognitive sciences (Open Science Collaboration, 2015) and new lines of inquiry into the evolution of cognition and the adaptive value of cognitive performance plasticity.

Do we know what we are doing?

In November 2012, the newly created Open Science Collaboration published a brief article announcing a multi-year effort to “estimate the reproducibility of psychological science.” The collaboration was directed by Brian Nosek of the University of Virginia and would eventually involve over 250 co-authors. According to the collaboration, reproducibility was one of, if not the single most defining feature of the social endeavor known as “science.” “Other types of belief,” the authors write, “depend on the authority and motivations of the source; beliefs in science do not.” The ability to reproduce scientific results across time and space – the ability to have results be independent of the individuals involved – is what the authors argued makes science science. And yet the eventual findings of the reproducibility project showed a remarkable reproductive failure. Over half of all studies failed to indicate similar effects upon replication. The very value upon which science was supposed to be founded appeared to be an exception rather than a norm.

A New Standard for the Analysis and Design of Replication Studies

SummaryA new standard is proposed for the evidential assessment of replication studies. The approach combines a specific reverse Bayes technique with prior-predictive tail probabilities to define replication success. The method gives rise to a quantitative measure for replication success, called the sceptical p-value. The sceptical p-value integrates traditional significance of both the original and the replication study with a comparison of the respective effect sizes. It incorporates the uncertainty of both the original and the replication effect estimates and reduces to the ordinary p-value of the replication study if the uncertainty of the original effect estimate is ignored. The framework proposed can also be used to determine the power or the required replication sample size to achieve replication success. Numerical calculations highlight the difficulty of achieving replication success if the evidence from the original study is only suggestive. An application to data from the Open Science Collaboration project on the replicability of psychological science illustrates the methodology proposed.

Confounds in "Failed" Replications.

Reproducibility is essential to science, yet a distressingly large number of research findings do not seem to replicate. Here I discuss one underappreciated reason for this state of affairs. I make my case by noting that, due to artifacts, several of the replication failures of the vastly advertised Open Science Collaboration’s Reproducibility Project: Psychology turned out to be invalid. Although these artifacts would have been obvious on perusal of the data, such perusal was deemed undesirable because of its post hoc nature and was left out. However, while data do not lie, unforeseen confounds can render them unable to speak to the question of interest. I look further into one unusual case in which a major artifact could be removed statistically—the nonreplication of the effect of fertility on partnered women’s preference for single over attached men. I show that the “failed replication” datasets contain a gross bias in stimulus allocation which is absent in the original dataset; controlling for it replicates the original study’s main finding. I conclude that, before being used to make a scientific point, all data should undergo a minimal quality control—a provision, it appears, not always required of those collected for purpose of replication. Because unexpected confounds and biases can be laid bare only after the fact, we must get over our understandable reluctance to engage in anything post hoc. The reproach attached to p-hacking cannot exempt us from the obligation to (openly) take a good look at our data.

FluidSim: Modular, Object-Oriented Python Package for High-Performance CFD Simulations

The Python package fluidsim is introduced in this article as an extensible framework for Computational Fluid Mechanics (CFD) solvers. It is developed as a part of FluidDyn project (Augier et al., 2018), an effort to promote open-source and open-science collaboration within fluid mechanics community and intended for both educational as well as research purposes. Solvers in fluidsim are scalable, High-Performance Computing (HPC) codes which are powered under the hood by the rich, scientific Python ecosystem and the Application Programming Interfaces (API) provided by fluiddyn and fluidfft packages (Mohanan et al., 2018). The present article describes the design aspects of fluidsim, viz. use of Python as the main language; focus on the ease of use, reuse and maintenance of the code without compromising performance. The implementation details including optimization methods, modular organization of features and object-oriented approach of using classes to implement solvers are also briefly explained. Currently, fluidsim includes solvers for a variety of physical problems using different numerical methods (including finite-difference methods). However, this metapaper shall dwell only on the implementation and performance of its pseudo-spectral solvers, in particular the two- and three-dimensional Navier-Stokes solvers. We investigate the performance and scalability of fluidsim in a state of the art HPC cluster. Three similar pseudo-spectral CFD codes based on Python (Dedalus, SpectralDNS) and Fortran (NS3D) are presented and qualitatively and quantitatively compared to fluidsim. The source code is hosted at Bitbucket as a Mercurial repository bitbucket.org/fluiddyn/fluidsim and the documentation generated using Sphinx can be read online at fluidsim.readthedocs.io.

What Can We Learn from Many Labs Replications?

Several hundred research groups attempted replications of published effects in so-called Many Labs studies involving thousands of research participants. Given this enormous investment, it seems timely to assess what has been learned and what can be learned from this type of project. My evaluation addresses four questions: First, do these replication studies inform us about the replicability of social psychological research? Second, can replications detect fraud? Third, does the failure to replicate a finding indicate that the original result was wrong? Finally, do these replications help to support or disprove any social psychological theories? Although evidence of replication failures resulted in important methodological changes, the 2015 Open Science Collaboration findings sufficed to make the point. To assess the state of social psychology, we have to evaluate theories rather than randomly selected research findings.

The assessment of intrinsic credibility and a new argument for p < 0.005.

The concept of intrinsic credibility has been recently introduced to check the credibility of ‘out of the blue’ findings without any prior support. A significant result is deemed intrinsically credible if it is in conflict with a sceptical prior derived from the very same data that would make the effect just non-significant. In this paper, I propose to use Bayesian prior-predictive tail probabilities to assess intrinsic credibility. For the standard 5% significance level, this leads to a new p-value threshold that is remarkably close to the recently proposed p < 0.005 standard. I also introduce the credibility ratio, the ratio of the upper to the lower limit (or vice versa) of a confidence interval for a significant effect size. I show that the credibility ratio has to be smaller than 5.8 such that a significant finding is also intrinsically credible. Finally, a p-value for intrinsic credibility is proposed that is a simple function of the ordinary p-value and has a direct frequentist interpretation in terms of the probability of replicating an effect. An application to data from the Open Science Collaboration study on the reproducibility of psychological science suggests that intrinsic credibility of the original experiment is better suited to predict the success of a replication experiment than standard significance.

A Case For a Study Quality Appraisal in Survey Studies in Psychology.

The lack of replication of key effects in psychology has highlighted some fundamental problems with reporting of research findings and methods used (Asendorpf et al., 2013; Open Science Collaboration, 2015). Problems with replication have been attributed to sources of bias such as questionable research practices like HARK-ing (Kerr, 1998) or p-hacking (Simmons et al., 2011). Another potential source of bias is lack of precision in the conduct and methods used in psychological research, which likely introduces systematic error into data collected with the potential to affect results. A related issue is lack of accuracy in reporting study methods and findings. There is, therefore, increased recognition in the importance of transparency when reporting study outcomes to enable the scientific community to make fair, unbiased appraisals of the implications and worthiness of study findings. Lack of transparency hinders scientific progress as it may lead to erroneous conclusions regarding the implications of research findings, and may impede comparison and synthesis of findings across studies. As a result, researchers have become interested in research quality and the need for comprehensive, transparent reporting of findings (Asendorpf et al., 2013). This has resulted in calls for appropriate reporting standards and means to assess study quality (Cooper, 2011; Greenhalgh and Brown, 2017). In the present article we review the issue of study quality in psychology, and argue for valid and reliable means to assess study quality in psychology. Specifically, we contend that appropriate assessment checklists be developed for survey studies, given the prominence of surveys as a research method in the field.

The replication and reproducibility crises: origins and consequences for studies of ecology and evolution

Watch the VIDEO.There is a widespread discussion around a scientific crisis, resulting from a lack of reproducibility of published scientific studies. This was exemplified by Ioannidis’ 2005 paper “Why most published research findings are false” or the 2015 Open Science Collaboration study assessing reproducibility of psychological science. An often-cited reason for this reproducibility crisis is a fundamental misunderstanding of what statistical methods, and in particular P-values, can achieve. In the context of studies of ecology and evolution, I will show how 1) the pressure for publishing “novel” results, 2) what Gelman has called the “garden of forking paths”, i.e. the fact that published analyses represent only one out of many possible analyses, and 3) the often fruitless dichotomy between a null and alternative hypotheses, has led to the present situation. While scientific progress is dependent of major breakthroughs, we also need to find a better balance between confirmatory research – understanding how known effects vary in size according to the context – and exploratory, non-incremental research – finding new effects.

Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science.

In psychological science, there is an increasing concern regarding the reproducibility of scientific findings. For instance, Replication Project: Psychology (Open Science Collaboration, 2015) found that the proportion of successful replication in psychology was 41%. This proportion was calculated based on Cumming and Maillardet (2006) widely employed capture procedure (CPro) and capture percentage (CPer). Despite the popularity of CPro and CPer, we believe that using them may lead to an incorrect conclusion of (a) successful replication when the population effect sizes in the original and replicated studies are different; and (b) unsuccessful replication when the population effect sizes in the original and replicated studies are identical but their sample sizes are different. Our simulation results show that the performances of CPro and CPer become biased, such that researchers can easily make a wrong conclusion of successful/unsuccessful replication. Implications of these findings are considered in the conclusion.

Issues in Reproducible Simulation Research.

In recent years, serious concerns have arisen about reproducibility in science. Estimates of the cost of irreproducible preclinical studies range from 28 billion USD per year in the USA alone (Freedman et al. in PLoS Biol 13(6):e1002165, 2015) to over 200 billion USD per year worldwide (Chalmers and Glasziou in Lancet 374:86-89, 2009). The situation in the social sciences is not very different: Reproducibility in psychological research, for example, has been estimated to be below 50% as well (Open Science Collaboration in Science 349:6251, 2015). Less well studied is the issue of reproducibility of simulation research. A few replication studies of agent-based models, however, suggest the problem for computational modeling may be more severe than for laboratory experiments (Willensky and Rand in JASSS 10(4):2, 2007; Donkin et al. in Environ Model Softw 92:142-151, 2017; Bajracharya and Duboz in: Proceedings of the symposium on theory of modeling and simulation-DEVS integrative M&S symposium, pp 6-11, 2013). In this perspective, we discuss problems of reproducibility in agent-based simulations of life and social science problems, drawing on best practices research in computer science and in wet-lab experiment design and execution to suggest some ways to improve simulation research practice.

Pre-acceptance as a method to combat publication bias in area studies: a pilot in the Caucasus

Many social scientists in the readership of Caucasus Survey will have heard of the replication crisis (Open Science Collaboration 2015). This refers to the problem whereby results from seminal stud...