Unsuccessful Replication Research Articles

Generalized anxiety disorder and selective attention: An unsuccessful replication of Yiend et al., (2015) in a student population

ABSTRACT Generalized Anxiety Disorder (GAD) is an anxiety disorder that is believed to affect attention (Stein, M. B., & Sareen, J. (2015). Generalized anxiety disorder. New England Journal of Medicine, 373(21), 2059–2068. https://doi.org/10.1056/NEJMcp1502514; Yiend, J., Mathews, A., Burns, T., Dutton, K., Fernández-Martín, A., Georgiou, G. A., Luckie, M., Rose, A., Russo, R., & Fox, E. (2015). Mechanisms of selective attention in generalized anxiety disorder. Clinical Psychological Science, 3(5), 758–771. https://doi.org/10.1177/2167702614545216). Previous literature has found that selective attention is changed when someone perceives threatening stimuli, such as an angry face, and that those with anxiety disorders, may have a heightened or delayed response to threatening stimuli (Richards, H. J., Benson, V., Donnelly, N., & Hadwin, J. A. (2014). Exploring the function of selective attention and hypervigilance for threat in anxiety. Clinical Psychology Review, 34(1), 1–13. https://doi.org/10.1016/j.cpr.2013.10.006; Stevens, C., & Bavelier, D. (2012). The role of selective attention on academic foundations: A cognitive neuroscience perspective. Developmental Cognitive Neuroscience, 2, S30–S48. https://doi.org/10.1016/j.dcn.2011.11.001), which may alter how fast a presented task is completed (Yiend et al., 2015). The present study aimed to reproduce findings by Yiend et al. (2015), which identified an unexpected pattern in those with GAD: faster disengagement from angry faces compared to positive (happy, neutral) faces. The present study recruited a larger (nonclinical) sample from a student population to achieve greater statistical power. None of the findings reported by Yiend and colleagues (Experiment 1; 2015) were replicated in a student sample. The implications are discussed.

Evaluative contexts facilitate implicit mentalizing: relation to the broader autism phenotype and mental health

One promising account for autism is implicit mentalizing difficulties. However, this account and even the existence of implicit mentalizing have been challenged because the replication results are mixed. Those unsuccessful replications may be due to the task contexts not being sufficiently evaluative. Therefore, the current study developed a more evaluative paradigm by implementing a prompt question. This was assessed in 60 non-autistic adults and compared with a non-prompt version. Additionally, parents of autistic children are thought to show a genetic liability to autistic traits and cognition and often report mental health problems, but the broader autism phenotype (BAP) is an under-researched area. Thus, we also aimed to compare 33 BAP and 26 non-BAP mothers on mentalizing abilities, autistic traits, compensation and mental health. Our results revealed that more evaluative contexts can facilitate implicit mentalizing in BAP and non-BAP populations, and thus improve task reliability and replicability. Surprisingly, BAP mothers showed better implicit mentalizing but worse mental health than non-BAP mothers, which indicates the heterogeneity in the broader autism phenotype and the need to promote BAP mothers’ psychological resilience. The findings underscore the importance of contexts for implicit mentalizing and the need to profile mentalizing and mental health in BAP parents.

The effect of practice on automatic evaluation: A registered replication

A basic idea in cognitive science is that practicing a response can lead to the automatic activation of the response. Fazio, Sanbonmatsu, Powell, and Kardes (1986) tested that idea on the automatic activation of attitudes. In the experiment that Fazio et al. conducted, participants (N = 18) repeatedly categorized eight nouns as good/bad and eight nouns (the control words) as having one syllable or more. The measure of automatic activation of attitudes was evaluative priming: Participants categorized target adjectives as good/bad faster if their valence matched the valence of a prime noun that appeared before them. This priming effect was stronger for repeatedly evaluated words than for control words. Many have cited this article as evidence that practice automatizes evaluation, but the published research that followed focused on the evaluative priming effect, providing only one incidental and unsuccessful replication for the effect of practice on automatic evaluation. In light of the importance of this finding on the one hand, and the lack of a solid evidential basis for it on the other hand, we conducted three experiments that tested the effect of practice on evaluative priming effect. We attempted to directly replicate the original procedure in Experiments 1a and 1b (N = 108, 102, respectively), with Experiment 1b fixing an unintended prime-target contingency in Experiment 1a. Experiment 2 (N = 172) provided a conceptual replication with modified procedures. Practice in evaluation increased priming only in Experiment 1a. The inconsistent results prevent strong conclusions that practicing an evaluative response automates it, necessitating further research.

Determination of Telomere Length to Infer Forensic Age

Accurate age information is crucial for the identification of criminal cases in the field of forensics. Although the forensic age can be roughly estimated via bone age detection, there was no mature and reliable method for forensic age inference utilizing biological materials such as soft tissue fragments and no other markings left at the scene of the crime. Telomeres are cap-like structures found at the end of chromosomes in numerous tissues and cells and they are made up of repetitive sequences and binding proteins. This paper discussed the possibility of using telomere length to confirm the forensic age of a victim by concerning the biological characteristic of telomere and relevant techniques of length measuring. The structure of telomere enables it to prevent gene loss, fuse with other chromosome ends. Comparisons concentrating on the pros and cons of the existing measure of investigating the telomere are provided in this study by referring to those possible realistic scenes of the crime. As telomere length typically declines with every cell division as a result of the lagging strand's unsuccessful replication, the relationship between the shortening of telomere length and forensic are also shown for various types of cells. Further research in the field of investigating outliers in the pattern of shortening in telomere should be discovered and listed for various types of cells in order to make this method perfect and generalize it. This essay can provide some ideas for the study of telomere length inference of forensic age.

Manipulating Belief in Free Will and Its Downstream Consequences: A Meta-Analysis

Ever since some scientists and popular media put forward the idea that free will is an illusion, the question has risen what would happen if people stopped believing in free will. Psychological research has investigated this question by testing the consequences of experimentally weakening people’s free will beliefs. The results of these investigations have been mixed, with successful experiments and unsuccessful replications. This raises two fundamental questions: Can free will beliefs be manipulated, and do such manipulations have downstream consequences? In a meta-analysis including 145 experiments (95 unpublished), we show that exposing individuals to anti–free will manipulations decreases belief in free will and increases belief in determinism. However, we could not find evidence for downstream consequences. Our findings have important theoretical implications for research on free will beliefs and contribute to the discussion of whether reducing people’s belief in free will has societal consequences.

Replication: Unsuccessful replications and extensions of Temporal Value Asymmetry in monetary valuation and moral judgment

The phenomenon that contemplating future events elicits stronger emotions than contemplating past events has been coined “temporal value asymmetry” (TVA) (Caruso et al., 2008). We conducted very close replications of three experiments derived from two influential TVA papers: Studies 1 and 4 in Caruso et al. (2008), demonstrating TVA in monetary valuation, and Study 1 in Caruso (2010), demonstrating TVA in moral judgment. We also attempted to conceptually replicate whether TVA in monetary valuation would extend to moral judgments. We failed to find support for TVA in monetary valuation (Caruso et al., 2008). We also failed to find support for TVA in moral judgments (Caruso, 2010) and in our conceptual extension. Exploratory analyses excluding potential outliers and z-transforming the dependent variable were consistent with our preregistered analyses. We discuss potential explanations for our results and future directions for research about the effects of time on judgments of value and morality.

Demarcating the boundary conditions of memory reconsolidation: An unsuccessful replication

Disrupting memory reconsolidation provides an opportunity to abruptly reduce the behavioural expression of fear memories with long-lasting effects. The success of a reconsolidation intervention is, however, not guaranteed as it strongly depends on the destabilization of the memory. Identifying the necessary conditions to trigger destabilization remains one of the critical challenges in the field. We aimed to replicate a study from our lab, showing that the occurrence of a prediction error (PE) during reactivation is necessary but not sufficient for destabilization. We tested the effectiveness of a reactivation procedure consisting of a single PE, compared to two control groups receiving no or multiple PEs. All participants received propranolol immediately after reactivation and were tested for fear retention 24 h later. In contrast to the original results, we found no evidence for a reconsolidation effect in the single PE group, but a straightforward interpretation of these results is complicated by the lack of differential fear retention in the control groups. Our results corroborate other failed reconsolidation studies and exemplify the complexity of experimentally investigating this process in humans. Thorough investigation of the interaction between learning and memory reactivation is essential to understand the inconsistencies in the literature and to improve reconsolidation interventions.

Super-Size Me: An Unsuccessful Preregistered Replication of the Effect of Product Size on Status Signaling

Dubois, Rucker, and Galinsky (2012, Experiment 1) found that consumers view larger-size options as a signal of higher status. We conducted a close replication of this finding (N = 415), and observed a nonsignificant effect in the opposite direction (small vs. large product size: doriginal = 1.49, 95%CI [1.09, 1.89], dreplication = 0.09 95%CI [-0.15, 0.33]; medium vs. large: doriginal = 0.89 95%CI [0.52, 1.26], dreplication = 0.11 95%CI [-0.13, 0.34]; small vs. medium: doriginal = 0.62 95%CI [0.26, 0.98], dreplication = -0.01 95%CI [-0.25, 0.23]). We discuss potential reasons for this unsuccessful replication as well as implications for the status-signaling literature in consumer psychology.

Testing the skill-based approach: Consolidation strategy impacts attentional blink performance.

Humans can learn simple new tasks very quickly. This ability suggests that people can reuse previously learned procedural knowledge when it applies to a new context. We have proposed a modeling approach based on this idea and used it to create a model of the attentional blink (AB). The main idea of the skill-based approach is that models are not created from scratch but, instead, built up from reusable pieces of procedural knowledge (skills). This approach not only provides an explanation for the fast learning of simple tasks but also shows much promise to improve certain aspects of cognitive modeling (e.g., robustness and generalizability). We performed two experiments, in order to collect empirical support for the model's prediction that the AB will disappear when the two targets are consolidated as a single chunk. Firstly, we performed an unsuccessful replication of a study reporting that the AB disappears when participants are instructed to remember the targets as a syllable. However, a subsequent experiment using easily combinable stimuli supported the model's prediction and showed a strongly reduced AB in a large group of participants. This result suggests that it is possible to avoid the AB with the right consolidation strategy. The skill-based approach allowed relating this finding to a general cognitive process, thereby demonstrating that incorporating this approach can be very helpful to generalize the findings of cognitive models, which otherwise tends to be rather difficult.

Evidence for goal- and mixed evidence for false belief-based action prediction in 2- to 4-year-old children: A large-scale longitudinal anticipatory looking replication study.

Unsuccessful replication attempts of paradigms assessing children's implicit tracking of false beliefs have instigated the debate on whether or not children have an implicit understanding of false beliefs before the age of four. A novel multi-trial anticipatory looking false belief paradigm yielded evidence of implicit false belief reasoning in 3- to 4-year-old children using a combined score of two false belief conditions (Grosse Wiesmann, C., Friederici, A. D., Singer, T., & Steinbeis, N. [2017]. Developmental Science, 20(5), e12445). The present study is a large-scale replication attempt of this paradigm. The task was administered three times to the same sample of N=185 children at 2, 3, and 4years of age. Using the original stimuli, we did not replicate the original finding of above-chance belief-congruent looking in a combined score of two false belief conditions in either of the three age groups. Interestingly, the overall pattern of results was comparable to the original study. Post-hoc analyses revealed, however, that children performed above chance in one false belief condition (FB1) and below chance in the other false belief condition (FB2), thus yielding mixed evidence of children's false belief-based action predictions. Similar to the original study, participants' performance did not change with age and was not related to children's general language skills. This study demonstrates the importance of large-scaled replications and adds to the growing number of research questioning the validity and reliability of anticipatory looking false belief paradigms as a robust measure of children's implicit tracking of beliefs.

Context Dependency as a Predictor of Replicability

We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

Effect of tDCS Over the Right Inferior Parietal Lobule on Mind-Wandering Propensity.

Mind-wandering is associated with switching our attention to internally directed thoughts and is by definition an intrinsic, self-generated cognitive function. Interestingly, previous research showed that it may be possible to modulate its propensity externally, with transcranial direct current stimulation (tDCS) targeting different regions in the default mode and executive control networks (ECNs). However, these studies used highly heterogeneous montages (targeting the dorsolateral prefrontal cortex (DLPFC), the right inferior parietal lobule (IPL), or both concurrently), often showed contradicting results, and in many cases failed to replicate. Our study aimed to establish whether tDCS of the default mode network (DMN), via targeting the right IPL alone, could modulate mind-wandering propensity using a within-subjects double-blind, counterbalanced design. Participants completed sustained attention to response task (SART) interspersed with thought-probes to capture their subjective reports of mind-wandering before and after receiving anodal, cathodal, or sham tDCS over the right IPL (with the reference over the left cheek). We found evidence for the lack of an effect of stimulation on subjective reports of mind-wandering (JZS-BF01 = 5.19), as well as on performance on the SART task (errors (JZS-BF01 = 6.79) and reaction time (JZS-BF01 = 5.94). Overall, we failed to replicate previous reports of successful modulations of mind-wandering propensity with tDCS over the IPL, instead of providing evidence in support of the lack of an effect. This and other recent unsuccessful replications call into question whether it is indeed possible to externally modulate spontaneous or self-generated cognitive processes.

Effects of charismatic leadership and rewards on individual performance

We report two replication attempts for the positive main effects of charismatic leadership and performance-based rewards on individual performance from a field experiment in a charitable context (Antonakis, d'Adda, Weber, and Zehnder, 2015). Using video based treatments - instead of the live treatments in the original study - we only replicate the effect of performance-based rewards in a sample of 118 participants; we do not find an effect of charisma on performance. In a second study, we address the reasons that could explain the unsuccessful replication of the charisma effect by: (a) using a larger sample (n = 274) and (b) ensuring the experimental task context to have ingroup - cultural and hence value - fit between the workers and the beneficiaries of a charity. In the second study we fully replicate the positive effects of charisma as well as performance-based rewards on individual performance. In extending the original work we tested and found no interaction between economic incentives and charisma. Furthermore, using the manipulations as experimentally randomized instrumental variables (ERIVs, see Sajons, 2020), we find that the effect of charisma on individual performance is channelled through the vision dimension of leadership.

What is replication?

Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research. This definition reduces emphasis on operational characteristics of the study and increases emphasis on the interpretation of possible outcomes. The purpose of replication is to advance theory by confronting existing understanding with new evidence. Ironically, the value of replication may be strongest when existing understanding is weakest. Successful replication provides evidence of generalizability across the conditions that inevitably differ from the original study; Unsuccessful replication indicates that the reliability of the finding may be more constrained than recognized previously. Defining replication as a confrontation of current theoretical expectations clarifies its important, exciting, and generative role in scientific progress.

Limited Usefulness of Capture Procedure and Capture Percentage for Evaluating Reproducibility in Psychological Science.

In psychological science, there is an increasing concern regarding the reproducibility of scientific findings. For instance, Replication Project: Psychology (Open Science Collaboration, 2015) found that the proportion of successful replication in psychology was 41%. This proportion was calculated based on Cumming and Maillardet (2006) widely employed capture procedure (CPro) and capture percentage (CPer). Despite the popularity of CPro and CPer, we believe that using them may lead to an incorrect conclusion of (a) successful replication when the population effect sizes in the original and replicated studies are different; and (b) unsuccessful replication when the population effect sizes in the original and replicated studies are identical but their sample sizes are different. Our simulation results show that the performances of CPro and CPer become biased, such that researchers can easily make a wrong conclusion of successful/unsuccessful replication. Implications of these findings are considered in the conclusion.

Do Researchers Anchor Their Beliefs on the Outcome of an Initial Study?

As a research field expands, scientists have to update their knowledge and integrate the outcomes of a sequence of studies. However, such integrative judgments are generally known to fall victim to a primacy bias where people anchor their judgments on the initial information. In this preregistered study we tested the hypothesis that people anchor on the outcome of a small initial study, reducing the impact of a larger subsequent study that contradicts the initial result. Contrary to our expectation, undergraduates and academics displayed a recency bias, anchoring their judgment on the research outcome presented last. This recency bias is due to the fact that unsuccessful replications decreased trust in an effect more than did unsuccessful initial experiments. We recommend the time-reversal heuristic to account for temporal order effects during integration of research results.

Four Bad Habits of Modern Psychologists.

Four data sets from studies included in the Reproducibility Project were re-analyzed to demonstrate a number of flawed research practices (i.e., “bad habits”) of modern psychology. Three of the four studies were successfully replicated, but re-analysis showed that in one study most of the participants responded in a manner inconsistent with the researchers’ theoretical model. In the second study, the replicated effect was shown to be an experimental confound, and in the third study the replicated statistical effect was shown to be entirely trivial. The fourth study was an unsuccessful replication, yet re-analysis of the data showed that questioning the common assumptions of modern psychological measurement can lead to novel techniques of data analysis and potentially interesting findings missed by traditional methods of analysis. Considered together, these new analyses show that while it is true replication is a key feature of science, causal inference, modeling, and measurement are equally important and perhaps more fundamental to obtaining truly scientific knowledge of the natural world. It would therefore be prudent for psychologists to confront the limitations and flaws in their current analytical methods and research practices.

Selection bias, vote counting, and money-priming effects: A comment on Rohrer, Pashler, and Harris (2015) and Vohs (2015).

When a series of studies fails to replicate a well-documented effect, researchers might be tempted to use a “vote counting” approach to decide whether the effect is reliable—that is, simply comparing the number of successful and unsuccessful replications. Vohs’s (2015) response to the absence of money priming effects reported by Rohrer, Pashler, and Harris (2015) provides an example of this approach. Unfortunately, vote counting is a poor strategy to assess the reliability of psychological findings because it neglects the impact of selection bias and questionable research practices. In the present comment, we show that a range of meta-analytic tools indicate irregularities in the money priming literature discussed by Rohrer et al. and Vohs, which all point to the conclusion that these effects are distorted by selection bias, reporting biases, or p-hacking. This could help to explain why money-priming effects have proven unreliable in a number of direct replication attempts in which biases have been minimized through preregistration or transparent reporting. Our major conclusion is that the simple proportion of significant findings is a poor guide to the reliability of research and that preregistered replications are an essential means to assess the reliability of money-priming effects.

Considering Generality in the Systematic Review and Meta-Analysis of Single-Case Research: A Response to Hitchcock et al.

This paper responds to the commentary provided by Hitchcock et al. (J Behav Educ, 2015) which provides updated information on the procedures for considering the generality of an intervention using the What Works Clearinghouse (WWC) single-case guidelines. The new information provided by Hitchcock et al. is useful and provides insight into the processes and procedures used by the WWC to make practice and policy recommendations. Moreover, the additional information requires the results of the original Maggin et al. (J Behav Educ 23: 287–311, 2014) paper to be updated and the additional attention to these issues is applauded. The Hitchcock et al. commentary also provides an occasion to consider the concept of generality within single-case research and to consider the role systematic reviews can have in documenting the conditions under which a functional relation is and is not likely to be replicated. It is asserted that systematic reviews of single-case research should be used to catalog the universe of replication attempts for a specified functional relation in order to better understand the individuals and contexts under which results can be expected to generalize. This process requires careful examination of both successful and unsuccessful replication attempts and the consideration of descriptive- and functional-level variables. This response will, therefore, provide an overview of the concept of generality from a single-case standpoint, consider the criteria outlined in the Hitchcock et al. commentary from this perspective, and conclude with recommendations and encouragement to the field to address these issues in future systematic reviews of single-case research. The intent is to engender discussion and development in this area with the broader goal of expanding the influence of single-case research within the domains of practice, policy, and research.

Replicability of Social-Cognitive Priming: The Case of Weight as an Embodiment of Importance

Several influential research programs in cognition have suggested that even subtle changes in such perceptual events can have a powerful influence on socially meaningful judgments and behavior. Recent extensions of such social-cognitive studies have involved physical experiences, like effects of physical weight on people’s judgments of importance, effects first obtained by Jostmann and colleagues (2009) and later by Chandler and colleagues (2012). In recent years, the field of social cognitive priming in general has become doubted, because of several unsuccessful replications, publication bias, and flexibility in reporting. In this paper, the authors report one successful conceptual replication and one successful close replication. The authors discuss implications of replications and further emphasize the importance of replication for studying human behavior and thought.