False-Positive Psychology

In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

<span class="s1">The data includes measures collected for the two experiments reported in “False-Positive Psychology” [1] where listening to a randomly assigned song made people feel younger (Study 1) or actually be younger (Study 2). These data are useful because they illustrate inflations of false positive rates due to flexibility in data collection, analysis, and reporting of results. Data are useful for educational purposes.</span>

rd graders and wants to determine whether the new method is more effective than the standard approach to teaching math in this grade. He assigns the students to be taught math the standard way or to be taught using his new method. After a pre-determined period of time all students take the same math competency test, and the researcher conducts statistical tests to compare math scores between the two groups of students. The null hypothesis is that there is no difference in the effectiveness of the teaching methods (test scores equal across two test groups), whereas the experimental (or alternative) hypothesis is that the new teaching method is more effective than the standard method (test scores will be higher for the students taught using the new compared to the standard method). The researcher looks at the test scores in the two groups and applies a statistical test that provides the probability of getting the results in the current sample given the null hypothesis is true. The researcher then makes a decision regarding the effectiveness of his new method relative to the standard teaching method, taking into consideration information about the methods and sample, as well as the results of the statistical test(s) used. The researcher subsequently attempts to communicate this decision to the broader academic community via a manuscript submitted for publication, contingent on the evaluation of the research by a few peers and a journal editor. In this type of research process, at least two types of errors can be made regarding the decision the researcher makes after considering all of the evidence. These errors are known as type I and type II errors: Type I error : deciding to reject the null hypothesis when in fact it is correct (deciding the new teaching method is better than the standard method, when in fact it is not better ).

There has been an abundance of foreign studies on queer leadership in schools, but significantly fewer in the country. This study was initiated for this very purpose: to have a vivid description of who the Filipino gay instructional leader is as embodied in his/her work ecology model. Through the use of grounded theory, the inclusion criteria were employed to ensure a thorough exploration of the experiences of seven (7) gay instructional leaders in leading their units. The leaders were selected using theoretical sampling to develop categories that explain the specific phenomenon and form a rich and nuanced theory. Throughout the process, the researcher considered theoretical sensitivity, purposeful sampling techniques, constant comparison, theoretical saturation, and flexibility in data collection and analysis. Queer leadership, as was grounded in the data collected, meant being challenged by the culture of heteronormative leadership in the country, which includes (but is not limited to) encountering pressure to conform to traditional gender roles, navigating toxic work environments, and facing gender stereotyping and biased treatment. Further, Filipino gay instructional leaders are distinguished in their PRIDE qualities-- Passion, Resilience, Inclusivity, Diversified Role as a Leader, and Empowering Others. These qualities help them to serve and lead schools effectively. The "Going Beyond the Margin" work ecology model, created using these same grounded qualities of a Filipino gay leader, is centered on inclusivity and symbolizes the progress made by these leaders toward achieving complete acceptance and acknowledgment of what they can contribute. However, there remain some individual and systemic issues in leadership that the Filipino gay leader continues to traverse.

A Web-based Dynamic User Customized Entry System for Public Health Surveillance

1. IntroductionCase studies are a critical and even attractive research methodology 1 (Bartunek, Rynes & Ireland, 2006; Eisenhardt & Graebner, 2007; Sato, 2009). 2 However, there are sometimes questions about whether the case study method is rigorous enough as a research methodology.In addressing this question, Eisenhardt (1989) is one of the most frequently cited studies on the methodological case study fundamentals in theory construction. Yet most studies that cite Eisenhardt's method emphasize only generalizability. As a result, the potential for theory construction using the case study method has not been sufficiently investigated.2. Case Study Method in Eisenhardt (1989)According to Eisenhardt (1989), the case study is a research strategy which focuses on understanding the dynamics present within single settings. Eisenhardt (1989) focuses on the case study as a research methodology for theory construction in particular. According to Eisenhardt, theory construction consists of nine steps.a) Getting startedWhen conducting a case study, the first thing to do is to formulate the research question. This enables one to specify what kind of organization the survey should study or what data the survey should collect.b) Selecting casesThe next problem is case selection. In a case study that has the goal of theory construction, sample selection usually depends on theoretical sampling.In statistical sampling, the sample is selected randomly from a population. Theoretical sampling on the other hand reiterates previous cases and uses what amounts to an extreme example that reproduces an existing case, builds upon the theory, or corresponds to a theoretical category.c) Crafting instruments and protocolsResearch on theory construction is able to combine multiple data sources. In particular, it combines quantitative and qualitative data. The case study can therefore use both types of data.d) Entering the fieldsAn especially notable characteristic of the case study in theory construction is the overlap of data collection and data analysis.Overlap of data collection and data analysis leads to flexibility in data collection. It enables the researcher to change the data collection methodology in the course of the research when the researchers comes up with a new idea that may generate a new theory.e) Analyzing within-case dataFirst, independent analysis of various cases enables the researcher to avoid becoming overwhelmed by too much data.In addition, by analyzing individual cases before comparing the various cases, the researcher is able to detect the particular patterns of the various cases, thereby deepening his or her understanding of the cases in point and facilitating cross-case comparisons.f) Searching for cross-case patternsBias can occur in cross-case comparisons, which could lead to erroneous conclusions. There are three ways to prevent bias.The first is to separate the cases into categories and look at the similarities within each group and the differences between the groups. The second is to divide the cases into pairs for comparison, then look for differences between the cases that are similar and for similarities between the cases that appear to be different. The third is to divide the data according to the data source and try to gain insight into the particulars of the heterogeneous data.g) Shaping hypothesisThe first step in hypothesis development is clarification of constructs. This step consists of two processes: elaborating on the constructs' definitions and establishing proofs for measuring these constructs. These processes constitute a hypothesis-testing type of research, in that they are similar to formulating a construct from several metrics. However, a case study for theory construction differs in that it is the analysis itself that generates the constructs and their measurement, and that there is no way to group several metrics into one construct. …

Practitioners in language and literacy education who are interested researching and furthering social justice issues often seek critical approaches to their work. Ultimately, the goal of such research under favorable conditions is to produce action for change, however, under difficult circumstances that approaches need to be adjusted. Today, practitioners may find resistance to critical work from neoliberal education policies, to which such work is undesirable in that it may harm profits, or from restrictive administrative regimes. First, this chapter gives an historical overview of critical approaches to educational research, and then discusses in detail an approach the authors call critical practitioner research. Drawing on a case study of research done at private language institute in South Korea, the authors highlight several issues which prohibited the use of other previously defined critical approaches to research, including restrictive school policies, and working with young learners. Ethical and reporting issues of such research are also discussed. Out of this work, the authors propose three broad principles for critical practitioner research: a focus on social justice issues, flexibility in data collection and exploration, and collaboration with participants to the extent it is possible while recognizing that fully participatory approaches may not always be possible.

RISC programmable controllers can bring programmability down to low-end remote sites

False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.

This article reframes the potential for partnerships between university researchers and counseling practitioners that yield more applied counseling research. The gap in practitioner research and its resultant fracturing of the counseling field are well documented in the literature. Guiding principles of these partnerships are offered as a method of grounding the relationship based on a case study example between a university researcher and a local counseling training organization. The four primary values that can guide this reframing of practitioner research are the emphasis on practitioner expertise, development of research identity for practitioners and future practitioners, flexibility in data collection and analysis, and establishment of trust between researchers and clinicians. These values address the barriers to practitioner research that are currently explored in the literature. Application of these values led to a successful partnership yielding ample data collection with plans to produce academic articles and new training programs. Exploring the potential for effective practitioner research has the potential to produce more applied research that is currently lacking in the field of counseling.

Most map building methods employed by mobile robots are based on the assumption that an estimate of robot poses can be obtained from odometry readings or from observing landmarks or other robots. In this paper we propose methods to build a global geometric map by integrating scans collected by laser range scanners without using any knowledge about the robots' poses. We consider scans that are collections of line segments. Our approach increases the flexibility in data collection, since robots do not need to see each other during mapping, and data can be collected by multiple robots or a single robot in one or multiple sessions. Experimental results show the effectiveness of our approach in different types of indoor environments.

The principles and practice of a methodology of cell cycle analysis that allows the estimation of the absolute length (in units of time) of all cell cycle stages (G1, S, and G2) are detailed herein. This methodology utilizes flow cytometry to take full advantage of the excellent stoichiometric properties of click chemistry. This allows detection, via azide-fluorochrome coupling, of the modified deoxynucleoside 5-ethynyl-2'-deoxyuridine (EDU) incorporated into replicated DNA through incremental pulsing times. This methodology, which we designated as EdU-Coupled Fluorescence Intensity (E-CFI) analysis, can be applied to cell types with very distinct cell cycle features, and has shown excellent agreement with established techniques of cell cycle analysis. Useful modifications to the original protocol (Pereira et al., Oncotarget, 8:40514-40,532, 2017) have been introduced to increase flexibility in data collection and facilitate data analysis.

The recent discovery of methodological flaws in experimental design and analysis in neuroscience research has raised concerns over the validity of certain techniques used in routine analyses and their corresponding findings. Such concerns have centered around selection bias whereby data is inadvertently manipulated such that the resulting analysis produces falsely increased statistical significance, i.e. type I errors. This has been illustrated recently in flv1RI studies, with excessive flexibility in data collection, and general experimental design issues. Current work from our group has shown how this problem extends to generic voxel-based analysis (and certain technique derivatives such as tract- based spatial statistics) using fractional anisotropy images derived from diffusion tensor imaging. In this work, we demonstrate how this circularity principle can potentially extend to the well-known optimized voxel-based morphometry technique for assessing cortical density differences whereby the principal cause of experimental corruption is due to normalization strategy. Specifically, the popular sum­ of-squared-differences (SSD) metric explicitly optimizes statistical findings potentially inflating type I errors. Additional experimentation demonstrates that this problem is not restricted to the SSD metric but extends to other commonly used metrics such as mutual information, neighborhood cross correlation, and Demons.

In this report, we illustrate the considerable impact of researcher degrees of freedom with respect to exclusion of participants in paradigms with a learning element. We illustrate this empirically through case examples from human fear conditioning research, in which the exclusion of 'non-learners' and 'non-responders' is common - despite a lack of consensus on how to define these groups. We illustrate the substantial heterogeneity in exclusion criteria identified in a systematic literature search and highlight the potential problems and pitfalls of different definitions through case examples based on re-analyses of existing data sets. On the basis of these studies, we propose a consensus on evidence-based rather than idiosyncratic criteria, including clear guidelines on reporting details. Taken together, we illustrate how flexibility in data collection and analysis can be avoided, which will benefit the robustness and replicability of research findings and can be expected to be applicable to other fields of research that involve a learning element.

The field of psychological science is in a pandemonium. With failures to replicate well-established effects, evidence for a skewed picture of science in the published literature, and a media hype about the replication crisis – what is left for us to believe in these days? Luckily, researchers have done what they do best – research – to try to establish the causes, and possible solutions to this replication crisis. A coherent picture has emerged. There are three key factors that seem to have led to the replication crisis: (1) Underpowered studies, (2) publication bias, and (3) questionable research practices. Studies in psychology often test a small number of participants. As effects tend to be small and measures noisy, larger samples are required to reliably detect an effect. An underpowered study, trying to find a small effect with a small sample sizes, runs a high probability of not finding an effect, even if it is real (Button et al., 2013; Cohen, 1962; Gelman &amp; Weakliem, 2009). By itself, this would not be a problem, because a series of underpowered studies can be, in principle, combined in a meta-analysis to provide a more precise effect size estimate. However, there is also publication bias, as journals tend to prefer publishing articles which show positive results. Authors often do not even bother trying to submit papers with non-significant results, leading to a file-drawer problem (Rosenthal, 1979). As the majority of research papers are underpowered, the studies that do show a significant effect capture the outliers of a normal distribution around a true effect size (Ioannidis, 2005; Schmidt, 1992, 1996). This creates a biased literature: even if an effect is small or non-existent, a number of published studies can provide apparently consistent evidence for a large effect size. The problems of low power and publication bias are further exacerbated by questionable research practices, where researchers – often unaware that they are doing something wrong – use little tricks to get their effects above a significance threshold, such as removing outliers until the threshold is reached, or including post-hoc covariates (John, Loewenstein, &amp; Prelec, 2012; Simmons, Nelson, &amp; Simonsohn, 2011). As a lot of research and discussion exists on how to fix the problem of publication bias and questionable research practices – which mostly require a top-down change of the incentive structure. Here, I focus on the issue of underpowered studies, as this can be addressed by individual researchers. Increasing power is in everyone’s best interests: It strengthens science, but it also gives the researcher a better chance to provide a meaningful answer to their question of interest. On the surface, the solution to the problem of underpowered studies is very simple: we just have to run bigger studies. The simplicity is probably why this issue is not discussed very much. However, the solution is only simple if you have the resources to increase your sample sizes. Running participants takes time and money. Therefore, this simple solution poses another problem: the possibility of creating a Matthew effect, where the rich get richer by producing large quantities of high-quality research, while researchers with fewer resources can produce either very few good studies, or numerous underpowered experiments for which they will get little recognition*. On the surface, the key to avoiding the Matthew effect is also simple: if the rich collaborate with the poor, even researchers with few resources can produce high-powered studies. However, in practice, there are few perceived incentives for the rich to reach out to the poor. There are also practical obstacles for the poor in approaching the rich. These issues can be addressed, and it takes very little effort from an average researcher to do so. Below, I describe why it is important to promote collaborations in order to improve replicability in social sciences, and how this could be achieved. Why? In order to ensure the feasibility of creating a large-scale collaboration network, it would be necessary to promote the incentives for reaching out to the poor. Collecting data for someone with fewer resources may seem like charity. However, I argue that it is a win-win situation. Receivers are likely to reciprocate. If they cannot collect a large amount of data for you, perhaps they can help you in other ways. For example, they could provide advice on a project with which you got stuck and which you had abandoned years ago; they could score that data that you never got around to having a look at; or simply discuss new ideas, which could give you a fresh insight into your topic. If they collect even a small amount of data, this could improve a dataset. In the case of international collaborations, you would be able to recruit a culturally diverse sample. This would ensure that our view of psychological processes is generalisable beyond a specific population (Henrich, Heine, &amp; Norenzayan, 2010). How? There are numerous ways in which researchers can reach out to each other. Perhaps one could create an online platform for this purpose. Here, anyone can write an entry for their study, which can be at any stage: it could be just an idea, or a quasi-finished project which just needs some additional analyses or tweaks before publication. Anyone can browse a list of proposed projects by topic, and contact the author if they find something interesting. The two researchers can then discuss further arrangements: whether together, they can execute this project, whether the input of the latter will be sufficient for co-authorship, or whether the former will be able to reciprocate by helping out with another project. Similarly, if someone is conducting a large-scale study, and if they have time to spare in the experimental sessions, they could announce this in a complementary forum. They would provide a brief description of their participants, and offer to attach another task or two for anyone interested in studying this population. To reach a wider audience, we could rely on social media. Perhaps a hashtag could be used on twitter. Perhaps #LOOC (“LOOking for Collaborator”)**? One could tweet: “Testing 100 children, 6-10 yo. Could include another task up to 15 minutes. #LOOC”. Or: “Need more participants for a study on statistical learning and dyslexia. #LOOC”, and attach a screen shot or link with more information. In summary, increasing sample sizes would break one of the three pillars of the replication crisis: large studies are more informative than underpowered studies, as they lead to less noisy and more precise effect size estimates. This can be achieved through collaboration, though only if researchers with resources are prepared to take on some amount of additional work by offering to help others out. While this may be perceived as a sacrifice, in the long run it should be beneficial for all parties. It will both become easier to diversify one’s sample, and help researchers who study small, specific populations (e.g., a rare disorder), to collaborate with others to recruit enough participants to draw meaningful conclusions. It will provide a possibility to connect with researchers from all over the world with similar interests and possibly complementary expertise. And in addition, it will lead to an average increase in sample sizes, and reported effects which can be replicated across labs. References Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., &amp; Munafo, M. R. (2013). Confidence and precision increase with high statistical power. Nature Reviews Neuroscience, 14(8). doi:10.1038/nrn3475-c4 Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65(3), 145-153. Gelman, A., &amp; Weakliem, D. (2009). Of beauty, sex and power: Too little attention has been paid to the statistical challenges in estimating small effects. American Scientist, 97(4), 310-316. Henrich, J., Heine, S. J., &amp; Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61-83. Ioannidis, J. P. A. (2005). Why most published research findings are false. Plos Medicine, 2(8), 696-701. doi:10.1371/journal.pmed.0020124 John, L. K., Loewenstein, G., &amp; Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 0956797611430953. Rosenthal, R. (1979). The "File Drawer Problem" and Tolerance for Null Results. Psychological Bulletin, 86(3), 638-641. Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173. Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1(2), 115-129. doi:10.1037//1082-989x.1.2.115 Simmons, J. P., Nelson, L. D., &amp; Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 0956797611417632. Footnotes * One may or may not consider this a problem – after all, the issue of the replicability crisis is solved. ** Urban dictionary tells me that “looc” means “Lame. Stupid. Wack. The opposite of cool. (Pronounced the same as Luke.)”