Retrospective Design Analysis Research Articles

Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education

Randomized controlled trials have proliferated in education, in part because they provide an unbiased estimator for the causal impact of interventions. It is increasingly recognized that many such trials in education have low power to detect an effect if indeed there is one. However, it is less well known that low powered trials tend to systematically exaggerate effect sizes among the subset of interventions that show promising results ( We conduct a retrospective design analysis to quantify this bias across 22 such promising trials, finding that the estimated effect sizes are exaggerated by an average of 52% or more. Promising trial bias can be reduced ex-ante by increasing the power of the trials that are commissioned and guarded against ex-post by including estimates of the exaggeration ratio when reporting trial findings. Our results also suggest that challenges around implementation fidelity are not the only reason that apparently successful interventions often fail to subsequently scale up. Instead, the effect from the initial promising trial may simply be exaggerated.

Never to be missed again - an analysis of 55 consecutive cases of traumatic diaphragmatic hernia

BACKGROUND: This report reviews our clinical experience with 55 cases of traumatic diaphragmatic hernia at a Brazilian university hospital. Traumatic diaphragmatic hernia is an uncommon injury and presents diagnostic and therapeutic challenges. Occasionally, it is missed in trauma patients and is usually associated with significant morbidity and mortality. This analysis aimed to improve the diagnostic outcomes of trauma patient care. METHODS: Retrospective design analysis of database records of trauma patients at HC-Unicamp were performed to investigate incidence, trauma mechanism, diagnosis, herniated organs, associated injuries, trauma score, morbidity and mortality. RESULTS: Fifty-five patients were analysed. Blunt trauma was two-fold more frequent than penetrating trauma and was associated with high-grade injury; motor vehicle collision was the most common mechanism. Left-sided hernia was four-fold more frequent than that on the right side, although critical injuries were more frequently associated with the right side. The stomach was the most herniated organ in both trauma mechanisms. Preoperative diagnosis was mostly performed using chest radiography (55%). Postoperative diagnosis was mostly performed via laparotomy rather than laparoscopy. Associated injuries were observed in 43 patients (78%) and the mortality rate was 20% for both the sides CONCLUSION: Isolated injuries are rare, and the presence of associated injuries increases morbidity and mortality. Chest radiography in the trauma bay is useful as an initial examination, although it is not suitable for use as a definitive method. Despite the use of laparoscopy in a few cases, laparotomy is the most common approach.

Designing Studies and Evaluating Research Results: Type M and Type S Errors for Pearson Correlation Coefficient

It is widely appreciated that many studies in psychological science suffer from low statistical power. One of the consequences of analyzing underpowered studies with thresholds of statistical significance is a high risk of finding exaggerated effect size estimates, in the right or the wrong direction. These inferential risks can be directly quantified in terms of Type M (magnitude) error and Type S (sign) error, which directly communicate the consequences of design choices on effect size estimation. Given a study design, Type M error is the factor by which a statistically significant effect is on average exaggerated. Type S error is the probability to find a statistically significant result in the opposite direction to the plausible one. Ideally, these errors should be considered during a prospective design analysis in the design phase of a study to determine the appropriate sample size. However, they can also be considered when evaluating studies’ results in a retrospective design analysis. In the present contribution, we aim to facilitate the considerations of these errors in the research practice in psychology. For this reason, we illustrate how to consider Type M and Type S errors in a design analysis using one of the most common effect size measures in psychology: Pearson correlation coefficient. We provide various examples and make the R functions freely available to enable researchers to perform design analysis for their research projects.

New approach to injection of pressurizing gas into fuel tanks of power units

Purpose. Determination a rational way to injection of high-temperature pressurizing gas into fuel tanks of large elongation. Determination of longitudinal overload effect on the Archimedes force during the gas jet penetration in the tank. Reducing the need for pressurizing gas, the mass of the storage system. Methodology. A retrospective design analysis of devices for injecting the gas into tanks and taxonomy basics are used. With their help, it is possible to determine the causes of a wide variety of device designs for injecting gas into tanks and the common fundamental disadvantages of all known devices. Findings. As a result of the research carried out, a new method for supplying hot gas to the tanks has been found and substantiated. It is suitable for most conditions and provides a reduction in the need for pressurizing gas, does not reduce the operating fuel reserves, shows the trends for further research. Originality. The main reason for the differences between the results of ground tests and flight tests in terms of the gas parameters in the tank and the temperature of its upper bottom has been determined. This is overload effect on the increase in the buoyancy force on hot pressurizing gas jet, which is injected traditionally from the upper tank bottom to the side of the lower tank bottom. In this case, the buoyancy force acts against the dynamic component, reduces the jet range and presses the hot gas to the upper bottom. A new method for injecting the hot pressurizing gas, devoid of the indicated drawback, has been proposed and developed by using a theory of similarity. This makes it possible to mix the gas in the free volume of the tank as much as possible due to the action of the Archimedes force, to equalize gas temperature, reducing the maximum temperature at the upper bottom, and noticeable mass transfer processes in the tank are excluded. Practical value. The application of the proposed method permits defining correctly and accurately the gas flow rate for tank pressurization, using it with a temperature of up to ~1800 K. The drop in gas pressure disappears in the tank at the initial moment of operation of the pressurization system, caused by the injection of a hot gas jet into the fuel surface. Depending on the conditions, the pressurizing gas requirement can be reduced by up to 50%. In this case, the main fuel reserves in the tank are not reduced.

Enhancing Statistical Inference in Psychological Research via Prospective and Retrospective Design Analysis.

In the past two decades, psychological science has experienced an unprecedented replicability crisis, which has uncovered several issues. Among others, the use and misuse of statistical inference plays a key role in this crisis. Indeed, statistical inference is too often viewed as an isolated procedure limited to the analysis of data that have already been collected. Instead, statistical reasoning is necessary both at the planning stage and when interpreting the results of a research project. Based on these considerations, we build on and further develop an idea proposed by Gelman and Carlin (2014) termed “prospective and retrospective design analysis.” Rather than focusing only on the statistical significance of a result and on the classical control of type I and type II errors, a comprehensive design analysis involves reasoning about what can be considered a plausible effect size. Furthermore, it introduces two relevant inferential risks: the exaggeration ratio or Type M error (i.e., the predictable average overestimation of an effect that emerges as statistically significant) and the sign error or Type S error (i.e., the risk that a statistically significant effect is estimated in the wrong direction). Another important aspect of design analysis is that it can be usefully carried out both in the planning phase of a study and for the evaluation of studies that have already been conducted, thus increasing researchers' awareness during all phases of a research project. To illustrate the benefits of a design analysis to the widest possible audience, we use a familiar example in psychology where the researcher is interested in analyzing the differences between two independent groups considering Cohen's d as an effect size measure. We examine the case in which the plausible effect size is formalized as a single value, and we propose a method in which uncertainty concerning the magnitude of the effect is formalized via probability distributions. Through several examples and an application to a real case study, we show that, even though a design analysis requires significant effort, it has the potential to contribute to planning more robust and replicable studies. Finally, future developments in the Bayesian framework are discussed.