Interpreting Effect Sizes Research Articles

What is the optimal approach to analyse ventilator-free days? A simulation study

BackgroundVentilator-free days (VFDs) are a composite outcome in critical care research, reflecting both survival and mechanical ventilation duration. However, analysis methods for VFDs are inconsistent, with some focusing on counts and others on time-to-event outcomes, while other approaches such as the multistate model and the win ratio have emerged. We aimed to evaluate various statistical models through simulations to identify the optimal approach for analysing VFDs.MethodsFirst, 16 datasets of 300 individuals were simulated, comparing a control group to an intervention with varying survival rates and ventilation durations. Various statistical models were evaluated for statistical power and Type I error rate. Four clinical trial datasets (LIVE study, NCT02149589; ARMA study, NCT00000579; ACURASYS study, NCT00299650; COVIDICUS study, NCT04344730) were then used to apply the same statistical models to analyse VFDs. Twelve statistical methods were evaluated, including count-based, time-to-event approaches, and the win-ratio. Additionally, sensitivity analyses were conducted.ResultsMost statistical methods effectively controlled Type I error rate, except for the zero-inflated and hurdle Poisson/negative binomial count submodels, as well as the cause-specific Cox regression model for death. The power to detect survival benefit and ventilation duration effects varied, with time-to-event approaches, the Mann–Whitney test, the proportional odds model and the win ratio generally performing best. Similar results were observed in sensitivity analyses. In the real datasets, the multistate model, the Mann–Whitney test, the proportional odds model and the win ratio generally showed a significant association between VFDs and randomisation groups.ConclusionsThe multistate model could be recommended as the optimal approach for analysing VFDs, as it outperformed the other methods and offers a more interpretable effect size than the proportional odds model and the win ratio.

Exploring the dynamic impact patterns of R&D on firm capability: asymmetric impact analysis

Purpose The purpose of this study is to explore the complex impact of research and development (R&D) on firm performance, as represented by firm capabilities. Specifically, it focuses on predictive impact analysis and delineates the versatile impact patterns of R&D, contingent on resource conditions and the level of capabilities. Design/methodology/approach This study presents an innovative three-stage data envelopment (DEA)-Tobit regression (Tobit)-multi-layer perceptron neural network (MLP) approach. The integrated analytic process includes an evaluation of firm capabilities (DEA), explanatory analysis (Tobit) and predictive pattern analysis (MLP). Findings This study finds that the effect of R&D on firm capabilities is significant, in general, with a quadratic effect (U-shaped). However, the delineated impact analysis reveals that the effect size varies, contingent on firm-specific conditions and the divergent impact patterns showing that one-size-fits-all solutions are not applicable. Practical implications The study’s methodological advancement provides industry managers with a meaningful decision-making aid. By understanding these complex impact patterns, managers can take advantage of what-if scenario testing for resource deployment and pursue feasible options best suited to firm-specific conditions. Originality/value By proposing a cascading analytic process, this study presents an innovative empirical approach in that it delves into the complex impact mechanism of R&D on capabilities as a holistic performance measure. Unlike prior efforts, this study focuses on capturing asymmetric impact patterns and predicting the interpretable effect size, which holds significant pragmatic value.

How to interpret correlational process-outcome effect sizes in psychotherapy: a meta-analytic benchmark study

ABSTRACT Objective This study aims at developing empirically driven criteria for correlational effect size interpretations grounded on the actual psychotherapy process-outcome effect size distributions. Method We performed a meta-analysis on PubMed and PsycINFO databases searching for meta-analyses reporting correlational process-outcome associations. Results Forty-three meta-analyses met inclusion criteria, reporting 1,637 effect sizes from 859 unique studies. A four-level meta-analytic model resulted in an estimated mean effect size of r = .24 (z = .24, 95% CI[.22,.27]), significantly different from Cohen’s proposed value for moderate effects (i.e.,.30), z = −.04, SE = 0.01, t(1628) = −4.17, p < .001). Percentiles derived from the models showed that Cohen’s criteria were too conservative, with the 25th percentile = .12, 50th percentile = .26, and 75th percentile = .39. Based on these findings, we suggest the benchmarks .10, .25, and .40, for small, moderate, and large effect sizes. Even when using these less restrictive criteria, the majority of the correlation analyses from primary studies (81.8%) were underpowered to identify at least a moderate effect size. Conclusion The current findings might help to enhance effect size interpretations and power calculations in psychotherapy process-outcome research. Further replications are necessary to extend these benchmarks to other areas of clinical psychology.

The Role of Statistical Power: A Study of Relationship Between Emotional and Conduct Problems, Sociodemographic Factors, and Smoking Behaviours in Large and Small Samples of Latvian Adolescents.

Background and Objectives: Adolescent smoking is influenced by sociodemographic and psychological factors, including emotional and conduct problems. Understanding how sample size impacts the interpretation of these associations is critical for improving study design and public health interventions. This study examines the relationships between smoking behaviours, sociodemographic factors, and emotional and conduct problems, focusing on how sample size affects statistical significance and effect size interpretation. Materials and Methods: Data from the Latvian Health Behaviour in School-aged Children study was analysed. Chi-square tests and logistic regression were conducted to evaluate associations between smoking behaviours, sociodemographic factors, and emotional and conduct problems. Analyses were performed on both a large general sample and ten smaller generated subsamples to compare the impact of sample size on statistical outcomes. Results: Age and conduct problems emerged as the most consistent predictors of adolescent smoking behaviours across large and small samples, while other predictors, such as family affluence and family structure, showed weaker and less consistent associations. A large sample produced significant results even for weak predictors. Conclusions: This study highlights the importance of integrating effect size interpretation with statistical significance, particularly in large datasets, to avoid overstating findings. By leveraging real-world data, it provides practical recommendations for improving study design and interpretation in behavioural, medical, and public health research, contributing to more effective interventions targeting adolescent smoking.

Defining Effect Size Standards in Temporomandibular Joint and Masticatory Muscle Research.

BACKGROUND The interpretation of effect sizes is critical in scientific research, particularly in health and medicine, as it helps assess the strength and significance of experimental results. However, standardized guidelines for interpreting effect sizes in temporomandibular joint (TMJ) and masticatory muscle research are lacking. This study aims to propose new guidelines for interpreting group differences in research on the TMJ and masticatory muscles. MATERIAL AND METHODS The study is a bibliometric analysis based on meta-analyses published in the top 20 ranked dental journals. The study included 16 meta-analyses, comprising a total of 456 studies. In these records, 26,662 participants were analyzed (12,102 in the first group and 14,560 in the second group). Effect size metrics, including Cohen's d, and Hedges' g, were analyzed. The primary outcomes were the 25th, 50th, and 75th percentiles of effect size measures for TMJ and masticatory muscle studies. Data were stratified by effect size metrics (Cohen's d, and Hedges' g). Statistical analyses were conducted using R programming to compute the percentiles of effect sizes. RESULTS For group differences, the values of Hedges' g corresponding to the 25th, 50th, and 75th percentiles were 0.11, 0.34, and 0.74, respectively. CONCLUSIONS Researchers studying the TMJ and masticatory muscles are encouraged to adopt the following thresholds: 0.1 for small effects, 0.3 for medium effects, and 0.7 for large effects, for both Cohen's d and Hedges' g. These guidelines provide a standardized approach to effect size interpretation, enhancing the reliability and relevance of findings in TMJ research.

Effect Size Interpretation in Structural Equation Models

Structural equation modeling (SEM) involves many complex statistical issues, but the ultimate purpose is to obtain parameter estimates that answer research questions about the associations among variables. These estimates provide key effect-size information, making it critical to report and interpret them well. Although there are many resources on effect-size reporting, none focus on SEM. Furthermore, reporting standards for SEM neglect the connection between parameter estimates and effect sizes and provide little interpretational guidance. Thus, this paper provides an overview of effect-size reporting and interpretation within SEM. After defining the effect-size concept broadly, we explain how effect sizes are represented in three common types of SEM: path analysis, confirmatory factor analysis, and structural regression models. Then, based on a brief literature review, we discuss the need for higher-quality effect-size interpretation in studies reporting SEM results.

A meta-analysis of the impact of TOE adoption on smart agriculture SMEs performance.

Agricultural SMEs face distinct challenges due to factors such as weather, climate change, and commodity price changes. Technology has become essential in helping SMEs overcome these challenges and grow their businesses. The relationship between technology and SMEs in the agriculture sector covers various aspects, such as using hardware and software, digital applications, sensors, and e-commerce strategies to be examined in further depth through literature study. The implementation of the TOE (technology, organization, and environment) framework in smart agriculture faces several challenges. To overcome these challenges, an integrated approach is needed that involves technological capacity building, organizational management changes, and adequate policy and infrastructure support to help SMEs in the agricultural sector develop their businesses. This research aims to demonstrate and identify how TOE plays an important role in the performance of SMEs, particularly with regard to agriculture in order to improve agricultural productivity, efficiency, and sustainability while enabling access to broader markets in several countries. This study employs a meta-analysis method using a quantitative approach taken by each publication, which typically used SEM. PRISMA technique was used to examine evidence from clinical trials, and clinical significance was determined using the GRADE approach. Statistical analysis was performed using the Fisher test to combine the results of several studies and Cohen's approach to interpreting effect sizes. The results of this study are in line with the findings of 27 previous studies which showed a direct positive relationship between TOE construction and the performance of agricultural SMEs, with variables including technological factors, organizational factors, environmental factors, and SME performance. The synergy between technology adoption by agricultural SMEs and Industry 4.0 can increase connectivity and automation in the agricultural sector. However, it is important to remember that adopting TOE to realize the smart agriculture concept has its own challenges and risks, such as resource management (technology), good organizational management (organization), and internal and external organizational environments (environments), including intense competition. TOE adoption improves access to information about competitors and customers, providing practitioners and decision-makers with a clearer understanding. It enables focus on factors with a significant impact on TOE adoption, so that they are more independent in developing effective business concepts that are adaptive to the era of agricultural technology 4.0.

Effectiveness of Robot Care Intervention and Maintenance for People with Dementia: A Systematic Review and Meta-Analysis.

Robots have the potential to improve the quality of life of people with dementia. This study examined the effectiveness of robot care intervention and maintenance effect for people with dementia. Meta-analytical procedures were used to identify and synthesize articles for analysis. Coding procedures were used to record the moderators, including robot type, outcomes, intervention length, intervention duration, and intervention frequency. Hedge's g statistic was employed to interpret effect sizes and quantify individual research findings. The literature review identified 20 eligible randomized controlled trials. Meta-analysis results indicated an overall small effect of g = 0.286 for robot care intervention and g = 0.279 for robot care maintenance. Outcomes for robot care intervention indicated a small and significant effect size at g >0.2, whereas the Bomy robot type had an insignificant effect size. Outcomes for robot care maintenance showed a medium and significant effect size. This study confirmed the intervention effect of robot care on people with dementia and its sustainability for neuropsychiatric and social health outcomes. This highlights the effectiveness of humanoid-type robots in dementia care.

Empirical Benchmarks for Effect Size Interpretation and Study Planning with Social and Behavioral Outcomes

As reporting of effect sizes in evaluation studies has proliferated, researchers and consumers of research need tools for interpreting or benchmarking the magnitude of those effect sizes that are relevant to the intervention, target population, and outcome measure being considered. Similarly, researchers planning education studies with social and behavioral outcomes need sources of appropriate effect size values for conducting power analysis. In this paper, we use a large meta-analysis to provide estimates of distributions of effect sizes that may be helpful for researchers and consumers of research to assist in interpreting the magnitude of effect sizes. These distributions can also be used to select effect sizes for power analysis for school-based randomized trials of interventions with social and behavioral outcomes. We introduce the Effect Size (ES) Contextualizer application (https://ebcontextualizer.shinyapps.io/EmpBench/), which summarizes the effect sizes in the meta-analysis and allows researchers to use the database to do their own benchmarking or study planning.

Inter‐ and intra‐subject variability of quantitative EEG biosignatures and their effect on interpretation of normalized effect size

Abstract BackgroundQuantitative EEG measures can be used as biosignatures of disease conditions. As such, the effect of interventions/treatments can be studied by longitudinal analysis of changes in these measures. The consistency of these measures can be assessed by test‐retest reliability scores such as intra‐class correlation coefficient (ICC) that depends on intra‐ and inter‐subject variability. The magnitude of an effect can be described by a normalized effect size (i.e. normalizing the effect with respect to the sample variability). However, the inherent variability of EEG and its effect on interpretation of effect size is less explored. The aim of this work is to investigate inter‐ and intra‐subject variability of PSD‐based resting‐state EEG measures and their effect on analysis of change scores.MethodsWe collected 20‐channel longitudinal EEG data (initial visit and 1‐year follow‐up) from healthy volunteers (n=61, ages 40‐82). After artifact decontamination, we computed power spectral density (PSD) at 1‐40 Hz frequency bins. At each frequency, we computed inter‐ and intra‐subject variability of PSD as well as ICC. We simulated a hypothetical effect by adding a constant value to the 2nd visit’s data equivalent to half the standard deviation of the sample. We computed the normalized effect size (Cohen’s d) using both the pooled variance as well the variability in the change score. We compared these effect sizes for different frequencies.ResultsOverall, absolute PSD exhibited high test‐retest reliability in the frequencies 2‐20 Hz (ICC&gt;0.7) with 7‐9 Hz having the highest ICC (ICC&gt;0.94). Inter‐subject variability was highest at 8Hz, and intra‐subject variability was lowest at 13‐15 Hz frequencies. For frequencies between 5 – 10 Hz (fast‐Theta/slow‐Alpha bands), the difference between the two types of normalized effect size ranged between 0.3 to 0.6.ConclusionsInter‐ and intra‐subject variability of PSD‐based EEG measures depend on frequency. As such, changes in PSD due to potential interventions should be interpreted with respect to the inherent variability of PSDs. In longitudinal analysis of paired data, normalized effect size should be reported using both population variability (inter‐subject) and change‐score variability (intra‐subject).

Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions

SummaryBackgroundDichotomisation of statistical significance, rather than interpretation of effect sizes supported by confidence intervals, is a long-standing problem.MethodsWe distributed an online survey to clinical trial statisticians across the UK, Australia and Canada asking about their experiences, perspectives and practices with respect to interpretation of statistical findings from randomised trials. We report a descriptive analysis of the closed-ended questions and a thematic analysis of the open-ended questions.ResultsWe obtained 101 responses across a broad range of career stages (24% professors; 51% senior lecturers; 22% junior statisticians) and areas of work (28% early phase trials; 44% drug trials; 38% health service trials). The majority (93%) believed that statistical findings should be interpreted by considering (minimal) clinical importance of treatment effects, but many (61%) said quantifying clinically important effect sizes was difficult, and fewer (54%) followed this approach in practice.Thematic analysis identified several barriers to forming a consensus on the statistical interpretation of the study findings, including: the dynamics within teams, lack of knowledge or difficulties in communicating that knowledge, as well as external pressures. External pressures included the pressure to publish definitive findings and statistical review which can sometimes be unhelpful but can at times be a saving grace. However, the concept of the minimally important difference was identified as a particularly poorly defined, even nebulous, construct which lies at the heart of much disagreement and confusion in the field.ConclusionThe majority of participating statisticians believed that it is important to interpret statistical findings based on the clinically important effect size, but report this is difficult to operationalise. Reaching a consensus on the interpretation of a study is a social process involving disparate members of the research team along with editors and reviewers, as well as patients who likely have a role in the elicitation of minimally important differences.

Association of interleukin-6 with sarcopenia and its components in older adults: a systematic review and meta-analysis of cross-sectional studies

Background Observational studies have documented increased serum IL-6 levels in elderly individuals afflicted with sarcopenia. Nevertheless, the relationship between serum IL-6 concentrations and sarcopenia prevalence in the aging population is yet to be defined. Methods We executed a systematic review and meta-analysis of cross-sectional studies that scrutinized serum IL-6 levels in older adults with and without sarcopenia. Relevant studies were sourced from PubMed, Scopus, Embase, Cochrane Library, and Web of Science from inception until 10 September 2023. The standard mean differences (SMDs) in serum IL-6 levels between studies were synthesized using a random-effects model. To examine the influence of demographic and clinical factors on these outcomes, we performed subgroup analyses and meta-regression, focusing on variables such as sex, age, and body mass index (BMI). We also assessed the relationship between serum IL-6 levels and the defining components of sarcopenia: muscle mass, muscle strength, and physical performance. We used Fisher’s Z transformation to standardize the interpretation of effect sizes from these relationships. The transformed values were then converted to summary correlation coefficients (r) for a clear and unified summary of the results. Results We included twenty-one cross-sectional studies involving 3,902 participants. Meta-analysis revealed significantly elevated serum IL-6 levels in older adults with sarcopenia compared with those without sarcopenia (SMD = 0.31; 95% CI 0.18, 0.44). The difference was highly pronounced in the subgroups of male and those with female percentage below 50% or a mean BMI below 24 kg/m2. Serum IL-6 levels were inversely correlated with muscle mass (summary r = −0.18; 95% CI −0.30, −0.06), but not with handgrip strength (summary r = −0.10; 95%CI: −0.25, 0.05) or gait speed (summary r = −0.09; 95%CI: −0.24, 0.07). Conclusions This meta-analysis establishes a link between increased serum IL-6 levels and sarcopenia in the elderly, particularly in relation to decreased muscle mass.

Better together against genetic heterogeneity: A sex-combined joint main and interaction analysis of 290 quantitative traits in the UK Biobank.

Genetic effects can be sex-specific, particularly for traits such as testosterone, a sex hormone. While sex-stratified analysis provides easily interpretable sex-specific effect size estimates, the presence of sex-differences in SNP effect implies a SNP×sex interaction. This suggests the usage of the often overlooked joint test, testing for an SNP's main and SNP×sex interaction effects simultaneously. Notably, even without individual-level data, the joint test statistic can be derived from sex-stratified summary statistics through an omnibus meta-analysis. Utilizing the available sex-stratified summary statistics of the UK Biobank, we performed such omnibus meta-analyses for 290 quantitative traits. Results revealed that this approach is robust to genetic effect heterogeneity and can outperform the traditional sex-stratified or sex-combined main effect-only tests. Therefore, we advocate using the omnibus meta-analysis that captures both the main and interaction effects. Subsequent sex-stratified analysis should be conducted for sex-specific effect size estimation and interpretation.

A Meta-Analytic Review of Job Embeddedness and Turnover Intention: Evidence from South-East Asia

This meta-analytic study examines the relationship between job embeddedness and turnover intention in Thailand and Indonesia. Through the analysis of 22 independent samples consisting of 8,196 participants, the aim is to determine the universal applicability of job embeddedness as a predictor of employee turnover. Using the Hunter and Schmidt method, ensuring methodological rigor and accuracy in the estimation and interpretation of effect sizes, the study finds a significant negative association between job embeddedness and turnover intention ( r = −.44). The dimensions of fit and sacrifice are both negatively correlated with employees’ decision to quit ( r = −.27 for both), while the dimension of link does not exhibit such a relationship. Organizational embeddedness and community embeddedness are both negatively associated with turnover intention, with organizational embeddedness being a stronger predictor. The moderating effect of gender on the relationship between job embeddedness and turnover intention is not supported. The study concludes by discussing the theoretical and practical implications of the findings, highlighting the need for further empirical research on job embeddedness and turnover intention in the Southeast Asian context.

DOP02 Developing explicit thresholds for outcomes to inform (GRADE) Evidence to Decision frameworks for Inflammatory Bowel Disease guidelines

Abstract Background Evidence to decision (EtD) frameworks are a key recent development in Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) methodology. They aim to ensure that decisions about healthcare interventions are based on a thorough, transparent and unbiased assessment of the available evidence. Core to this is to agree the importance of outcomes to be used, the specific measures for each outcome that have the most consensus and finally to develop explicit thresholds for interpretation of effect sizes of outcomes to inform (GRADE) evidence to decision frameworks. This ensures decisions are made in an unbiased fashion and allows stakeholders to focus on discussions based on their significant experience and expertise, rather than interpretation of evidence. Therefore, our study aims to develop these thresholds for the EtD framework to establish their use in several key IBD guidelines. Methods This online survey using Survey Monkey was conducted by inviting experts and stakeholders from the United Kingdom and British Society of Gastroenterology. Each expert was asked to select important clinically relevant outcomes and were asked about what magnitude of the effect they consider large, moderate, small, or trivial for each of the clinical, endoscopic, radiological, biochemical, histological outcomes and adverse events. Questions were framed as neutral statements without introducing a specific direction. Response options included sliding bars going from 1 to 100% for the desirable magnitude of effect for each outcome (Figure 1). The mean average of responses of experts were calculated. Results A total of 89 clinical experts/stakeholders participated in this online survey. Clinical remission, clinical response, endoscopic remission, withdrawal due to adverse events, and serious adverse events were considered critical outcomes. Trivial to small, small to moderate and moderate to large thresholds were as follows - clinical remission (10%, 20% and 31%), clinical response (13 %, 24% and 36%), endoscopic remission (9%, 17% and 28%), withdrawal due to AE (7 %, 14% and 23%), serious AE (6%, 11% and 17%) (Table 1). Conclusion This is the first study to develop thresholds for outcomes in IBD which can be used in EtD for the development of IBD guidelines. These thresholds have been used in the development of upcoming British Society of Gastroenterology guidelines for the management of IBD.

Evidence-Based Practice Competencies among Nutrition Professionals and Students: A Systematic Review

Evidence-Based Practice Competencies among Nutrition Professionals and Students: A Systematic Review

Redefining effect size interpretations for psychotherapy RCTs in depression

IntroductionEffect sizes are often used to interpret the magnitude of a result and in power calculations when planning research studies. However, as effect size interpretations are context-dependent, Jacob Cohen's suggested guidelines for what represents a small, medium, and large effect are unlikely to be suitable for a diverse range of research populations and interventions. Our objective here is to determine empirically-derived effect size thresholds associated with psychotherapy randomized controlled trials (RCTs) in depression by calculating the effect size distribution. Material and methodsWe extracted effect sizes of 366 RCTs provided by the systematic review of Cuijpers and colleagues (2020) on psychotherapy for depressive disorders across all age-groups. The 50th percentile effect size, as this represents a medium effect size, and the 25th (small) and 75th (large) percentile effect sizes were calculated to determine empirically-derived effect size thresholds. ResultsAfter adjusting for publication bias, 0.27, 0.53, and 0.86 represent small, medium, and large effect sizes, respectively, in psychotherapy for depressive disorders. DiscussionThe effect size distribution for psychotherapy treatment in depression indicates that observed effect size thresholds are larger than Cohen's suggested effect size thresholds (0.2, 0.5, and 0.8). These results have implications for the interpretation of study effects and the planning of future studies via power analyses, which often use effect size thresholds.

How to Interpret Effect Sizes for Biopsychosocial Outcomes and Implications for Current Research

How to Interpret Effect Sizes for Biopsychosocial Outcomes and Implications for Current Research

Vitality and Application of Effect Size for Quality Research

This review article highlights the importance of effect size in research. P-value alone is not sufficient to determine the practical significance of a study, making effect size an essential component in hypothesis testing. Choosing the appropriate effect size for a specific study design can be challenging, and its interpretation may require modifications and personal judgement. Therefore, researchers should exercise caution when reporting and interpreting effect size, as it provides valuable information about the practical significance of their study, complementing hypothesis testing results. In conclusion, effect size should not be overlooked and should be carefully chosen and interpreted to ensure the validity and reliability of research results

How to analyze continuous and discrete repeated measures in small-sample cross-over trials?

To optimize the use of data from a small number of subjects in rare disease trials, an at first sight advantageous design is the repeated measures cross-over design. However, it is unclear how these within-treatment period and within-subject clustered data are best analyzed in small-sample trials. In a real-data simulation study based upon a recent epidermolysis bullosa simplex trial using this design, we compare non-parametric marginal models, generalized pairwise comparison models, GEE-type models and parametric model averaging for both repeated binary and count data. The recommendation of which methodology to use in rare disease trials with a repeated measures cross-over design depends on the type of outcome and the number of time points the treatment has an effect on. The non-parametric marginal model testing the treatment-time-interaction effect is suitable for detecting between group differences in the shapes of the longitudinal profiles. For binary outcomes with the treatment effect on a single time point, the parametric model averaging method is recommended, while in the other cases the unmatched generalized pairwise comparison methodology is recommended. Both provide an easily interpretable effect size measure, and do not require exclusion of periods or subjects due toincompleteness.