Reliability Generalization Meta-analysis Research Articles

Reliability generalization meta-analysis of measuring Islamic work ethic

Reliability generalization meta-analysis of measuring Islamic work ethic

Reliability of Theory of Mind Tasks in Schizophrenia, ASD, and Nonclinical Populations: A Systematic Review and Reliability Generalization Meta-analysis.

Though theory of mind (ToM) is an important area of study for different disciplines, however, the psychometric evaluations of ToM tasks have yielded inconsistent results across studies and populations, raising the concerns about the accuracy, consistency, and generalizability of these tasks. This systematic review and meta-analysis examined the psychometric reliability of 27 distinct ToM tasks across 90 studies involving 2771 schizophrenia (SZ), 690 autism spectrum disorder (ASD), and 15,599 nonclinical populations (NC). Findings revealed that while all ToM tasks exhibited satisfactory internal consistency in ASD and SZ, about half of them were not satisfactory in NC, including the commonly used Reading the Mind in the Eye Test and Hinting Task. Other than that, Reading the Mind in the Eye Test showed acceptable reliability across populations, whereas Hinting Task had poor test-retest reliability. Notably, only Faux Pas Test and Movie for the Assessment of Social Cognition had satisfactory reliability across populations albeit limited numbers of studies. However, only ten studies examined the psychometric properties of ToM tasks in ASD adults, warranting additional evaluations. The study offered practical implications for selecting ToM tasks in research and clinical settings, and underscored the importance of having a robust psychometric reliability in ToM tasks across populations.

Internal Consistency of the DELTA Scale Assessing Psychotic-Like Experiences

Abstract: This study’s objective was to conduct a reliability generalization meta-analysis of the DELTA scale capturing Disintegration, i.e., proneness to psychotic-like experiences and behaviors. Eligible studies applied various versions of the DELTA scale; 573 α coefficients nested in 76 reports were identified. The pooled effect of all included α coefficients was calculated by applying a 4-level meta-analytic random-effects model. Test length, sample type, demographic sample characteristics (mean age, share of females), language, the proportion of reverse-keyed items, publication status, documentation quality of reliability estimates (reported vs. post hoc computed reliabilities), and DELTA score M and SDs were tested as moderators. The model yielded an average Cronbach’s α for DELTA and its facets of .86 (95% CI [.84, .88]), with medium and long DELTA versions reaching .96 (95% CI [.94, .96]) and .97 (95% CI [.95, .98]), respectively. Moderators explaining the heterogeneity among α coefficients were as follows: test length (expectedly, higher reliability estimates for longer tests), mean scores and SD of the DELTA scores (lower α estimates for larger means and lower SDs), and language (highest α estimates for the German language). Overall, the DELTA scales used in past research showed high-reliability estimates.

Systematic Review and Reliability Generalization Meta-Analysis of the Video Game Dependency Scale

Objective Video Game Dependency Scale (VGDS) evaluates degrees of online and offline gaming disorder. In this study, we examined the average reliability of VGDS scores and whether they provide values that justify their use, characterized the variation in reliability estimates across studies, and identified study characteristics that explain heterogeneity in reliability estimates. Method A reliability generalization meta-analysis based on the REGEMA guidelines was conducted. From the 19 selected articles that applied VGDS, 14 (16 independent samples) reported 17 reliability estimates (16 alpha coefficients and 1 test-retest reliability coefficient). A random-effects model was applied in the statistical analyses. Results The average reliability of the total VGDS scores assessed as internal consistency by Cronbach’s alpha coefficient was 0.925 (95% CI [.901,.942]). The results also found these coefficients to be significantly heterogeneous. Conclusions According to the psychometric theory, VGDS scores can be considered reliable for exploring gaming disorder in field research and clinical practice. Significance statement This study provides evidence for the score reliability of the VGDS, a relevant tool for assessing Internet gaming disorder. It highlights the importance of addressing score reliability and advancing measurement and assessment in practice and research fields.

Career Adapt-Abilities Scale: Reliability Generalization Meta-Analysis.

The concept of career adaptability emerged from the broader field of career development theory and has received increasing attention in the past two decades. To measure career adaptability, there are different scale development studies in the literature, but the most widely used and preferred one is the Career Adapt-abilities Scale developed by Savickas and Porfeli. Therefore, in the present study, the general reliability of the Career Adapt-abilities Scale was measured through meta-analysis. One hundred forty nine study (N = 82519) were included in the analyses. For the CAAS overall score, the average reliability coefficient among the 171 reliability estimations was high. However, reliability estimates in the studies included in the research show high heterogeneity. As a result of the moderator analysis, it was concluded that reliability estimates of these scores differ by item level, culture, language, category of items, different forms of CAAS, age, and SD. Regarding the application of the CAAS in research, the consequences of these findings are discussed in light of the relevant literature.

Frequency of suicidal ideation inventory: A reliability generalization meta-analysis

The objectives of this reliability generalization meta-analysis were to estimate the overall reliability of the Frequency of Suicidal Ideation Inventory (FSII), a short, easily administered test developed to assess suicidal ideation and to examine possible moderating variables. A systematic search was conducted in five databases from the year of publication (2016) of the FSII until March 2024. All studies that applied the FSII and reported a Cronbach’s alpha coefficient were included. Analyses were performed using a random effects model applying Bonett’s transformation. Our analysis of 32 studies (k = 39) yielded an overall alpha of .91 (95% CI = .90, .92). We found that the type of administration, population, and language of the FSII moderated its reliability, which is high overall and appropriate for use in both research and clinical settings.

A Meta-Analysis of the Reliability of Second Language Listening Tests (1991-2022).

To investigate the reliability of L2 listening tests and explore potential factors affecting the reliability, a reliability generalization (RG) meta-analysis was conducted in the present study. A total number of 122 alpha coefficients of L2 listening tests from 92 published articles were collected and submitted to a linear mixed effects RG analysis. The papers were coded based on a coding scheme consisting of 16 variables classified into three categories: study features, test features, and statistical results. The results showed an average reliability of 0.818 (95% CI: 0.803 to 0.833), with 40% of reliability estimates falling below the lower bound of CI. The presence of publication bias and heterogeneity was found in the reliability of L2 listening tests, indicating that low reliability coefficients were likely omitted from some published studies. In addition, two factors predicting the reliability of L2 listening tests were the number of items and test type (standardized and researcher- or teacher-designed tests). The study also found that reliability is not a moderator of the relationship between L2 listening scores and theoretically relevant constructs. Reliability induction was identified in reporting the reliability of L2 listening tests, too. Implications for researchers and teachers are discussed.

Screening for orthosomnia: a reliability generalization meta-analysis of the Anxiety and Preoccupation about Sleep Questionnaire (APSQ)

Screening for orthosomnia: a reliability generalization meta-analysis of the Anxiety and Preoccupation about Sleep Questionnaire (APSQ)

A comparison of univariate and meta-analytic structural equation modeling approaches to reliability generalization applied to the Maslach Burnout Inventory.

Reliability is a property of tests scores that varies from sample to sample. One way of generalizing reliability of a test is to perform a meta-analysis on some reliability estimator. In 2011, a reliability generalization meta-analysis on the Maslach Burnout Inventory (MBI) was conducted, concluding that average alpha values for the MBI dimensions ranged from 0.71 to 0.88. In the present study, we aimed to update the average reliability values of the MBI by conducting a literature search from 2010 until now and comparing to statistical procedures of meta-analysis: the Univariate approach, that were used in the previous study, and a novel meta-analytic approach based on structural equation modeling. An estimation of average reliability was done based on 69 independent primary reliability coefficients for the Univariate approach. The average reliability was based on 9 independent studies in the case of the Meta-analytic Structural Equation Modeling (MASEM) approach. Given that MASEM has the additional capability of testing the internal structure of a test, we also fitted several models. The data was well-suited to the bifactor model, revealing the dominance of the general factor over the domain-specific ones. Acceptable overall alpha and omega coefficients were achieved for the two of the MBI dimensions, having depersonalization reliability estimates below recommendations. In general, the MBI can be viewed as a highly interconnected three-factor scale, being its appropriate for research purposes.

Reliability generalization meta-analysis of orthorexia nervosa using the ORTO-11/12/15/R scale in all populations and language versions

BackgroundThe ORTO scale was developed in 2004 as a self-report questionnaire to assess symptoms of orthorexia nervosa (ON). ON is an unhealthy preoccupation with eating healthy food. The scale aims to measure obsessive attitudes and behaviors related to the selection, purchase, preparation, and consumption of pure, healthy food. Since its development, the ORTO-15 has been adapted into several shorter versions. The objective was to conduct a reliability generalization meta-analysis of the ORTO scale and its variant versions in all populations and languages.MethodsA systematic literature search was conducted to identify studies reporting the internal consistency of ORTO. Random-effect models were used to evaluate summary statistics of reliability coefficients, weighting the coefficients by the inverse variance using the restricted maximum likelihood method. The heterogeneity among the reliability coefficients was evaluated and assessed using numerous statistical metrics. The tau (τ), tau2 (τ2), I2, H2, R2, df, and the Q-statistic are among those obtained. Meta-regression analyses were used to examine moderators such as age and sex.ResultsTwenty-one studies (k = 21) involving 11,167 participants (n = 11,167) were analyzed. The overall effect estimate on internal consistency was 0.59 (95% CI 0.49–0.68), with a minimum reliability coefficient of 0.23 and a maximum reliability coefficient of 0.83. The heterogeneity statistics were found to have an I2 of 99.31%, which suggested high heterogeneity owing to a decrease in the confidence interval (95% CI) and an increase in variability. Sensitivity analysis revealed that a few studies strongly influenced the overall estimate. Egger’s test suggested possible publication bias. Neither age nor sex significantly moderated reliability via meta-regression.ConclusionsThe ORTO scale has a relatively low pooled reliability coefficient. Alternative ON assessment tools with enhanced psychometric properties are needed. Clinicians should not base diagnoses or treatment decisions on ORTO alone. Comprehensive psychiatric assessment is essential for accurate ON evaluation.

A Reliability Generalization Meta-Analysis of Scores on the Professional Quality of Life (ProQOL) Scale Across Sample Characteristics

The Professional Quality of Life Scale (ProQOL) has an extensive history of use that often relies on inductions of reliability from precedent literature. We completed a systematic of the literature and extracted sample-specific reliability estimates for ProQOL subscale scores. Random effects meta-analytic modeling was implemented to identify mean estimates of reliability. Secondary analyses were completed to depict generalization across samples and possible scores in future studies. Our analyses detected mean internal consistency estimates for the Compassion Satisfaction (.875), Secondary Traumatic Stress (.808), and Burnout (.754) similar to those reported in the ProQOL manual. However, secondary analyses revealed statistical variations associated with the professional identity and proportions of ethnicity represented within samples. ProQOL score reliability has remained fairly stable over time with scores for Compassion Satisfaction and Secondary Traumatic Stress supporting use in basic research, but not clinical decision making, the Burnout scale was associated with more questionable precision.

Reliability Generalization Meta-Analysis of the Child PTSD Symptom Scale for DSM-5 (CPSS-5)

This study aims to identify the mean alpha coefficients of the Child PTSD Symptom Scale (CPSS-5) and examine the moderator role of trauma type and study location. We included studies using the CPSS-5 and reporting a Cronbach’s alpha coefficient. Our analysis of 27 studies (k = 31) revealed a mean alpha coefficient of .91 for the CPSS-5. Mean internal consistency scores for trauma type ranged from .87 to .95. Additionally, studies conducted in North America yielded significantly higher mean alpha scores (M α = .94) than international studies (M α = .87). The CPSS-5 demonstrated reliable measurements across studies, ranging from good to excellent. Trauma type and location (or version of the CPSS-5) may be related to the reliability scores. Individual study characteristics should be considered, and reliability scores should be calculated for individual studies for the suitability of alpha scores for clinical decision-making.

Reliability Generalization Meta-Analysis of Seven Wisdom Self-Rating Scales From 2004 to 2023

Accurate measurement of wisdom is the cornerstone of wisdom research. To provide a representative reference for the reliability level and moderating factors of various wisdom self-rating scales, we carried out a reliability generalization meta-analysis of Chinese and English references retrieved from 2004 to 2023. A total of 149 articles were included, as well as seven wisdom self-rating scales. The results showed that all seven wisdom self-rating scales had good internal consistency reliability. The reliability of the original scale was higher than that of the simplified version, and the reliability of the total of the original scale was higher than those of each dimension. Future research should avoid “reliability introduction,” and rather focus on cross-cultural translation of the wisdom scale, selecting the most appropriate measurement tool for the particular field of wisdom, and strengthen theoretical interpretations by connotation and structure of wisdom itself.

Internal consistency reliability of the Revised Illness Perceptions Questionnaire: A systematic review and reliability generalization meta-analysis.

The Revised Illness Perception Questionnaire (IPQ-R) assesses patients' perspectives of their illnesses. Original psychometric testing occurred in limited populations. The purpose of this reliability generalization meta-analysis was to: (1) estimate internal consistency reliability of each IPQ-R subscale, and (2) test moderators of these estimates. Web of Science was searched in July 2022 for articles citing the original IPQ-R paper that reported IPQ-R reliability data. Cronbach's alphas (⍺) were pooled for each IPQ-R subscale using inverse variance weighting and DerSimonian and Laird estimation. Sixty-six studies met criteria. Overall pooled ⍺ estimates were acceptable: 0.71-0.87. Treatment control reliability was reduced among cardiac (⍺ = 0.68), diabetes/kidney disease (⍺ = 0.63), and mixed/other (⍺ = 0.66) samples; cyclical reliability was reduced in cancer (⍺ = 0.65) samples. Age, gender, and race were also significant moderators. Subscale reliability varied based on sample characteristics. Adapting IPQ-R subscales to account for sample variation could improve measurement of illness perception constructs.

Reliability generalization meta-analysis: comparing different statistical methods

AbstractReliability generalization (RG) is a kind of meta-analysis that aims to characterize how reliability varies from one test application to the next. A wide variety of statistical methods have typically been applied in RG meta-analyses, regarding statistical model (ordinary least squares, fixed-effect, random effects, varying-coefficient models), weighting scheme (inverse variance, sample size, not weighting), and transformation method (raw, Fisher’s Z, Hakstian and Whalen’s and Bonett’s transformation) of reliability coefficients. This variety of methods compromise the comparability of RG meta-analyses results and their reproducibility. With the purpose of examining the influence of the different statistical methods applied, a methodological review was conducted on 138 published RG meta-analyses of psychological tests, amounting to a total of 4,350 internal consistency coefficients. Among all combinations of procedures that made theoretical sense, we compared thirteen strategies for calculating the average coefficient, eighteen for calculating the confidence intervals of the average coefficient and calculated the heterogeneity indices for the different transformations of the coefficients. Our findings showed that transformation methods of the reliability coefficients improved the normality adjustment of the coefficient distribution. Regarding the average reliability coefficient and the width of confidence intervals, clear differences among methods were found. The largest discrepancies were found between the different strategies for calculating confidence intervals. Our findings point towards the need for the meta-analyst to justify the statistical model assumed, as well as the transformation method of the reliability coefficients and the weighting scheme.

Reliability Generalization Meta-Analysis of the Brief Self-Control Scale (BSCS): Reliability Evidence across Age Groups and Languages

Objective: Brief Self-Control Scale (BSCS) is the most widely used instrument to assess self-control. The purpose of this reliability generalization meta-analysis was to examine the degree to which consistency reliability coefficients for scores on the BSCS generalize across age groups and languages.Method: We included studies using the BSCS and reporting a Cronbach’s alpha coefficient score since January 2018.Results: Our analysis (j = 180; k = 211) yielded an overall Cronbach’s alpha value of .81 for the unidimensional 13-item BSCS. Moderator analysis showed that reliability scores ranged from .71 to .81 across age groups and from .77 to .90 across languages.Conclusions: Though all internal consistency scores fell in either a good or acceptable range, researchers can consider using a child-specific instrument to assess self-control among children. Reliability scores were also higher in Western areas (e.g., North America) than in non-Western countries.

Instruments of Child-to-Parent Violence: Systematic Review and Meta-Analysis.

This systematic review and reliability generalization meta-analysis synthesized psychometric literature on instrumentation assessing child-to-parent violence published through September 2023 across four databases. In the screening, we identified studies reporting Cronbach's alpha internal consistency estimates for the child-to-parent violence scales. The eligible reliability coefficients ranged from 0.610 to 0.930, mostly exceeding the minimum threshold of 0.700. Random-effects models calculated pooled Cronbach's alphas separately for global, father-specific, and mother-specific subscales. The results demonstrated cumulative values of 0.83 (global: standard error = 0.0129), 0.800 (fathers: standard error = 0.0203), and 0.81 (mothers: standard error = 0.0179), denoting largely adequate reliability. However, significant between-study heterogeneity was observed. While the mean alpha levels seem acceptable for most tools, substantial variability coupled with the possibility of some studies violating reliability assumptions indicates that a conservative interpretation is warranted. Ongoing scale refinement and additional psychometric evaluations will strengthen the rigor methodology in this developing research domain. However, these results should be interpreted with caution, as there is a high level of heterogeneity, and it is possible that some studies have not verified the assumptions underlying Cronbach's alpha.

A Reliability Generalization Meta-Analysis of the Child and Adolescent Mindfulness Measure (CAMM).

We carried out the present meta-analysis study to secure the reliability generalization of the Child and Adolescent Mindfulness Scale (CAMM) based on its reliability findings in different studies. The study only considered full-text papers in English that were indexed in Web of Science, PubMed, ProQuest, and Scopus and presented a Cronbach's alpha value of the measurements with the CAMM (209 alpha values out of 155 papers). Using the random effects model and the Bonnet transformation, the relevant analyses yielded the transformed Cronbach's alpha value of the CAMM to be 0.80 (CI = 0.79-0.81). The present meta-analysis study concluded that the CAMM is rather reliable to be utilized in child and adolescent populations. Besides, moderator analyses to scrutinize the alpha values yielded significant differences in favor of the origin continent of the research and the CAMM-short version (25-item/10-item). Yet, it was not the case with the publication type and the adaptation status of the scale. In a nutshell, future research may confidently employ the 10-item short form of the CAMM to assess mindfulness among children and adolescents.

Reliability generalization meta-analysis of the Prodromal Questionnaire-Brief (PQ-B): Implications for Asian psychiatry.

Reliability generalization meta-analysis of the Prodromal Questionnaire-Brief (PQ-B): Implications for Asian psychiatry.

Supportive Care Needs Survey: A reliability generalization meta-analysis.

The purpose of this study is to investigate the reliability generalization of 2 forms of the Supportive Care Needs Survey (SCNS), the questionnaires commonly used to assess the unmet needs of cancer patients. Reviewed articles were retrieved through databases including PubMed, Ovid, Embase, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Web of Science, Scopus, and ProQuest. The inclusion criteria were quantitative studies that assessed the unmet needs of cancer patients using the SCNS and presented reliability coefficients with sample size. Two independent reviewers examined the studies according to inclusion criteria and quality. The final studies included in the meta-analysis were determined by consensus. A random effects model was adopted for the analysis. To estimate reliability coefficients, the alpha coefficients for each study were transformed into the Z statistic for normalization and back to alpha. The values were weighted by the inverse of the studies' variance. The Higgins I2 statistic was used to test for heterogeneity, and the Egger's test and funnel plot were performed to evaluate publication bias. Out of 12,522 studies, 26 studies were included in the meta-analysis. The overall mean weighted effect size of the SCNS long-form (LF) was 0.90 and the subdomains ranged from 0.90 to 0.97. The overall alpha for the SCNS short-form (SF) was 0.92, and the alphas for the subdomains were between 0.81 and 0.92. The estimated reliability coefficients in both LF and SF were highest in psychological and health information needs and lowest in sexuality. No publication bias was indicated in this study. In this study, the overall reliability of SCNS was presented and the factors affecting the reliability of SCNS were identified. The results of this study may help clinicians or researchers make decisions about selecting tools to measure unmet needs of cancer patients.