Higher-order Confirmatory Factor Analysis Research Articles

Validation of the Arabic version of the cyberchondria severity scale 12 items (CSS-12-Ar) among a sample of Lebanese adults

BackgroundTo the best of our knowledge, the Cyberchondria Severity Scale-12 (CSS-12) has not been translated into Arabic; therefore, our objective was to assess the psychometric properties of the Arabic version of the CSS (CSS-12-Ar) among a sample of Lebanese adults.MethodsParticipants were enrolled in January 2021. A confirmatory factor analysis (CFA) was carried out using the MPlus software v.7.2, reporting several goodness-of-fit indicators: Relative Chi-square (χ2/df), Root Mean Square Error of Approximation (RMSEA), Comparative Fit Index (CFI) and Tucker Lewis Index (TLI). To evaluate measurement invariance across gender, we conducted higher-order multiple group confirmatory analysis using lavaan software.Results449 participants enrolled in this study (mean age: 24.34 ± 8.22 years, 70.6% females). Since the correlations between the four-factor model were very high (r > 0.8), we ran the higher-order CFA in which all first-order latent variables were loading a general factor. The analyzed model was well-fitted to the data χ2(50) = 173.34; p < 0.001; CFI = 0.926; RMSEA = 0.074 [0.062, 0.086]. The Cronbach’s alpha values were good for the total score (0.92), as well as for excessiveness (0.80), distress (0.77), reassurance (0.81) and compulsion (0.76). The results provided evidence of full scalar invariance across gender. The comparison of latent mean scores revealed no significant differences across gender, in either the cyberchondria total score or its facets. The CSS-12 score was positively associated with anxiety (r = 0.10; p = 0.003) (convergent validity), OCD (r = 0.11; p = 0.016) and stress (r = 0.35; p < 0.001) (concurrent validity).ConclusionThe CSS-12-Ar was deemed a suitable scale to measure the severity of cyberchondria among Lebanese university students. We hope that researchers and clinicians can benefit now from this scale.

Developing Successful Commercialisation Model Through Technology Transfer Assessing its Validity and Reliability by Using Structural Equation Modelling

The aim of this research is to develop and validate a suitable structural model for commercialization of successful innovations/ideas/methodology of R&amp;D organizations through technology transfer. The survey data was extracted from 231 responses out of 310 target participants against the structured questionnaire containing 60 questions designed based on the literature survey. The factor structure and preliminary model are evaluated through Exploratory Factor Analysis. The first order and higher order Confirmatory Factor Analysis(CFA) models are evolved by estimating coefficients, covariance matrix, variances and performance indices. Structural Equation Modelling (SEM) is employed to develop and assess the validity and reliability of the structural model. The results implied that the developed structural model has achieved the required level of metrics and it can be implemented in to R&amp;D organizations and industries for successful commercialization.

사이버 언어폭력 피해 척도 개발 및 타당화 연구

The purpose of this study was to define a sub-concept of complex posttraumatic stress disorder (CPTSD) symptoms caused by cyber verbal bullying and to develop and validate the Cyber Verbal Bullying Distress Scale (CVBDS) which can measures the degree of the symptoms. For this purpose, the concept was set based on the literature review of CPTSD, PTSD, cyberbullying and interviews of 15 people who access cyber verbal bullying data and a total of 111 preliminary questions were selected. By verifying the facial and content validity, 71 questions were selected. 201 participants were collected to select the final items, and exploratory factor analysis, confirmatory factor analysis, higher-order confirmatory factor analysis was conducted in a row. As a result, a total of 30 items were extracted with the structure of 2-7 factors (PTSD: Intrusion, Arousal &amp; Reactivity, Avoidance; DSO: Negative affect, Negative cognitions about the world and the others, Negative cognitions about the self, Disturbances in relationship). To find if cyber verbal bullying data had affected CPTSD symptoms, we conducted paired sample t-test using pre-post measurements and revealed that cyber verbal bullying data had influence on CPTSD symptoms. Through this study, it was proposed that the CVBDS can be used to measure CPTSD symptoms caused by cyber verbal bullying data as a reliable and valid measurement. This study could be provided as an important foundation for the study exploring symptoms and factor structures of CPTSD caused by cyber verbal bullying.

The Japanese version of the Generalized Problematic Internet Use Scale 2 (GPIUS2): Psychometric evaluation and analysis of the theoretical model.

The Generalized Problematic Internet Use Scale 2 (GPIUS2) is a self-administered questionnaire that evaluates problematic internet use (PIU) from a multidimensional perspective. We analysed the psychometric properties and adequacy of the theoretical model of Japanese version of the GPIUS2. This study included 291 healthy Japanese adults (median age = 25 years; interquartile range 22-43 years; 128 women) who completed the GPIUS2 and several other questionnaires evaluating the degree of PIU, self-esteem, depression, and impulsivity. Exploratory factor analysis (EFA) revealed a similar factor structure between the original and Japanese versions of the GPIUS2, with only minor differences in item composition. Higher-order confirmatory factor analyses revealed a good overall fit for the factorial model suggested by EFA, indicating adequate construct validity. The model showed acceptable internal consistency. Partial correlation analyses between GPIUS2 and other measures, with age as a control variable, revealed good convergent validity. Finally, structural equation modelling showed a good fit to the data, supporting the cognitive-behavioural model of Caplan (2010). The Japanese version of the GPIUS2 has good psychometric properties and the theoretical model of the original GPIUS2 is applicable to Japanese adults.

Social Support and Daily Life Activity: Determinants of Aging Well

ABSTRACT Despite the importance and prevalence in the use of the term “aging well,” there has been an absence of an empirically valid aging well model for conceptualization and measurement of the phenomenon. The purpose of the study was to conceptualize and test a model for aging well. A sample of 554 non-institutionalized individuals was taken using primary sampling units (PSUs), and a higher-order confirmatory factor analysis was conducted to test a model for aging well. This study confirmed a five-factor structure of aging well, including (a) daily life activity, (b) social resources and supports, (c) mental efficacy, (d) physical health and functional status, and (e) material security. Although all five dimensions strongly contribute to the aging well construct, the statistical results have shown that Social Resources and Support (SRS) and Daily Life Activity are the most important in explaining aging well. Material Security is the weakest latent variable in the model. The study indicates the model for aging well is a single, holistic, and multidimensional framework with a strong theoretical and empirical base.

Anxiety Levels of Dental PhD and Specialty Program Students in Turkey About COVID-19 Pandemic Process

Objective: The aim of this study was to evaluate the anxiety levels about education, clinical practice and professional development of dental PhD and specialty program students in Turkey during pandemic process. &#x0D; Methods: In this study, a scale consisting of two parts was developed by researchers and an online survey portal was used to reach the participants. The first part included personal information of participants. In the second part, there were 5-point scale items with three sub-dimensions to determine the anxiety levels about COVID-19. Item pool was constructed with 23 scale items. For the construct validity, exploratory factor analysis and confirmatory factor analysis was conducted. After the exploratory factor analysis, 3 items exluded from the scale and it resulted in three-factor structure. According to the theoretical background, factors were called as “clinical practice process”, "education process" and "professional development process". When the items related to the sub-dimensions were examined, it was seen that education process includes 8 items, clinical practice process includes 9 items and professional process includes 3 items. Then, this three-factor structure with 20 items by the way one factor model based on these three subfactors were tested through confirmatory factor analysis and higher order confirmatory factor analysis. After exploratory factor analysis and confirmatory factor analysis, it was concluded that there was enough evidence for the construct validity. For the reliability study Cronbach alfa coefficients and corrected item-total correlations was calculated. It was observed that the Cronbach alpha coefficients for the education process sub-dimension, the clinical practice process sub-dimension and the professional development process sub-dimension were 0.87, 0.86 and 0.70, respectively. The stratified Cronbach alpha coefficient calculated for the overall scale was 0.90. It was referred that the scores obtained from the sub-dimensions of the scale and by the way for general score were reliable. For the corrected item-total correlations, education process sub-dimension was in the range of 0.559-0.661, clinical practice sub-dimension was in the range of 0.407-0.812, and professional development process sub-dimension was in the range of 0.487-0.534. It was concluded that all items were sufficiently distinctive and served its purpose. Moreover the data based on the scale were analyzed with t-test and Kruskal Wallis H test. With the statistically significant results of Kruskal Wallis H test, Mann-Whitney U test were conducted as post hoc. &#x0D; Results: A total of 208 students (156 females, 52 males) participated. There was no significant difference among anxiety levels of the groups in terms of gender and department variables (p &gt;0.05). However, there was statistically significant difference in terms of age and education year. The students with the ages in the interval of 22-24 had significantly lower anxiety levels regarding the clinical practice process (p

A psychometric study of the Emotional Availability Scales: Construct validity and measurement invariance between depressed and nondepressed mother-infant dyads.

The Emotional Availability Scales (EAS) are the most widely reported observational assessment measure of parent-child relationships and has been of particular interest in understanding differences between samples of depressed and nondepressed mothers and their offspring. Despite its widespread use, psychometric validation of the factor structure in normative samples and the measurement of invariance within clinical samples has not been published. We evaluated the internal structure (dimensionality, reliability, convergent, and discriminant validity) of the EAS fourth edition using a nondepressed sample of 157 Australian women and their infants aged 6 months, including testing the measurement invariance of the EAS between the same nondepressed sample (n = 157), and a depressed group (n = 185) of mother-infant dyads, using MPlus. Participants were recruited from tertiary hospitals, and depression status was established using a diagnostic measure. Higher-order confirmatory factor analyses on the EAS' six dimensions supported a unidimensional factor solution in our data. Full measurement invariance was not demonstrated due to metric noninvariance of the maternal nonintrusiveness and child responsiveness dimensions. Full scalar invariance supported mean comparisons, and a medium effect of .78SD lower mean emotional availability for the depressed group was found; Cohen's d = .63, 95% CI [.41, .85]. While arguments exist for the clinical utility of differentiating between multiple dimensions of emotional availability, the current findings do not support a multidimensional factor structure or full multigroup measurement invariance of the EAS. Similar psychometric investigations of the EAS in clinical and nonclinical samples are needed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Enhancing Assessment Literacy in Professional Counseling: A Practical Overview of Factor Analysis

Assessment literacy is an essential competency area for professional counselors who administer tests and interpret the results of participants’ scores. Using factor analysis to demonstrate internal structure validity of test scores is a key element of assessment literacy. The underuse of psychometrically sound instrumentation in professional counseling is alarming, as a careful review and critique of the internal structure of test scores is vital for ensuring the integrity of clients’ results. A professional counselor’s utilization of instrumentation without evidence of the internal structure validity of scores can have a number of negative consequences for their clients, including misdiagnoses and inappropriate treatment planning. The extant literature includes a series of articles on the major types and extensions of factor analysis, including exploratory factor analysis, confirmatory factor analysis (CFA), higher-order CFA, and multiple-group CFA. However, reading multiple psychometric articles can be overwhelming for professional counselors who are looking for comparative guidelines to evaluate the validity evidence of scores on instruments before administering them to clients. This article provides an overview for the layperson of the major types and extensions of factor analysis and can serve as reference for professional counselors who work in clinical, research, and educational settings.

The intuitive eating scale-2: re-evaluating its factor structure using a bifactor exploratory structural equation modelling framework.

Previous studies examining the appropriateness of the 4-factor model of intuitive eating scale-2 (IES-2) scores have returned equivocal results, which may reflect methodological limitations in the way IES-2 scores are modelled. Here, we applied a bifactor-exploratory structural equation modelling (B-ESEM) framework to better understand IES-2 multidimensionality. A total of 603 participants from the United States completed the IES-2, alongside measures of body appreciation, body acceptance from others, and self-esteem. Our analyses compared the fit of various hypothesised models of IES-2 scores. Models of IES-2 scores based on confirmatory factor analysis (CFA) uniformly showed poor fit. ESEM models showed superior fit to CFA representations and a B-ESEM model showed improved fit over higher-order CFA and B-CFA representations of IES-2 scores. The optimal model was a B-ESEM model that accounted for, through correlated uniqueness (CU), the methodological artefact introduced by negatively-worded IES-2 items. This B-ESEM-CU model was fully invariant across gender and showed adequate construct validity. The B-ESEM-CU framework appears well-suited to understand the multidimensionality of IES-2 scores. A model of IES-2 scores that yields a reliable latent indicator of global intuitive eating while allowing for simultaneous consideration of additional specific factors will likely provide more accurate accounting of the nature and outcomes of intuitive eating. Level III, cohort study.

BASIC PSYCHOLOGICAL NEEDS, MOTIVATIONAL ORIENTATIONS, EFFORT, AND VOCABULARY KNOWLEDGE

AbstractThis study examined an integrated process model of second language motivation based on the framework of self-determination theory (SDT). Specifically, this research investigated the extent to which satisfying basic psychological needs (BPN) are related to SDT orientations and, in turn, to the effort expended and how these factors relate collectively to vocabulary knowledge. Revised scales assessing students’ BPN (the Basic Psychological Needs of Second Language Scale) and SDT orientations (the Self-Determination Theory of Second Language Scale) were considered and tested using a higher-order confirmatory factor analysis solution. The results of the structural model showed that BPN were only directly related to vocabulary knowledge, which indicated their direct importance for the attainment of the vocabulary. No other indirect effects through SDT orientations or effort were identified. Nonetheless, SDT orientations were both directly and indirectly linked to vocabulary knowledge over and above the role of expended effort. The results elucidated motivational pathways that yielded pedagogical implications for language learning.

Toward a culture-sensitive approach to playfulness research: Development of the Adult Playfulness Trait Scale-Chinese version and an alternative measurement model

The growing global interest in playfulness research calls for increasing research cross-pollination that often necessitates instrument translation with cultural sensitivity. Taking a collaborative and iterative translation approach, we developed a Chinese version of the Adult Playfulness Trait Scale–an instrument originally developed and validated in North America and increasingly used in international applications. Reliability and structural validity of the translated scale were examined using data from a national adult sample of Chinese adults (N = 1561). The results confirmed the scale’s composite reliability and a common playfulness factor shared by four sub-dimensions. Higher-order confirmatory factor analyses validated via split-sample testing revealed a distinct factorial configuration, indicating culture-specific characteristics in Chinese participants’ cognitive organization of the construct—particularly the critical role of a free, non-conforming attribute in the expression of playfulness in Chinese society. Underlying cultural causes are discussed as are implications for cultural adaptation of translated instruments and cross-cultural playfulness research.

A Resilience Model of Adult Native Hawaiian Health Utilizing a Newly Multi-Dimensional Scale

Resilience has conventionally focused on an individual’s ability to overcome adversity. Recent research expands on this definition, making resilience a multi-dimensional construct. Native Hawaiians experience health disparities compared to the general population of Hawai’i. Despite the pressing need to address health disparities, minimal research examines resilience factors that serve as buffers for adverse experiences of Native Hawaiians. The purpose of this study was to estimate psychometric properties of scales that measured resilience-based factors through multiple levels using higher-order confirmatory factor analyses (CFA) and ascertain if this construct of resilience mediated or moderated adversity experienced by a sample of Native Hawaiians. Participants included 125 adults who participated in the Hawaiian Homestead Health Survey. Based on higher-order CFA, resilience comprised internal assets measured by hope, satisfaction with life, and environmental mastery, and external resources measured by social support and Native Hawaiian cultural identity. Results of the structural equation models were consistent with literature focusing on resiliency and health. Findings emphasized the importance of enhancing resilience by considering strengths and resources on the individual, interpersonal, and community levels. Findings also demonstrated the need to address adversity factors directly, with a specific need of addressing socio-economic status factors. According to structural equation models, resilience slightly mediated and moderated the effect of adversity related to socio-economic status. These findings have implications for future research exploring resilience as a mediator or moderator of adversity among Native Hawaiians and emphasize a multi-faceted construct of resilience to promote better health outcomes.

Culture-centeredness in community-based participatory research: contributions to health education intervention research.

Health education research emphasizes the importance of cultural understanding and fit to achieve meaningful psycho-social research outcomes, community responsiveness and external validity to enhance health equity. However, many interventions address cultural fit through cultural competence and sensitivity approaches that are often superficial. The purpose of this study was to better situate culture within health education by operationalizing and testing new measures of the deeply grounded culture-centered approach (CCA) within the context of community-based participatory research (CBPR). A nation-wide mixed method sample of 200 CBPR partnerships included a survey questionnaire and in-depth case studies. The questionnaire enabled the development of a CCA scale using concepts of community voice/agency, reflexivity and structural transformation. Higher-order confirmatory factor analysis demonstrated factorial validity of the scale. Correlations supported convergent validity with positive associations between the CCA and partnership processes and capacity and health outcomes. Qualitative data from two CBPR case studies provided complementary socio-cultural historic background and cultural knowledge, grounding health education interventions and research design in specific contexts and communities. The CCA scale and case study analysis demonstrate key tools that community-academic research partnerships can use to assess deeper levels of culture centeredness for health education research.

A Contribution to Validation of the Short Schema Mode Inventory in an Italian Clinical Versus Non-clinical Population

Schema therapy (ST) has been proven to be an effective psychotherapy model in the treatment of complex personality disorders. ST helps analyze causes of emotional, cognitive, and behavioral alteration in patients using schemas and modes (i.e., sets of emotional states, coping responses and schemas active in a given moment). ST finds its empirical validation in the short Schema Mode Inventory (SMI), a practical tool consisting of 14 subscales assessing 14 different mode categories, grouped in 4 (child, coping, parent and adult modes) high-order categories used to assess different modes at different times. We introduced the Italian validation of the short SMI to a sample of 707 participants, of whom 230 were psychiatric patients. Confirmatory factor analysis (CFA) provides evidence that the 14-dimensional model best describes the SMI’s structure at a lower level. Higher-order CFA provides evidence for both four higher-level mode categories and one higher-level mode category, the four-mode category being the best approach. Internal reliability, test–retest stability and the relationship between the SMI’s subscales have been evaluated with promising results. Clinical vs nonclinical subjects were compared with a multigroup CFA in order to test invariance and with a MANOVA and Bonferroni post hoc comparisons in order to test mean differences. A linear thread was found for all modes except Bully/Attack. While contributing to international research and to the diffusion of SMI and schema therapy, our results also suggest that SMI is a powerful tool for the assessment of modes in ST, both in therapeutic and diagnostic contexts.

Self–Compassion is Best Measured as a Global Construct and is Overlapping with but Distinct from Neuroticism: A Response to Pfattheicher, Geiger, Hartung, Weiss, and Schindler (2017)

Pfattheicher and colleagues recently published an article entitled ‘Old Wine in New Bottles? The Case of Self–compassion and Neuroticism’ that argues the negative items of the Self–compassion Scale (SCS), which represent reduced uncompassionate self–responding, are redundant with neuroticism (especially its depression and anxiety facets) and do not evidence incremental validity in predicting life satisfaction. Using potentially problematic methods to examine the factor structure of the SCS (higher–order confirmatory factor analysis), they suggest a total self–compassion score should not be used and negative items should be dropped. In Study 1, we present a reanalysis of their data using what we argue are more theoretically appropriate methods (bifactor exploratory structural equation modelling) that support use of a global self–compassion factor (explaining 94% of item variance) over separate factors representing compassionate and reduced uncompassionate self–responding. While self–compassion evidenced a large correlation with neuroticism and depression and a small correlation with anxiety, it explained meaningful incremental validity in life satisfaction compared with neuroticism, depression, and anxiety. Findings were replicated in Study 2, which examined emotion regulation. Study 3 established the incremental validity of negative items with multiple well–being outcomes. We conclude that although self–compassion overlaps with neuroticism, the two constructs are distinct. © 2018 European Association of Personality Psychology

Structural validity of the Wechsler Intelligence Scale for Children-Fifth Edition: Confirmatory factor analyses with the 16 primary and secondary subtests.

The factor structure of the Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V; Wechsler, 2014a) standardization sample (N = 2,200) was examined using confirmatory factor analyses (CFA) with maximum likelihood estimation for all reported models from the WISC-V Technical and Interpretation Manual (Wechsler, 2014b). Additionally, alternative bifactor models were examined and variance estimates and model-based reliability estimates (ω coefficients) were provided. Results from analyses of the 16 primary and secondary WISC-V subtests found that all higher-order CFA models with 5 group factors (VC, VS, FR, WM, and PS) produced model specification errors where the Fluid Reasoning factor produced negative variance and were thus judged inadequate. Of the 16 models tested, the bifactor model containing 4 group factors (VC, PR, WM, and PS) produced the best fit. Results from analyses of the 10 primary WISC-V subtests also found the bifactor model with 4 group factors (VC, PR, WM, and PS) produced the best fit. Variance estimates from both 16 and 10 subtest based bifactor models found dominance of general intelligence (g) in accounting for subtest variance (except for PS subtests) and large ω-hierarchical coefficients supporting general intelligence interpretation. The small portions of variance uniquely captured by the 4 group factors and low ω-hierarchical subscale coefficients likely render the group factors of questionable interpretive value independent of g (except perhaps for PS). Present CFA results confirm the EFA results reported by Canivez, Watkins, and Dombrowski (2015); Dombrowski, Canivez, Watkins, and Beaujean (2015); and Canivez, Dombrowski, and Watkins (2015). (PsycINFO Database Record

An ethical evaluation index system for clinical approval of medical technology in China: A structural equation model study.

To construct a quantitative ethical evaluation index system for the clinical approval of medical technology in China. Exploratory factor analysis (EFA) and first-order confirmatory factor analysis (CFA) based on a structure equation model (SEM), higher-order CFA and normalisation were used to establish an ethical evaluation index system for the clinical approval of medical technology. Data were processed in SPSS 13.0 and Lisre l5.3. There were 52 third class indices, 15 second class indices, and 3 first class indices in this ethical evaluation index system. The weight of each index was calculated by normalisation. This study developed a three-level ethical evaluation index system, comprising 70 indices, for the clinical approval of medical technology.

Examining a Metacognitive Instruction Model

This study aims to examine a metacognitive instruction model conducted in the English as Foreign Language (EFL) classroom and to examine the impact of metacognitive instruction (MI) on students’ metacognition, self-efficacy, interaction strategies, and oral communication (OC) by using structural equation modeling (SEM). To examine these impacts, t-tests, a higher-order confirmatory factor analysis, latent variable path analyses, and multi-group comparison analyses were conducted. The participants in this study were Japanese sophomores. They attended an OC course where MI was conducted for three months. They were guided to develop their metacognitive knowledge and self-regulatory strategies regarding OC. The participants’ pre- and post-questionnaire answers and their pre- and post-OC test grades were analyzed using SEM. Results indicated that after the intervention the path from strategy use to OC scores was significant. Moreover, two causal relationships were also strengthened: (a) between metacognition and strategy use and (b) between metacognition and self-efficacy. Furthermore, between the pre-group and the post-group, a significantly different path was found: metacognition → self-efficacy. Taken together, these findings suggest that activated metacognition can lead to more effective learning.

Cross-Validation of the Work Organization Assessment Questionnaire Across Genders: A Study in the Australian Health Care Sector.

The main aim of this study was to examine the measurement invariance of the Work Organization Assessment Questionnaire (WOAQ) across genders in a group of health care employees, using bifactor modeling. There is a very limited research that uses invariance testing of bifactor models, despite their usefulness. Establishing validity of the WOAQ in this way is important for demonstrating its relevance for both men and women. A bifactor modeling procedure was used here to examine the validity of the WOAQ with a sample of 946 paramedics employed in a large Australian organization in the health care sector. The results of this study show that the WOAQ has good psychometric properties across genders in health care settings. In addition, there were significant mean differences between men and women in their perceptions of "quality of relationships with colleagues," and "reward and recognition." There were no differences between men and women in the remaining factors: "quality of relationships with the management," "quality of relationships with colleagues," and "quality of the physical environment." The use of bifactor modeling to establish the cross-validity of the WOAQ across male and female paramedics adds to evidence for the measure's good psychometric properties. The findings confirm those of previous research that has used higher order confirmatory factor analysis. Moreover, mean differences between men and women were found to be significant in two of the five WOAQ subscales. These findings have practical implications for health care organizations, in terms of assessing work characteristics and developing activities to support the health and well-being of their employees.

Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications

ABSTRACTConstruct: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME). Background: Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS. Approach: Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013–June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined. Results: Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001). Conclusions: Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.