Higher-order Confirmatory Factor Analysis Research Articles

Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications

ABSTRACTConstruct: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME). Background: Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS. Approach: Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013–June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined. Results: Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001). Conclusions: Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.

The Social and Economic Mission of Social Enterprises: Dimensions, Measurement, Validation, and Relation

Social entrepreneurs have a dominant social mission and generate revenue to ensure financial viability. However, most research treats the extent to which social entrepreneurs actually adhere to social and economic mission as a black box. Performing higher order confirmatory factor analysis on a sample of social enterprises (N∼270), this study identifies dimensions and validates measures for understanding and delineating social and economic missions, and shows how the two constructs relate to each other. The theoretical untangling and the empirical validation of social and economic missions as distinct constructs—and multiple potential constellations of attached relative importance—opens up opportunities for quantitative hypothesis–testing research in social entrepreneurship.

25 years of higher-order confirmatory factor analysis in the organizational sciences: A critical review and development of reporting recommendations

Summary We discuss how confirmatory factor analysis results should be used to examine potential higher-order constructs and advocate that researchers present five types of evidence, which are as follows: (1) the ability of the higher-order model to reproduce the observed covariation among manifest variables; (2) the ability of the higher-order model to reproduce the observed covariation among manifest variables better than more parsimonious alternative models—and no less well than less parsimonious alternative models; (3) the ability of the higher-order model to reproduce the observed covariation among lower-order factors; (4) the ability of the higher-order factor to explain variation in lower-order factors; and (5) the ability of the higher-order factor to explain variation in manifest variables. We illustrate how this type of evidence could be presented with a worked example and contrast our recommendations with the manner in which higher-order confirmatory factor analysis has been used in the organizational sciences over the past 25 years to support claims regarding higher-order constructs such as core self-evaluations and transformational leadership. Our review shows that a substantial proportion of the 44 examined articles failed to present enough evidence to allow readers to understand the size and importance of higher-order factors. Copyright © 2015 John Wiley & Sons, Ltd.

The Cyberchondria Severity Scale (CSS): An examination of structure and relations with health anxiety in a community sample

McElroy and Shevlin (2014) developed the 33-item Cyberchondria Severity Scale (CSS) to allow for a multidimensional assessment of cyberchondria (compulsion, distress, excessiveness, reassurance, and mistrust of medical professional). The present study evaluated psychometric properties of the CSS, including its factor structure, internal consistency, convergent validity, and incremental validity, using a large sample of community adults located in the United States (N=539). Results from a confirmatory factor analysis (CFA) supported the adequacy of the five-factor structure of the CSS. However, results from a higher-order CFA indicated that the mistrust of medical professional factor does not assess the same construct as the other factors of the CSS. The CSS scales evidenced adequate internal consistency and significantly correlated with health anxiety. The distress, excessiveness, and mistrust of medical professional scales correlated significantly more strongly with health anxiety than obsessive–compulsive symptoms and these three scales were the only CSS scales to share unique variance with health anxiety. Implications of these results for future research are discussed.

Achieving quality performance and environmental sustainability through the genius loci of quality management systems

Purpose – The purpose of this study is threefold. First, the authors will confirm the existence and the intensity of the latent “genius loci” within quality management systems. Second, the authors will confirm the extent of the genius loci with environmental sustainability. Finally, the authors will try to understand whether a quality management system's performance can be affected by its genius loci, and whether a quality management system's environmental sustainability performance can be affected by the extent of environmental sustainability in its genius loci. Design/methodology/approach – The extent of the genius loci in quality management systems and the genius loci with environmental sustainability are tested through higher-order confirmatory factor analysis. Findings – The findings of this study are as follows. First, genius loci partially exist in quality management systems. Second, the genius loci with the environmental sustainability concept also exist partially in certain respects. There is a lack of recognition and actions on environmental sustainability activities in organizations. The lack of recognition and actions leads to the unsatisfactory performance of quality management systems in environmental sustainability. As a result, business goals with environmental sustainability aspects cannot be fully supported and achieved. Originality/value – The results of this study can help small- and medium-sized OEM and ODM computer accessories companies in Taiwan and Southern China to understand the level of genius loci within their quality management systems and the extent that genius loci are linked with environmental sustainability. This understanding can be beneficial to them in achieving successful quality management systems with environmental sustainability.

Does relationship quality matter in service relationships?

Attracting and retaining customers is a crucial issue for service firms. This research explores how relationship quality influences behavioural, attitudinal loyalty and customers' advocacy intentions in service relationships. In this study we propose relationship quality to be a second-order construct consisting of trust, satisfaction, customer advocacy, perceived service quality and perceived service fairness. In a digitally enabled era giving greater freedom for consumers to express their views, these dimensions are very much to the fore. Data were collected from mobile phone service customers using a survey. Data were analysed using higher order confirmatory factor analysis and structural equation modelling. We found support for the higher-order conceptualization of relationship quality with five first-order components: i.e. customer advocacy; service quality; trust; satisfaction; and perceived service fairness. Findings of the study will go a long way in helping service providers in building trust and loyalty among customers and fuelling customer's advocacy intentions.

Predicting the influence of CPTED on perceived neighbourhood cohesion: Considering differences across age

The utility of crime prevention through environmental design (CPTED) in reducing crime and increasing social cohesion has found broad theoretical support, but the multivariate relationships of CPTED have not been fully explored empirically. Although there are a large number of empirical studies regarding the influence of age on an individual's fear, there have been no empirical studies that focus on the influence of age on CPTED and perceived neighbourhood cohesion. This study investigated the hypothesised relationships between CPTED and perceived neighbourhood cohesion (PNC) in elderly and non-elderly homeowners in a typical neighbourhood in Penang, Malaysia. An observation tool is designed as the survey instrument to assess the level of CPTED, completed by an independent observer. This step was followed by a face-to-face questionnaire survey to examine the residents' PNC. The observation tool is subdivided into four domains to reflect the CPTED dimensions, whereas two scales were used to measure PNC. In all, 294 respondents were involved in this study. The results were validated by employing higher-order Confirmatory Factor Analysis to determine the final PNC and CPTED measurement models. A multigroup analysis using structural equation modelling indicated a significant positive effect of CPTED on both PNC scales for the elderly respondents but not for the non-elderly respondents.

Intellect: A theoretical framework for personality traits related to intellectual achievements.

The present article develops a theoretical framework for the structure of personality traits related to intellectual achievements. We postulate a 2-dimensional model, differentiating between 2 processes (Seek and Conquer) and 3 operations (Think, Learn, and Create). The framework was operationalized by a newly developed measure, which was validated based on 2 samples. Subsequently, in 3 studies (overall N = 1,478), the 2-dimensional structure of the Intellect framework was generally supported. Additionally, subdimensions of the Intellect framework specifically predicted conceptually related criteria, including scholastic performance, vocational interest, and leisure activities. Furthermore, results from multidimensional scaling and higher order confirmatory factor analyses show that the framework allows for the incorporation of several constructs that have been proposed on different theoretical backgrounds, such as need for cognition, typical intellectual engagement, curiosity, intrinsic motivation, goal orientation, and openness to ideas. It is concluded that based on the Intellect framework, these constructs, which have been researched separately in the literature, can be meaningfully integrated.

An Examination of the Latent Structure of the Difficulties in Emotion Regulation Scale

The Difficulties in Emotion Regulation Scale (DERS; Gratz and Roemer, Journal of Psychopathology and Behavioral Assessment 26:41–54, 2004) is a popular multidimensional self-report measure of emotion regulation. The present study sought to examine the latent factor structure of the DERS. An examination of latent factor intercorrelations and a higher-order confirmatory factor analysis (CFA) suggested that the DERS-AWARENESS dimension may not represent the same higher-order emotion regulation construct as the other five DERS dimensions. Furthermore, findings supported the adequacy of a revised five-factor model of the DERS in which the AWARENESS dimension was removed. This revised DERS total scale did not diminish concurrent relations between the DERS and outcomes relevant to the emotion regulation domain (i.e., depression, anxiety, posttraumatic stress symptoms). Implications for the conceptualization and assessment of emotion regulation are discussed.

A study of brand attributes: Cross-industries and implications

This study examined the model of brand attributes proposed by Keller (1993). Higher-order confirmatory factor analysis (HCFA) was utilized to confirm each dimension of brand attributes and examine measurement properties. A survey was conducted by both paper-based and online questionnaire. Levi jeans and Nokia cell phones are the two product categories. Reliability and factor analyses show that brand attributes can be differentiated based on intrinsic attributes and extrinsic attributes. Additionally, extrinsic attributes can be divided into the following four hierarchical measurement indexes: price, user imagery, usage imagery, and brand personality. Furthermore, the explanation powers of both extrinsic and intrinsic attributes are different from Levi and Nokia. Key words: Brand attributes, confirmatory factor analysis, brand personality, brand image.

의복 불만족의 고차요인구조 -오프라인 의복구매 및 사용-

This study investigates the ultimate factor structure of consumer dissatisfaction with the off-line purchase and usage of clothing. It identifies the determinant dimensions of consumer dissatisfaction on clothing purchase and usage and investigates the hierarchical structure of consumer dissatisfaction by assessing and comparing the effectiveness of five alternative factor structure models. A total of 300 women were online-surveyed to assess the level of dissatisfaction based on their dissatisfying experience with clothing purchases and usage in terms of product quality, price, salesperson's attitude, and store environment. The exploratory factor analysis identified the underlying dimensions of dissatisfaction: Handling, Aesthetics, Salesperson, Size, Price, Product Quality, Service, and Environment. By employing a first-order confirmatory factor analysis and higher-order confirmatory factor analysis, consumer dissatisfaction was confirmed to have a hierarchical structure with three second-order constructs Intrinsic instrument is manifested by handling, quality, and size, Intrinsic expression consists of service, salesperson, and environment, and Extrinsic circumstance contains aesthetics and price. On empirically demonstrating the multi-dimensional constructs of consumer dissatisfaction by identifying its hierarchical structure, the study provides the theoretical and practical insights to comprehend consumer purchase and post-purchase behavior. Specifically, it affords an empirical platform to extend the scope of research with condensed concepts of dissatisfaction to researchers. In addition, it also enables marketers to take a broader view of consumer dissatisfaction by providing cues about potential problems and identifying the source of those problems.

Alternative Models of Trustworthiness of Service Providers

ABSTRACT The structure and dimensionality of the trustworthiness construct are important theoretical and practical issues. Building on the work of Ennew and Sekhon and based on a sample of 625 respondents, this research identifies a six-factor structure of trustworthiness of service providers in the retail banking sector: customer orientation, integrity and honesty, communication and similarity, shared values, expertise, and ability and consistency. After assessing the reliability and validity of this factor structure, the study also compares the five alternative models of trustworthiness. Results indicated that the trustworthiness model with the six first-order factors has the best model fit. Higher order confirmatory factor analysis was used to identify three second-order trustworthiness factors, which are competency, openness, and benevolence. The various dimensions of trustworthiness are viewable as levers of improving a bank's trustworthiness in the minds of its current and potential customers.

Impact of quality antecedents on taxpayer satisfaction with online tax-filing systems—An empirical study

This paper discusses taxpayer satisfaction with an online system for filing individual income tax returns. Online tax-filing system quality encompasses information, system, and service qualities, which are the antecedents of user satisfaction with any system. To measure online tax-filing system quality, a second-order measurement model was tested using higher-order confirmatory factor analysis. Hierarchical factor analysis supported the measurement model of IS quality in the research setting. Path analysis using structural equation modeling confirmed that the quality antecedents strongly influenced taxpayer satisfaction with the online tax-filing system. Furthermore, the factors of information and system quality were more important than service quality in measuring taxpayer satisfaction, demonstrating the importance to taxpayers of system outputs and processing ability. Based on the findings, implications for online tax-filing systems are discussed. Finally, limitations are presented.

Confirmatory factor structure of the Kaufman Assessment Battery for Children--Second Edition: Consistency with Cattell-Horn-Carroll theory.

The Kaufman Assessment Battery for Children-Second Edition (KABC-II) is a departure from the original KABC in that it allows for interpretation via two theoretical models of intelligence. This study had two purposes: to determine whether the KABC-II measures the same constructs across ages and to investigate whether those constructs are consistent with Cattell-Horn-Carroll (CHC) theory. Multiple-sample analyses were used to test for equality of the variancecovariance matrices across the 3- to 18-year-old sample. Higher-order confirmatory factor analyses were used to compare the KABC-II model with rival CHC models for children ages 6 to 18. Results show that the KABC-II measures the same constructs across all ages. The KABC-II factor structure for school-age children is aligned closely with five broad abilities from CHC theory, although some inconsistencies were found. Models without time bonuses fit better than those with time bonuses. The results provide support for the construct validity of the KABC-II. Additional research is needed to more completely understand the measurement of fluid reasoning and the role of time bonuses on some tasks.

Conceptualizing and Measuring Collaboration

This article conceptualizes and measures collaboration. An empirically validated theory of collaboration, one that can inform both theory and practice, demands a systematic approach to understanding the meaning and measurement of collaboration. We present findings from a study that develops and tests the construct validity of a multidimensional model of collaboration. Data collected using a mail questionnaire sent to 1382 directors of organizations that participate in a large national service program provides the basis for a higher order confirmatory factor analysis. The model that emerges from this analysis demonstrates an overall close fit with the empirical data and the high, standardized gamma coefficients estimated in the model confirm that five key dimensions contribute to an overall construct of collaboration. The primary purpose of this research was to stimulate interest in measurement of collaboration and refinement of the model. As such, we present a detailed description of the analytical process, identify areas that affect interpretation of the data (such as possible selection bias), and propose areas for future research. We believe this effort to conceptualize and measure collaboration offers a foundation for further research. A growing body of multidisciplinary research suggests that we live in an increasingly ‘‘networked’’ world that demands forms of organizing quite different from bureaucracies or firms (O’Toole 1997; Powell 1990). Devolution, increasingly rapid changes in technology, scarce resources, and rising organizational interdependence are factors that explain the emergence of these new forms. Interorganizational collaboration is a term used by scholars and practitioners to describe a process that can emerge as organizations interact with one another to create new organizational and social structures. Collaboration is emerging as a distinct focus of scholarly research. Although the literature is vast, multidisciplinary, and rich with case research, it also lacks coherence across disciplines. A wide range of theoretical perspectives results in an equally wide variety of definitions and understandings of the meaning of collaboration. Although multiple conceptualizations of collaboration add richness to the research, they often impede its rigor and cumulativeness. To put it simply, lack of consensus among scholars on the

The Development and Validation of the Goal Orientation and Learning Strategies Survey (Goals-S)

This article outlines the construction and validation of the Goal Orientation and Learning Strategies Survey (GOALS-S). This 84-item survey was designed to measure students’ motivational goal orientations and their cognitive and metacognitive strategies. Results of first-order confirmatory factor analyses (CFAs) supported the factorial validity of the GOALS-S scales measuring students’ goals and strategies (with goodness-of-fit indices in post-hoc models ranging from .908 to .981). In addition, higher order CFAs (HCFAs) support hierarchical structure of the GOALS-S scales (with goodness-of-fit indices ranging from .904 to .980). Finally, tests of invariance supported the factorial stability of the GOALS-S scales across gender groups (with goodness-of-fit indices ranging from .901 to .981).

How do preschool children feel about themselves? Unraveling measurement and multidimensional self-concept structure.

Theoretical models suggest that 4- and 5-year-old children should be able to differentiate between multiple dimensions of self-concept, but empirical support is limited. A new 38-item Self Description Questionnaire for Preschoolers (SDQP) that measures 6 self-concept factors (Physical, Appearance, Peers, Parents, Verbal, and Math) was developed and tested. Through an individual-interview procedure, young children (4.0-5.6 years) completed the SDQP and achievement tests. The self-concept scales were reliable (.75-.89), first-order and higher order confirmatory factor analysis models fit the data, and factor correlations were mostly moderate (-.03-.73; Mdn = .29). Achievement test scores correlated modestly with academic self-concept factors (rs = .15-40) but were nonsignificantly or significantly negatively related to nonacademic self-concepts. The results contribute to the critical debate about the validity of self-reports for preschool children, who distinguished between multiple dimensions of self-concept at an even younger age than suggested by previous self-concept research.

SELF-CONCEPT: THE GENERALIZABILITY OF RESEARCH ON THE SDQ, MARSH/SHAVELSON MODEL AND I/E FRAME OF REFERENCE MODEL TO UNITED ARAB EMIRATES STUDENTS

This study examined the structure of self-concept (SC) and the internal/external (I/E) frame of reference model for elementary (n = 276) and junior high (n = 293) students in the United Arab Emirates. The results provided support for the multifaceted interpretation for both groups of students. However, the study provided stronger support for the hierarchical structure with junior high than with elementary students' data. Higher-order confirmatory factor analysis (HCFA) of each group revealed that the second-order factor model was the best fitting model. The pattern of correlations indicated that SC in various areas becomes more differentiated with age. Self-concept dimensions were more distinct for junior high students than for elementary students. SDQ-I subscales were more reliable with junior high students (range: .72-.94) than with elementary students (range: .74-.90). The study provided partial support to the propositions of the I/E frame of reference model for both groups of students. MANOVA revealed that elementary students demonstrated significantly higher mean scores than did junior high students on two of the seven SC subscales (math and school SCs).

Measurement Characteristics of the Levels of Institutionalization Scales: Examining Reliability and Validity

The Level of Institutionalization (LoIn) scales were developed to assess the extent to which a health promotion program has become integrated into a health care organization. The instrument was designed specifically to measure the amount of routinization and niche saturation of four subsystems (production, maintenance, supportive, and managerial) believed to make up an organization. In this study, the LoIn scales were completed for diabetes programs in 102 general hospitals and 30 home health agencies in Maryland and Pennsylvania. Reliability estimates across the four subsystems for routines (alpha = .61) and for niche saturation (alpha = .44) were substandard. Average correlation among the four subsystems for routines was .67, and among the four subsystems for niche saturation was .38, indicating moderate to large amounts of shared variance among subsystems and challenging claims of discriminant validity. Given these large correlations and a poor fit when testing the eight-factor model, higher-order confirmatory factor analyses were carried out. Results supported the existence of two second-order factors. When collapsed into two factors, the reliabilities were adequate (routines alpha = .90; niche saturation alpha = .80). Criterion-related validity also was found between length of program existence and the routine factor.

The Structure of Mental Health: Higher-Order Confirmatory Factor Analyses of Psychological Distress and Well-Being Measures

This paper addresses the question of whether psychological distress and subjective well-being are the opposite poles of the same axis of mental health or independent constructs that should be measured on two independent axes. The measures used in this study originate from a preliminary ethnosemantic study and the content analysis of narratives of psychological distress and well-being episodes experienced by a random sample of francophone Quebecers (Canada). Two scales were produced: a Psychological Distress Manifestation Scale (PDMS) based on 23 items and four factors (Anxiety/Depression, Irritability, Self-Depreciation, and Social Disengagement), and a Psychological Well-Being Manifestation Scale with 25 items and six factors (Self-Esteem, Social Involvement, Mental Balance, Control of Self and Events, Sociability, and Happiness). Structural equation modeling analyses confirm that these 10 factors can be viewed as components of two correlated dimensions (psychological distress and well-being) (r = −0.65) of a two-dimensional latent construct which reflects a higher-order concept of mental health. We conclude that assessment of mental health in general populations should use concomitant measures of psychological distress and well-being.