Construct Validity Problems Research Articles

Manipulation checks in the managerial psychology experiment in China: Current status, problems, and suggestions

<p id="p00005">Manipulation checks are important method to ensure the construct validity of managerial psychology experiment, which are also regarded as the basic technology of modern psychology experiments. Although manipulation checks have many advantages and have received widespread attention, current experimental method in managerial psychology literatures mainly focus on the overall experiment design and the strategy of data analysis, rarely investigating the details of manipulation checks design. Although some psychological research published in top international journals has discussed the problem of construct validity and the statistical analysis of manipulation checks, these limited studies have not carried out an integrated review from the key design process of manipulation checks and provided targeted coping strategies. This may lead researchers to undertake manipulation checks designs without solid theoretical support, and even conduct subjective design based on incomplete understanding and draw biased experimental conclusions. <break/>In order to effectively improve the normativity of manipulation checks in the managerial psychology field in China, this study summarized the functions of direct manipulation checks, discriminant manipulation checks and instructional manipulation checks, and then coded 144 experiments drawn from 49 studies in the field of managerial psychology which were published in 4 top-tier journals in 2020 (i.e., Acta Psychologica Sinica, Journal of Psychological Science, Management World, and Nankai Business Review). The present study analyzed the current status, problems and coping strategies of manipulation checks in the managerial psychology field in China from the perspective of the implementation form, the error control, the location choice, the implementation timing and the analytic strategy. <break/>Results showed that the application of manipulation checks in the managerial psychology field were not normative enough in China. Specifically, the way to implement manipulation checks appeared monistic with self-report dominating among these experiments. In terms of error control, many studies did not use maturity measurement scale, and seldomly employed new manipulation checks strategies (e.g., discriminant manipulation checks and instructional manipulation checks) to exclude competitive explanations and invalid experimental samples. For the choice of location, only a few managerial psychology studies paid attention to the influence of manipulation checks location on experiments, and most managerial psychology experiments placed manipulation checks in front of dependent variable. Furthermore, in the aspect of the implementation timing, most managerial psychology experiments completed manipulation checks during the experiment process, and manipulation checks were less implemented in the form of pre manipulation checks (e.g., pilot experiment) and post manipulation checks (e.g., qualitative coding). Finally, as for the strategy of experimental data analysis, the utilization rate of manipulation checks data was low. For example, most managerial psychology studies only focused on analysis of variance of the independent variable between groups and ignored the strength and mediation analysis strategy of the experimental manipulation. <break/>Manipulation checks without enough normativity cannot strongly guarantee the construct validity of experiment and may even interference the experimental process and get incorrect conclusions. To avoid it, future managerial psychology research should pay attention to the application of non-interventional forms to reduce the interference of manipulation checks on the experimental process. Besides, implementing error control strategies should be advocated to eliminate invalid data and competitive explanations. At the same time, the present study suggest that researchers should consider the characteristics of manipulation checks when they decide the appropriate position of manipulation checks included in the experiment material. Researchers are also encouraged to use pre and post manipulation checks to ensure flexibility. In the data analysis phase, we recommend researchers use manipulation strength and mediation analysis to guarantee the internal and external validity of the managerial psychology experiment. By doing so, this study expects the normativity of manipulation checks will be improved to a great extent.

Construct jangle or construct mangle? Thinking straight about (nonredundant) psychological constructs

Psychological science aims to make the abstract measurable and quantifiable. As psychologists it is our challenge and charge to capture complex abstractions (e.g., authoritarianism, prejudice, depression) accurately and with transparency. But recent concerns have been raised about the proliferation of constructs across psychology sub-disciplines, with construct redundancy now rife. Critics charge that we do not take seriously construct validity, especially discriminant validity, which exacerbates the replication crisis. Here the author outlines the problem and discusses at a conceptual level how latent modeling can aptly capture constructs and their interrelations without error, isolating and helping to circumvent construct validity problems. At the core of the issue lies a mathematical reality that seems to be largely ignored in psychology: if correlations within and between indicators of constructs are roughly comparable, their latent factors will correlate near-perfectly and thus be redundant. Thoughts about how the field arrived at this juncture are discussed, along with a recommendation to avoid using cute labels (e.g., jingle-jangle fallacies) to represent very serious problems (better labeled as construct redundancy fallacies). Recommendations for thinking straight about constructs, their validation, and their uniqueness, are offered.

Depression, Anxiety and Stress Scales (DASS-21): Construct Validity Problem in Hispanics.

The main purpose of this research was to examine the construct validity of the Depression, Anxiety and Stress Scales (DASS-21) in order to determine whether it is able to adequately discriminate between symptoms of depression and anxiety in the Hispanic population in Puerto Rico. This study has an instrumental design. A total of 1073 Hispanics participated in this psychometric study. The results showed that the DASS-21 has serious psychometric deficiencies, especially related to the construct validity, as well as convergent and discriminatory validity. In addition, it was shown that DASS-21 do not replicate the three-dimensional structure of the original instrument in the Hispanic community. Finally, it was confirmed that the DASS-21 have difficulty in properly identifying and discriminating between symptoms associated with depression and anxiety in a Hispanic population.

Unresolved issues in defining and assessing writing motivational constructs: A review of conceptualization and measurement perspectives

Abstract Motivational variables significantly influence learners’ writing experiences and performance. Diagnosing learners’ affective perceptions and beliefs using accurate measures is a prerequisite for identifying the optimal ways for motivating them to write. Though writing motivation has been researched for more than four decades, some issues in defining and assessing its constructs are yet to be resolved. This paper critically reviews the developments in conceptualizing and measuring writing motivational constructs. While the review indicates a consistency in the definitions and scales of writers’ outcome expectancy and achievement goals, there are some problematic issues in those of the other constructs. Specifically, the conceptual and terminological overlap and scale construct validity problems are noted in writing apprehension, attitude and anxiety research. Meanwhile, a wide variance occurs in the conceptualizations and measures of writing self-efficacy. Four other constructs (writing self-concept, implicit theories of writing, and incremental and entity theories of writing intelligence) appear to be identical, on the one hand, and close to writing self-efficacy on the other one. The paper ends with discussing some issues that need to be considered in future writing motivation research.

Does schooling actually perpetuate educational inequality in mathematics performance? A validity question on the measures of opportunity to learn in PISA

International assessments of mathematics have shown persistent and widely intensified socioeconomic inequalities in achievement worldwide over the last decades. Such achievement gaps may partly be due to the differences in students’ personal and family characteristics. They may also be attributed to the schooling itself if school systems provide differentiated opportunity to learn (OTL) for children from privileged versus disadvantaged backgrounds. Previous research on the mechanism of the joint relationship among socioeconomic status (SES), academic achievement, and OTL produced inconclusive results. The main aim of the present study is to test whether schooling actually perpetuates social inequality in achievement, by reanalyzing PISA data. Specifically, we scrutinize the construct validity of the OTL measure in PISA that has been used in previous research. Our analyses reveal two latent dimensions of the OTL indicators in PISA, namely an unbiased OTL dimension and a self-concept dimension. The relationship between social background and mathematics achievement was only weakly mediated by OTL, when effect of students’ self-concept was controlled for. Our results suggest that the previous research finding that schooling perpetuates social gaps in mathematics performance suffers from a construct validity problem.

Exploring DSM-5 ADHD criteria beyond young adulthood: phenomenology, psychometric properties and prevalence in a large three-decade birth cohort.

There are still uncertainties on the psychometric validity of the DSM-5 attention deficit hyperactivity disorder (ADHD) criteria for its use in the adult population. We aim to describe the adult ADHD phenotype, to test the psychometric properties of the DSM-5 ADHD criteria, and to calculate the resulting prevalence in a population-based sample in their thirties. A cross-sectional evaluation using the DSM-5 ADHD criteria was carried out in 3574 individuals from the 1982 Pelotas Birth Cohort. Through receiver operator curve, latent and regression analyses, we obtained parameters on construct and discriminant validity. Still, prevalence rates were calculated for different sets of criteria. The latent analysis suggested that the adult ADHD phenotype is constituted mainly by inattentive symptoms. Also, inattention symptoms were the symptoms most associated with impairment. The best cut-off for diagnosis was four symptoms, but sensitivity and specificity for this cut-off was low. ADHD prevalence rates were 2.1% for DSM-5 ADHD criteria and 5.8% for ADHD disregarding age-of-onset criterion. The bi-dimensional ADHD structure proposed by the DSM demonstrated both construct and discriminant validity problems when used in the adult population, since inattention is a much more relevant feature in the adult phenotype. The use of the DSM-5 criteria results in a higher prevalence of ADHD when compared to those obtained by DSM-IV, and prevalence would increase almost threefold when considering current ADHD syndrome. These findings suggest a need for further refinement of the criteria for its use in the adult population.

An Empirical Investigation of the Relationship Between Emotional Intelligence, Transactional and Transformational Leadership Styles in Banking Sector

Leadership theory and research have not adequately considered how leader's emotions influence their effectiveness. While there exists a significant amount of research on leadership, literature is comparatively thin on the relationship between leadership and emotional intelligence. A better understanding of emotional intelligence and its relationship to leadership style can address the existing gaps in literature and provide a more informed link between theory and practice. This study attempts to broaden the knowledge base of Human Resource Development through the investigation of emotional intelligence and leadership style. The total sample of 381 employees participated in the study. The findings of this study suggest that the ability perspective of emotional intelligence does not have any relationship with perceptions of leadership style. This finding is contrary to what one would expect from reviewing the literature and the associated citations. Mixed reliability results obtained within the correlations among the items, pointing to some potential construct validity problems served as a possible explanation for no significant correlations found between the undertaken dimensions.

A conceptual approach for measuring farmers’ attitudes to integrated soil fertility management in Kenya

A sustainable increase in agricultural productivity is essential in assuring food security in developing countries. Low soil fertility is a major contributing factor to the current vicious cycle of low agricultural productivity and inadequate livelihoods among smallholder farmers. Integrated soil fertility management (ISFM) is one way of achieving sustainable agricultural development, but improving soil fertility through ISFM requires interventions that match the behavioural inclinations of farmers and their decision making. Using survey data on 125 commercial peri-urban farmers growing kale (Brassica oleracea) around Nairobi, Kenya, this study examined two conceptual approaches for measuring ISFM attitudes. A Rasch model, where the odds ratio for engaging in an ISFM practice is given by the difference between farmers’ attitude and the difficulty of the practice in terms of behavioural cost, identified ISFM attitudes as a unidimensional concept. However, assessing attitudes based on a standard valence method raised problems of construct validity. Accounting for behavioural costs as determinants of ISFM, in addition to other pecuniary costs, may improve our understanding of how farmers deal with complex choices in the ISFM context. Our findings suggest that high behavioural costs in relation to use of human faeces as manure, use of crop residues and transport impede adoption of ISFM practices vital to increased productivity. These findings can be used to develop ISFM communications and improve the efficacy of different interventions intended to increase potential uptake of ISFM practices.

Resolving the assessment center construct validity problem (as we know it).

Ongoing concern about the construct validity of assessment center dimensions has focused on postexercise dimension ratings (PEDRs) that are consistently found to reflect exercise variance to a greater degree than dimension variance. Here, we present a solution to this problem. Based on the argument that PEDRs are an intermediate step toward an overall dimension rating, and that the overall dimension rating should be the focus of inquiry, we demonstrate that correlated sources of dimension variance accumulate and increasingly displace uncorrelated sources of both systematic variance and error. Viewing overall dimension ratings as a composite of PEDRs, we show dimension variance will commonly quickly overtake exercise-specific variance as the dominant source of variance as ratings from multiple exercises are combined. We embed our results in a new framework for categorizing different levels of construct variance dominance, and our results indicate that with as few as two exercises, dimension variance can reach our lowest level of construct variance dominance. However, the largest source of dimension variance is a general factor. We conclude that the construct validity problem in assessments centers never existed as historically framed, but the presence of a general factor may limit interpretation for developmental purposes.

SEM Approach to Teen Influence in Family Decision Making

The primary purpose of this paper is to provide an empirical design on teen influence and its predictors in the context of family decision making. Structure equation modeling (SEM) is employed to test the proposed conceptual model on teen influence and its predictors. The approach also provides for the construct validity of the key measures. Results indicate that teenage children in urban Indian households significantly influence purchase-related decisions in the family. The findings from the empirical data suggest that peers, shopping and the Internet are significant predictors of teen influence in family purchases, although the media’s role is marginalized (as a predictor) to the influence that teenagers have in family decision making. This study is significant, as great scholastic interest is placed on such contemporary research, which aims to discern the role of a variety of factors that impact teen influence in the family. This study also addresses the problem of construct validity of key measures, which has been somewhat overlooked in the past research. Keywords: Indian Teenagers, Family Decision Teen Influence, Consumer Socialization, SEM, Internet, Peers, Media and Shopping To cite this document: Anupriya Kaur and Yajulu Medury, SEM Approach to Teen Influence in Family Decision Making, Contemporary Management Research, Vol.9, No.3, pp.323-342, 2013. Permanent link to this document: http://dx.doi.org/10.7903/cmr.11080

A Commentary on Short Questionnaires for Assessing Usability

Usability is an important aspect of any interactive system, but only one aspect leading to its success. Thus, when evaluating a system, it is useful to have short questionnaires that can lead to reliable, sensitiveandvalidmeasuresofusability.Inthiscommentary,Idiscusstwopapersthataimtodevelop two such measures: one is for a four-item usability metric and the other a single-item usability.Whilst care is taken in both cases to produce suitable metrics, the processes have flaws that influence the value of these metrics. The underlying problem seems to lie with the application of psychometric methods when there are better methods available in this context. Even allowing for this, in the end there remains the problem of construct validity that undermines any such efforts.

Deconstructing the brand nomological network

Construct validity problems in prior research have stimulated concept fragmentation in some marketing areas. Weaknesses in prior brand research and sources for construct invalidity are discussed. To bring greater clarity and understanding to the brand-related constructs, major brand concepts are described and defined in order to differentiate them from proximate constructs in their nomological net. Managerial implications are discussed.

The Role of Theory in Furthering Evidence-Based Practice

This issue of HERD presents a fascinating array of research ranging from the development of substantive findings, to measurement development, to meta-analysis. All of these research purposes are exceptionally important for the development of a robust research base for evidence-based design. In these articles, the authors note the limitations of their work in regard to generalizability, measurement problems, and so forth, all of which remain symptomatic of much research in this domain. Simultaneously, the studies reflect a willingness to push the domain ahead, whether in terms of research strategy (e.g., retrospective data analysis or meta-analysis) or instrument development. There is much to learn from this collection of articles; a brief discussion of each follows.Brooks and colleagues conducted a two-part, highly exploratory pilot study regarding the design of nightstands. The first study involved 28 participants in assisted living or rehabilitation settings who were interviewed in regard to their use and perceptions of nightstands. The second study involved interviews of 36 university students and 36 older adults regarding perceptions of a smart design for a nightstand or side table. One should be circumspect about drawing conclusions from such an exploratory study because of sample size, problems of construct validity, and issues associated with research design.Lorenz and Dreher conducted a retrospective study that compared private and shared hospital rooms with regard to the occurrence of falls, hospital-acquired infection (HAI) rates, and risk for social isolation. Unfortunately the sample was extremely small (five falls in 166 patients; nine occurrences of HAI); thus, empirical conclusions in the study should be considered with caution. The authors acknowledge this limitation.Foureur and colleagues provide an update on the ongoing development of the Birth Unit Design Spatial Evaluation Tool (BUDSET), designed to assess the optimality of birth settings. The intriguing conceptual model underlying the development of this tool posits that optimal birth settings both facilitate effective communication and decrease patient and staff stress, which are mediated by the design of the birth unit as well as the model of care employed. The authors identify four critical spatial domains for birthing units: responses to the fear cascade, and facility characteristics, aesthetics, and support. Each of these is manifested by multiple characteristics, which are assessed through specific observable phenomena. The tool could discriminate between labor ward settings and birth centers, but the internal validity of half of the characteristics was unacceptable, suggesting the need for significant refinement of the tool.Algase and colleagues report on a cross-sectional observational study that compares two approaches to assessing the experiential attribute of crowding in long-term care settings. Crowding was assessed in two ways: first, by a simple count of people present in the room or area being observed; second, by means of the Long-Term Care Crowding Index (LTC-CI) developed by the authors. This study highlights different levels of crowding between nursing homes and assisted living facilities as well as variations between types of room and time of day. The LTC-CI advances the body of knowledge by attempting to account for density and to identify the role proximity plays in the assessment of crowding.Marquardt's contribution is a meta-analysis of the literature on spatial wayfinding for people with dementia. This extensive literature review makes clear a tangible connection between neuroscience and architectural design by suggesting that the optimization of spatial configuration and environmental cueing affects the orientation processes of people with dementia. Marquardt determines that limiting choice points, making environmental amenities visually accessible, and reinforcing wayfinding through appropriate environmental cues and signage are likely to provide a more supportive wayfinding environment. …

Extending Religion-Health Research to Secular Minorities: Issues and Concerns

Claims about religion's beneficial effects on physical and psychological health have received substantial attention in popular media, but empirical support for these claims is mixed. Many of these claims are tenuous because they fail to address basic methodological issues relating to construct validity, sampling methods or analytical problems. A more conceptual problem has to do with the near universal lack of atheist control samples. While many studies include samples of individuals classified as "low spirituality" or religious "nones", these groups are heterogeneous and contain only a fraction of members who would be considered truly secular. We illustrate the importance of including an atheist control group whenever possible in the religiosity/spirituality and health research and discuss areas for further investigation.

The construct validity of conceptual and operational definitions of contextual performance and related constructs

A number of theorists and researchers have distinguished between the constructs of task performance (i.e., non-discretionary work behaviors) and contextual performance (CP), as well as the related constructs of organizational citizenship behavior, prosocial behavior, and extra-role behavior. In addition, measures of CP have been used in a large number of studies that have attempted to show both their validity and utility. However, an analysis of the conceptual and operational definitions of the CP reveals a number of serious construct validity problems. For example, items in extant CP measures index what are typically regarded as required (non-discretionary) work behaviors. Thus, we describe several CP-related construct validity problems, and illustrate their nature using data from a sample of 98 job descriptions from the Dictionary of Occupational Titles and the online Occupational Information Network. The same data showed that behaviors that are generally viewed as representative of CP are frequently regarded as being exemplars of task performance. The important implications of the confounding of task performance and CP constructs are considered.

Criminal Laws on the Fringe: An Analysis of Legislated Punishments for Morality Crimes in the 50 States

Each of the 50 states currently attempts to prohibit acts that could be called victimless, public order, or morality-based crimes. The first objective of this research is to identify conceptually similar criminal offenses regulating adult behavior in the United States. The state-level crimes ultimately selected for our analysis of 2006 online criminal code sanctions include: prostitution (separately reviewing “anti-john” statutes), gambling, marijuana possession, obscenity, adultery, and public intoxication. Due to construct validity problems encountered within the literature when we wanted to first select, and then later describe, our collection of seven crimes, we introduce a new term: Fringe Laws. Second, we will use descriptive statistics to analyze the states’ statutes, using their allowable incarceration amounts and maximum fines to examine how each of these crimes are punished across the 50 states, and to determine which states are most and least punitive in terms of average punishment level across the seven Fringe Laws. Finally, using the Excel map tool, we looked for any obvious spatial patterns among the states.

Why Assessment Centers Do Not Work the Way They Are Supposed To

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing “construct validity problem” for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task- or role-based ACs and away from traditional dimension-based ACs.

Improving Assessment Centers by the Trimodal Concept of Personnel Assessment

Lance (2008) attacks the ‘‘assessment center validity paradox,’’ arguing that construct validity is not as low as a number of researchers seem to have agreed about. But he neglects the second part of the supposed paradox, predictive validity. Assessment center predictive validity is not as high as is circulated by many researchers—and all the more by most of the users. This seems to me the even more important deficit. My suggestion is to solve the theoretical problem of construct validity as well as the pragmatic problem of predictive validity by (re)conceiving the assessment center as a truly multimodal procedure. Assessment centers used to be valuable diagnostic tools—at times when they were run as procedures integrating a broad methodological diversity. One of the icons of assessment center research, the extensive ATT that is, there is a rather continuous decline of assessment center validity over the past 40 years. There are several possible explanations for this decline. My suggestion is that assessment centers often perform poorly because too simplistic methods are employed. Their attractiveness for managers and practitioners in personnel departments is connected with concentrating on ‘‘exercises’’such as a group discussion, roleplay, and presentation, which allow for behavioral observations and a lively personal impression formation but are essentially nonpsychometric tools. Correspondence concerning this article should be addressed to Heinz Schuler. E-mail: schuler@unihohenheim.de Address: Department of Psychology (540F), University of Hohenheim, D-70593 Stuttgart, Germany Heinz Schuler, Department of Psychology (540F), University of Hohenheim Industrial and Organizational Psychology, 1 (2008), 128–130. Copyright a 2008 Society for Industrial and Organizational Psychology. 1754-9426/08

A test of the quasi-circumplex structure of human values

Schwartz’s value system (SVS) has been widely used in different disciplines (e.g., psychology, management, and marketing). Although the value structure seems to be validated when data are analyzed through multidimensional scaling, we show that the quasi-circumplex structure of human values is not supported when confirmatory analysis approaches (e.g., CIRCUM and constrained confirmatory factor analysis) are used. Based on two samples of French and Swiss respondents, confirmatory tests of SVS provide little support for its quasi-circumplex structure, mainly due to problems of construct and discriminant validity resulting from multicollinearity between value types.

Conceptual Complexity and Construct Validity

Despite a century of methodological and conceptual advances in the technology of psychosocial measurement, poor correspondence between indicators and the constructs they are intended to represent remains a limiting factor to the accumulation of scientific knowledge. Longstanding conventions in measurement may contribute to the failure to develop optimal criteria. These conventions include the focus on complex over simple constructs and the use of multi-item measures of disparate content to represent those constructs. Several arguments suggest that such a measurement model compromises the potential for developing measures that accurately reflect psychosocial phenomena. The article concludes with some preliminary suggestions concerning an alternative model that may address this construct validity problem more effectively.