Ipsative Data Research Articles

Item Response Theory Models for Polytomous Multidimensional Forced‐Choice Items to Measure Construct Differentiation

AbstractMultidimensional forced‐choice (MFC) items have been found to be useful to reduce response biases in personality assessments. However, conventional scoring methods for the MFC items result in ipsative data, hindering the wider applications of the MFC format. In the last decade, a number of item response theory (IRT) models have been developed, majority of which are for MFC items with binary responses. However, MFC items with polytomous responses are more informative and have many applications. This paper develops a polytomous Rasch ipsative model (pRIM) that can deal with ipsative data and yield estimates that measure construct differentiation—a latent trait that describes the degree to which the personality constructs (e.g., interests) distinguish between each other. The pRIM and its simpler form are applied to a career interests assessment containing four‐category MFC items and the measures of interests differentiation are used for both intra‐ and interpersonal comparisons. Simulations are conducted to examine the recovery of the parameters under various conditions. The results show that the parameters of the pRIM can be well recovered, particularly when a complete linking design and a large sample are used. The implications and application of the pRIM in the personality assessment using MFC items are discussed.

Factor Analysis: Dealing with Response Bias

This paper proposes an innovative method for factor analyzing data that potentially contains individual response bias. Past methods include the use of “ipsative” data, or, related to that, “ipsatized” data. Unfortunately, factor analysis as the main method used for analyzing the dimensionality of data, cannot be applied to ipsative data. In contrast, normalization of data as an alternative method to filter out response bias, is not hampered by the technical statistical issues inherent to applying multivariate techniques to ipsative data. Using high-quality data from a survey in Nepal that makes use of – among others – the High-Performance Organizations (HPO) framework, this paper shows that the traditional approach of directly applying Confirmatory Factor Analysis (CFA) starting from an existing model or theory, is inferior to our approach. Even applying Exploratory Factor Analysis (EFA) to the raw (non-normalized) data before using CFA, is unable to detect the optimal dimensionality, or structure, in the data. A better structure can be obtained by performing EFA on normalized data that corrects for response bias in the raw data. This paper convincingly shows that the newly identified structure is superior to the original structure suggested by the HPO framework. Applying a CFA using the newly detected structure on the raw data, gives excellent goodness-of-fit statistics, with more items retained, and no need of forced methods to improve the model fit. The findings suggest that existing models and questionnaires based on these models, are not necessarily as valid and reliable as empirical studies that make use of traditional analyses seem to suggest. When adopting existing instruments, researchers are advised to critically check the validity and reliability of these instruments – especially those vulnerable to response bias - and to apply the procedures laid out in this paper, in order to enhance the quality of their research, and to inform future researchers who consider using the same instruments or to warn them about the potential shortcomings of these instruments.

Q-sort Methodology - the systematic study of participant viewpoints in Social Research

In social sciences research there are instruments made up of questions and that aim to ascertain opinions, behaviors and attitudes, aim to find out how the person would act in a given situation, instead of putting the person to the test. Although there are no right or wrong answers, there is a tendency to respond in a socially acceptable way, even if the answer does not correspond to reality. This problem can be circumvented through the Q-sort methodology that combines quantitative and qualitative data and analytical techniques that are not present in other methods. In this way, it consists of presenting participants with a set of statements on a given topic and asking them to classify them according to their opinion, according to a predefined distribution, which, in general, is approximately normal. This methodology forces participants to distribute the score among the items on the scale, thus avoiding the constraints associated with social desirability and the tendency to respond in the same way or always through the midpoint to different questions. Another advantage is that it provides linearity and near-normality to the scale, which makes it possible to compare subjects more easily with each other. Notwithstanding its advantages, Q-sort methodology also has negative points because forced-choice measures produce ipsative data that lead to distorted scales and problematic psychometric properties. As the data are obtained by ordering a set of items or by forcefully choosing one item over another, it is impossible to reach very high or very low values on all scales, which gives rise to a large number of negative values that, in turn, they result in an average correlation between the scales, which is also negative. In view of the above, it was considered pertinent to review the research carried out on the q-sort methodology, presenting its main advantages and disadvantages.

The Multidimensional Forced-Choice Format as an Alternative for Rating Scales

The Multidimensional Forced-Choice Format as an Alternative for Rating Scales

On the Validity of Forced Choice Scores Derived From the Thurstonian Item Response Theory Model.

Forced choice (FC) measures may be a desirable alternative to single stimulus (SS) Likert items, which are easier to fake and can have associated response biases. However, classical methods of scoring FC measures lead to ipsative data, which have a number of psychometric problems. A Thurstonian item response theory (TIRT) model has been introduced as a way to overcome these issues, but few empirical validity studies have been conducted to ensure its effectiveness. This was the goal of the current three studies, which used FC measures of domains from popular personality frameworks including the Big Five and HEXACO, and both statement and adjective item stems. We computed TIRT and ipsative scores and compared their validity estimates. Convergent and discriminant validity of the scores were evaluated by correlating them with SS scores, and test-criterion validity evidence was evaluated by examining their relationships with meaningful outcomes. In all three studies, there was evidence for the convergent and test-criterion validity of the TIRT scores, though at times this was on par with the validity of the ipsative scores. The discriminant validity of the TIRT scores was problematic and was often worse than the ipsative scores.

Broadening Humor: Comic Styles Differentially Tap into Temperament, Character, and Ability.

The present study introduces eight comic styles (i.e., fun, humor, nonsense, wit, irony, satire, sarcasm, and cynicism) and examines the validity of a set of 48 marker items for their assessment, the Comic Style Markers (CSM). These styles were originally developed to describe literary work and are used here to describe individual differences. Study 1 examines whether the eight styles can be distinguished empirically, in self- and other-reports, and in two languages. In different samples of altogether more than 1500 adult participants, the CSM was developed and evaluated with respect to internal consistency, homogeneity, test–retest reliability, factorial validity, and construct and criterion validity. Internal consistency was sufficiently high, and the median test-retest reliability over a period of 1–2 weeks was 0.86 (N = 148). Exploratory and confirmatory factor analyses showed that the eight styles could be distinguished in both English- (N = 303) and German-speaking samples (N = 1018 and 368). Comparing self- and other-reports (N = 210) supported both convergent and discriminant validity. The intercorrelations among the eight scales ranged from close to zero (between humor and sarcasm/cynicism) to large and positive (between sarcasm and cynicism). Consequently, second-order factor analyses revealed either two bipolar factors (based on ipsative data) or three unipolar factors (based on normative data). Study 2 related the CSM to instruments measuring personality (N = 999), intelligence (N = 214), and character strengths (N = 252), showing that (a) wit was the only style correlated with (verbal) intelligence, (b) fun was related to indicators of vitality and extraversion, (c) humor was related to character strengths of the heart, and (d) comic styles related to mock/ridicule (i.e., sarcasm, cynicism, but also irony) correlated negatively with character strengths of the virtues temperance, transcendence, and humanity. By contrast, satire had a moral goodness that was lacking in sarcasm and cynicism. Most importantly, the two studies revealed that humor might be related to a variety of character strengths depending on the comic style utilized, and that more styles may be distinguished than has been done in the past. The CSM is recommended for future explorations and refinements of comic styles.

The Motivational Value Systems Questionnaire (MVSQ): Psychometric Analysis Using a Forced Choice Thurstonian IRT Model.

This study presents a new measure of value systems, the Motivational Value Systems Questionnaire (MVSQ), which is based on a theory of value systems by psychologist Clare W. Graves. The purpose of the instrument is to help people identify their personal hierarchies of value systems and thus become more aware of what motivates and demotivates them in work-related contexts. The MVSQ is a forced-choice (FC) measure, making it quicker to complete and more difficult to intentionally distort, but also more difficult to assess its psychometric properties due to ipsativity of FC data compared to rating scales. To overcome limitations of ipsative data, a Thurstonian IRT (TIRT) model was fitted to the questionnaire data, based on a broad sample of N = 1,217 professionals and students. Comparison of normative (IRT) scale scores and ipsative scores suggested that MVSQ IRT scores are largely freed from restrictions due to ipsativity and thus allow interindividual comparison of scale scores. Empirical reliability was estimated using a sample-based simulation approach which showed acceptable and good estimates and, on average, slightly higher test-retest reliabilities. Further, validation studies provided evidence on both construct validity and criterion-related validity. Scale score correlations and associations of scores with both age and gender were largely in line with theoretically- and empirically-based expectations, and results of a multitrait-multimethod analysis supports convergent and discriminant construct validity. Criterion validity was assessed by examining the relation of value system preferences to departmental affiliation which revealed significant relations in line with prior hypothesizing. These findings demonstrate the good psychometric properties of the MVSQ and support its application in the assessment of value systems in work-related contexts.

Study Protocol on Intentional Distortion in Personality Assessment: Relationship with Test Format, Culture, and Cognitive Ability

Self-report personality questionnaires, traditionally offered in a graded-scale format, are widely used in high-stakes contexts such as job selection. However, job applicants may intentionally distort their answers when filling in these questionnaires, undermining the validity of the test results. Forced-choice questionnaires are allegedly more resistant to intentional distortion compared to graded-scale questionnaires, but they generate ipsative data. Ipsativity violates the assumptions of classical test theory, distorting the reliability and construct validity of the scales, and producing interdependencies among the scores. This limitation is overcome in the current study by using the recently developed Thurstonian item response theory model. As online testing in job selection contexts is increasing, the focus will be on the impact of intentional distortion on personality questionnaire data collected online. The present study intends to examine the effect of three different variables on intentional distortion: (a) test format (graded-scale versus forced-choice); (b) culture, as data will be collected in three countries differing in their attitudes toward intentional distortion (the United Kingdom, Serbia, and Turkey); and (c) cognitive ability, as a possible predictor of the ability to choose the more desirable responses. Furthermore, we aim to integrate the findings using a comprehensive model of intentional distortion. In the Anticipated Results section, three main aspects are considered: (a) the limitations of the manipulation, theoretical approach, and analyses employed; (b) practical implications for job selection and for personality assessment in a broader sense; and (c) suggestions for further research.

An Alternative Approach to Analyze Ipsative Data. Revisiting Experiential Learning Theory.

The ritualistic use of statistical models regardless of the type of data actually available is a common practice across disciplines which we dare to call type zero error. Statistical models involve a series of assumptions whose existence is often neglected altogether, this is specially the case with ipsative data. This paper illustrates the consequences of this ritualistic practice within Kolb's Experiential Learning Theory (ELT) operationalized through its Learning Style Inventory (KLSI). We show how using a well-known methodology in other disciplines—compositional data analysis (CODA) and log ratio transformations—KLSI data can be properly analyzed. In addition, the method has theoretical implications: a third dimension of the KLSI is unveiled providing room for future research. This third dimension describes an individual's relative preference for learning by prehension rather than by transformation. Using a sample of international MBA students, we relate this dimension with another self-assessment instrument, the Philosophical Orientation Questionnaire (POQ), and with an observer-assessed instrument, the Emotional and Social Competency Inventory (ESCI-U). Both show plausible statistical relationships. An intellectual operating philosophy (IOP) is linked to a preference for prehension, whereas a pragmatic operating philosophy (POP) is linked to transformation. Self-management and social awareness competencies are linked to a learning preference for transforming knowledge, whereas relationship management and cognitive competencies are more related to approaching learning by prehension.

Organization and visualization for initial analysis of forced-choice ipsative data

Forced-choice ipsative data are common in personality, philosophy and other preference-based studies. However, this type of data inherently contains dependencies that are challenging for usual statistical analysis. In order to utilize the structure of the data as a guide for analysis rather than as a challenge to manage, a visualisation tool was developed to emphasize the established framework and enhance analysis tools. This visualisation has many strength, including the display of significant amounts of data at once in three dimensions with little loss of detail, incorporating the theoretical framework for the data into the structure of the visualisation and organizing the data for use with common statistical tests and analysis tools.

An empirical study of the career anchors that govern career decisions

Purpose – The purpose of this paper is to examine two of Feldman and Bolino's proposals: career anchor plurality and career anchor relationships. Design/methodology/approach – A novel method for examining the relationships between career anchors called “indices of mutual presence” is developed for this study to generate meaningful results from ordinal and ipsative career anchor data. Findings – Evidence for some individuals having multiple career anchors was found. Complementary and exclusivity career anchor relationships are identified and a model for representing them is presented. The importance and possible benefit of understanding both an individual's preferred and “unpreferred” anchors is discussed. The non-reflexive nature of career anchors is explored and the idea of “mutually” exclusive career anchors is rejected. Weaknesses in the octagon shaped career anchor relationships diagram presented by Feldman and Bolino are discussed. Research limitations/implications – Despite the benefits associated with forced-choice assessments, some have expressed concern because of the nature of this type of evaluation. Each time an item is preferred, another item must be “unpreferred.” Thus, for one item to have a high preference count, some other item must necessarily have lower preference counts. The resulting data is ordinal rather than interval or ratio. It contains information regarding order of preference, but provides little insight into magnitude of preference. This makes it difficult to identify and examine how much more or less one individual prefers an item when compared to another individual. Originality/value – The second property of forced-choice data that raises concern is its ipsative nature. As respondents are constrained to unprefer an item each time the prefer one, the total preference counts remain the same for every individual. As a result, the preference scores for every individual will always sum to the same value. When data has this property, it is called ipsative. Ipsative data is not free to vary, and thus statistical methods which analyze variance may yield spurious results. Thus, traditional factorial statistical methods cannot be appropriately used with ipsative data (Baron, 1996; Bartram, 1996; Closs, 1996). It is commonly believed that researchers trade ease of use and accuracy for fewer available statistical tools when using forced-choice methods. However, this paper attempts to use “indices of mutual presence” developed for this study (described below) that do not rely on variance to generate meaningful results from ipsative career anchor data.

Ipsative measurement and the analysis of organizational values: an alternative approach for data analysis

In this paper, the analysis and test of ipsative data will be discussed, and some alternative methods will be suggested. Following a review of the literature about ipsative measurement, the Competing Values Framework will be presented as a major application in the field of organizational culture and values. An alternative approach for the intra-individual analysis and test of ipsative data will be suggested, which consists of: (i) a method that uses closed part-wise geometric means as a descriptive statistic; (ii) a nonparametric bootstrap test to create confidence intervals; and (iii) a permutation test to evaluate equivalence between ipsative scores. All suggested methods satisfy the three basic statistical requirements for the analysis of ipsative data, that is: scale invariance, permutation invariance, and subcompositional coherence. Our suggested approach can correctly compute and compare organizational culture profiles within the same organization, as will be demonstrated with an example. However, the problem of drawing inter-organizational contrasts in ipsative measurement still remains unsolved. Also, our alternative approach only allows for a relative interpretation of the results.

Análisis factorial de ítems de respuesta forzada: una revisión y un ejemplo

Forced-choice tests are widely used in order to reduce the impact of different response set biases typically associated to psychological tests (e.g. acquiescence or social desirability). However, these tests produce ipsative data which have undesirable properties, thereby making an inappropriate application of classical factor analysis techniques for psychometric evaluation commonly used by researchers. This paper explains the analytical properties of forced-choice tests, along with an example that illustrates how these properties have an impact on the application of conventional statistical techniques and produce improper results. Additionally, one of the current proposals is presented in order to analyze these data based on the comparative judgment model by Thurstone, along with the results of a simulation study which illustrates its implementation and effectiveness in recovering the original factor structure.

How IRT can solve problems of ipsative data in forced-choice questionnaires.

In multidimensional forced-choice (MFC) questionnaires, items measuring different attributes are presented in blocks, and participants have to rank order the items within each block (fully or partially). Such comparative formats can reduce the impact of numerous response biases often affecting single-stimulus items (aka rating or Likert scales). However, if scored with traditional methodology, MFC instruments produce ipsative data, whereby all individuals have a common total test score. Ipsative scoring distorts individual profiles (it is impossible to achieve all high or all low scale scores), construct validity (covariances between scales must sum to zero), criterion-related validity (validity coefficients must sum to zero), and reliability estimates. We argue that these problems are caused by inadequate scoring of forced-choice items and advocate the use of item response theory (IRT) models based on an appropriate response process for comparative data, such as Thurstone's law of comparative judgment. We show that when Thurstonian IRT modeling is applied (Brown & Maydeu-Olivares, 2011), even existing forced-choice questionnaires with challenging features can be scored adequately and that the IRT-estimated scores are free from the problems of ipsative data.

Fitting a Thurstonian IRT model to forced-choice data using Mplus

To counter response distortions associated with the use of rating scales (a.k.a. Likert scales), items can be presented in a comparative fashion, so that respondents are asked to rank the items within blocks (forced-choice format). However, classical scoring procedures for these forced-choice designs lead to ipsative data, which presents psychometric challenges that are well described in the literature. Recently, Brown and Maydeu-Olivares (Educational and Psychological Measurement 71: 460-502, 2011a) introduced a model based on Thurstone's law of comparative judgment, which overcomes the problems of ipsative data. Here, we provide a step-by-step tutorial for coding forced-choice responses, specifying a Thurstonian item response theory model that is appropriate for the design used, assessing the model's fit, and scoring individuals on psychological attributes. Estimation and scoring is performed using Mplus, and a very straightforward Excel macro is provided that writes full Mplus input files for any forced-choice design. Armed with these tools, using a forced-choice design is now as easy as using ratings.

Measurement Quality in Indicators of Compositions. A Compositional Multitrait-Multimethod Approach

Compositional data, also called multiplicative ipsative data, are common in survey research instruments in areas such as time use, budget expenditure and social networks. Compositional data are usually expressed as proportions of a total, whose sum can only be 1. Owing to their constrained nature, statistical analysis in general, and estimation of measurement quality with a confirmatory factor analysis model for multitrait-multimethod (MTMM) designs in particular are challenging tasks. Compositional data are highly non-normal, as they range within the 0-1 interval. One component can only increase if some other(s) decrease, which results in spurious negative correlations among components which cannot be accounted for by the MTMM model parameters. In this article we show how researchers can use the correlated uniqueness model for MTMM designs in order to evaluate measurement quality of compositional indicators. We suggest using the additive log ratio transformation of the data, discuss several approaches to deal with zero components and explain how the interpretation of MTMM designs differs from the application to standard unconstrained data. We show an illustration of the method on data of social network composition expressed in percentages of partner, family, friends and other members in which we conclude that the face-to-face collection mode is generally superior to the telephone mode, although primacy effects are higher in the face-to-face mode. Compositions of strong ties (such as partner) are measured with higher quality than those of weaker ties (such as other network members).

Item Response Modeling of Forced-Choice Questionnaires

Multidimensional forced-choice formats can significantly reduce the impact of numerous response biases typically associated with rating scales. However, if scored with classical methodology, these questionnaires produce ipsative data, which lead to distorted scale relationships and make comparisons between individuals problematic. This research demonstrates how item response theory (IRT) modeling may be applied to overcome these problems. A multidimensional IRT model based on Thurstone’s framework for comparative data is introduced, which is suitable for use with any forced-choice questionnaire composed of items fitting the dominance response model, with any number of measured traits, and any block sizes (i.e., pairs, triplets, quads, etc.). Thurstonian IRT models are normal ogive models with structured factor loadings, structured uniquenesses, and structured local dependencies. These models can be straightforwardly estimated using structural equation modeling (SEM) software Mplus. A number of simulation studies are performed to investigate how latent traits are recovered under various forced-choice designs and provide guidelines for optimal questionnaire design. An empirical application is given to illustrate how the model may be applied in practice. It is concluded that when the recommended design guidelines are met, scores estimated from forced-choice questionnaires with the proposed methodology reproduce the latent traits well.

Internal Validity and Reliability of Kolb’s Learning Style Inventory Version 3 (1999)

This study explores the internal validity and reliability of Kolb's revised Learning Style Inventory (LSI-2A and LSI-3) in a sample of 221 graduate and undergraduate business students. Research on the LSI is also reviewed and the implications of conducting factor analysis using ipsative data are explored. Experiential Learning Theory is presented and the concept of learning styles explained. This study largely supports prior research supporting the internal reliability of scales. Principle Component Analysis provides evidence for a 2 factor structure as hypothesized by Kolb.

Psychometric problems and issues involved with creating and using ipsative measures for selection

Data are described as ipsative if a given set of responses always sum to the same total. However, there are many properties of data collection that can give rise to different types of ipsative data. In this study, the most common type of ipsative data used in employee selection (forced‐choice ipsative data; FCID) is discussed as a special case of other types of ipsative data. Although all ipsative data contains constraints on covariance matrices (covariance‐level interdependence), FCID contains additional item‐level interdependencies as well. The psychological processes that give rise to FCID and the resultant psychometric properties are discussed. In addition, data from which both normative and ipsative responses were provided by job applicants illustrate very different patterns of correlations as well as very different selection decisions between normative, FCID and ipsatized measures.

A Direct Estimation Method on Analyzing Ipsative Data With Chan and Bentler's (1993) Method

Ipsative data (individual scores subject to a constant-sum constraint), suggested to minimize response bias, are sometimes observed in behavioral sciences. Chan and Bentler (1993, 1996) proposed a method to analyze ipsative data in a single-group case. Cheung and Chan (2002) extended the method to multiple-group analysis. However, these methods require tedious procedures on formulating within- and between-group constraints and recovering the parameter estimates and their standard errors. A direct estimation method, which is equivalent to Chan and Bentler's method with an alternative model specification, is proposed in this article. The 1st-order factor-analytic ipsative model in Chan and Bentler's method is reparameterized as a restricted 2nd-order factor-analytic model with fixed factor loading matrix reflecting the ipsative properties in the direct estimation method. The direct estimation method can be easily extended to test measurement invariance properties in multiple-group analysis. Issues related to ipsative models are also addressed.