Forced-choice Format Research Articles

Not Liking the Likert? A Rasch Analysis of Forced-choice Format and Usefulness in Survey Design

We have less time and focus than ever before, while the demand for attention is increasing. Therefore, it is no surprise that when answering questionnaires, we often choose to strongly agree or be neutral, producing problematic and unusable data. The current study investigated forced-choice (ipsative) format compared to the same questions on a Likert-type as a viable alternative. An established motivation questionnaire was administered in two versions, forced-choice and Likert-type, to 1088 first-year engineering students. Descriptive, non-parametric statistics and Rasch measurement models were applied to assess usefulness, validity and reliability. Results: The ipsative version had a higher response rate, less missing data, and the motivations emerged more clearly. Evidence for the reliability and validity of the forced-choice version was excellent. The forced-choice format is recommended as an alternative to the Likert types when collecting human or social survey data.

Though Forced, Still Valid: Examining the Psychometric Performance of Forced-Choice Measurement of Personality in Children and Adolescents.

Unveiling the roles personality plays during childhood and adolescence necessitates its accurate measurement, commonly using traditional Likert-type (LK) scales. However, this format is susceptible to various response biases, which can be particularly prevalent in children and adolescents, thus likely undermining measurement accuracy. Forced-choice (FC) scales appear to be a promising alternative because they are largely free from these biases by design. However, some argue that the FC format may not perform satisfactorily in children and adolescents due to its complexity. Little empirical evidence exists regarding the suitability of the FC format for children and adolescents. As such, the current study examined the psychometric performance of an FC measure of the Big Five personality factors in three children and adolescent samples: 5th to 6th graders (N = 428), 7th to 8th graders (N = 449), and 10th to 11th graders (N = 555). Across the three age groups, the FC scale demonstrated a better fit to the Big Five model and better discriminant validity in comparison to the LK counterpart. Personality scores from the FC scale also converged well with those from the LK scale and demonstrated high reliability as well as sizable criterion-related validity. Furthermore, the FC scale had more invariant statements than its LK counterpart across age groups. Overall, we found good evidence showing that FC measurement of personality is suitable for children and adolescents.

Can Forced-Choice Response Format Reduce Faking of Socially Aversive Personality Traits?

Self-report assessments are the standard for personality measurement, but motivated respondents are able to manipulate or fake their responses to typical Likert scale self-report. Although progress has been made in research seeking to reduce faking, most of it has focused on normative personality traits such as those measured by the five factor model. The measurement of socially aversive personality (e.g., the Dark Triad) is less well-researched. The negative aspects of socially aversive traits increase the opportunity and motivation of respondents to fake typical single-stimulus self-report assessments underscoring the need for faking resistant response formats. A possible way to reduce faking that has been explored in basic personality research is the use of the forced-choice response format. This study applied this method to socially aversive traits and illustrated best practices to create new multidimensional forced-choice and single-stimulus measures of socially aversive personality traits. Results indicated that participants were able to artificially alter their scores when asked to respond like an ideal job applicant, and counter to expectations, the forced-choice format did not decrease faking. Our results indicate that even when best practices are followed, forced-choice format is not a panacea for respondent faking.

Dialect identification, intelligibility ratings, and acceptability ratings of dysarthric speech in two American English dialects

ABSTRACT The current study explored the intelligibility and acceptability ratings of dysarthric speakers with African American English (AAE) and General American English (GAE) dialects by listeners who identify as GAE or AAE speakers, as well as listener ability to identify dialect in dysarthric speech. Eighty-six listeners rated the intelligibility and acceptability of sentences extracted from a passage read by speakers with dysarthria. Samples were used from the Atlanta Motor Speech Disorders Corpus and ratings were collected via self-report. The listeners identified speaker dialect in a forced-choice format. Listeners self-reported their dialect and exposure to AAE. AAE dialect was accurately identified in 63.43% of the the opportunities; GAE dialect was accurately identified in 70.35% of the opportunities. Listeners identifying as AAE speakers rated GAE speech as more acceptable, whereas, listeners identifying as GAE speakers rated AAE speech as more acceptable. Neither group of listeners demonstrated a difference in intelligibility ratings. Exposure to AAE had no effect on intelligibility or acceptability ratings. Listeners can identify dialect (AAE and GAE) with a better than chance degree of accuracy. One’s dialect may have an effect on intelligibility and acceptability ratings. Exposure to a dialect did not affect listener ratings of intelligibility or acceptability.

Detecting Careless Responding in Multidimensional Forced-Choice Questionnaires.

The multidimensional forced-choice (MFC) format is an alternative to rating scales in which participants rank items according to how well the items describe them. Currently, little is known about how to detect careless responding in MFC data. The aim of this study was to adapt a number of indices used for rating scales to the MFC format and additionally develop several new indices that are unique to the MFC format. We applied these indices to a data set from an online survey (N = 1,169) that included a series of personality questionnaires in the MFC format. The correlations among the careless responding indices were somewhat lower than those published for rating scales. Results from a latent profile analysis suggested that the majority of the sample (about 76-84%) did not respond carelessly, although the ones who did were characterized by different levels of careless responding. In a simulation study, we simulated different careless responding patterns and varied the overall proportion of carelessness in the samples. With one exception, the indices worked as intended conceptually. Taken together, the results suggest that careless responding also plays an important role in the MFC format. Recommendations on how it can be addressed are discussed.

Inventário de Suporte para Competências Socioemocionais: Versão Escolha Forçada

Abstract Socioemotional skills are essential for students’ development at school. However, school support has barely generated debates. We aimed to analyze the evidence of the internal structure and convergent-discriminant validity of the scores of the forced-choice version of the Inventory of Supporting for Socio-Emotional Skills (ISSE). In total, 443 middle and high school students (54% of whom were girls) participated in this study. Results indicated that the revised instrument was consistent with its two-dimension theoretical model (support and demand); the forced-choice version showed an appropriate internal structure and no bias by acquiescence. Moreover, based on a multitrait-multimethod matrix, the scores of the Likert forced-choice version showed convergent and discriminant validity. This study contributes to the literature by proposing a multidimensional heteroevalutive instrument to assess support for socioemotional development in the forced-choice format, which reduces response bias.

Linear Factor Analytic Thurstonian Forced-Choice Models: Current Status and Issues.

Thurstonian forced-choice modeling is considered to be a powerful new tool to estimate item and person parameters while simultaneously testing the model fit. This assessment approach is associated with the aim of reducing faking and other response tendencies that plague traditional self-report trait assessments. As a result of major recent methodological developments, the estimation of normative trait scores has become possible in addition to the computation of only ipsative scores. This opened up the important possibility of comparisons between individuals with forced-choice assessment procedures. With item response theory (IRT) methods, a multidimensional forced-choice (MFC) format has also been proposed to estimate individual scores. Customarily, items to assess different traits are presented in blocks, often triplets, in applications of the MFC, which is an efficient form of item presentation but also a simplification of the original models. The present study provides a comprehensive review of the present status of Thurstonian forced-choice models and their variants. Critical features of the current models, especially the block models, are identified and discussed. It is concluded that MFC modeling with item blocks is highly problematic and yields biased results. In particular, the often-recommended presentation of blocks with items that are keyed in different directions of a trait proves to be counterproductive considering the goal to reduce response tendencies. The consequences and implications of the highlighted issues are further discussed.

Moving beyond Likert and Traditional Forced-Choice Scales: A Comprehensive Investigation of the Graded Forced-Choice Format

The graded forced-choice (FC) format has recently emerged as an alternative that may preserve the advantages and overcome the issues of the dichotomous FC measures. The current study presented the first large-scale evaluation of the performance of three types of FC measures (FC2, FC4 and FC5 with 2, 4 and 5 response options, respectively) and compared their performance to their Likert (LK) counterparts (LK2, LK4, and LK5) on (1) psychometric properties, (2) respondent reactions, and (3) susceptibility to response styles. Results showed that, compared to LK measures with the same number of response options, the three FC scales provided better support for the hypothesized factor structure, were perceived as more faking-resistant and cognitive demanding, and were less susceptible to response styles. FC4/5 and LK4/5 demonstrated similarly good reliability, while LK2 provided more reliable scores than FC2. When compared across the three FC measures, FC4 and FC5 displayed comparable psychometric performance and respondent reactions. FC4 exhibited a moderate presence of extreme response style, while FC5 had a weak presence of both extreme and middle response styles. Based on these findings, the study recommends the use of graded FC over dichotomous FC and LK, particularly FC5 when extreme response style is a concern.

Improving Reactions to Forced-Choice Personality Measures in Simulated Job Application Contexts Through the Satisfaction of Psychological Needs

There is a wealth of evidence justifying the use of personality assessments for selection. Nonetheless, some reluctance to use these assessments stems from their perceived vulnerability to response distortion (i.e., faking) and the somewhat negative applicant reactions they elicit, when compared to other assessments. Adopting a forced-choice personality assessment format appears to alleviate the former problem but exacerbates the latter. In this study, we introduce basic psychological needs as a theoretical foundation to develop interventions to improve reactions to forced-choice personality assessments. We propose that the forced-choice format impedes respondents’ desire to respond to items in a preferred way, interfering with autonomy need satisfaction, and constrains respondents’ opportunity to show their capabilities, interfering with competence need satisfaction. In this pre-registered between-subjects experiment (N = 1565), we investigated two modifications to a ranked forced-choice personality questionnaire and compared these to traditional forced-choice and single-stimulus (Likert) formatted questionnaires. One modification, where participants could write a free-text response following the assessment, did not show significant effects on reactions. The second modification allowed participants to view all items they had ranked last (first) and then identify any the participant believed in fact described them well (poorly). That modification positively affected perceived autonomy- and competence-support, and fairness perceptions, bridging approximately half of the gap between reactions to forced-choice and single-stimulus assessment formats. This study suggests that a modification to forced-choice personality questionnaires may improve applicant reactions and that basic psychological needs theory may be a fruitful lens through which to further understand reactions to assessments.

Quasi-ipsative Forced-Choice Personality Inventories and the Control of Faking: The Biasing Effects of Transient Error

To date, experimental research on the effect of faking on personality measures has used two types of designs: within-subject designs and between-subjects designs. None of these designs permit us to control for the effects of transient error on faking. Using a Latin-square design (LSD), the current study examines the effects of faking on the Big Five as assessed by a quasi-ipsative forced-choice (FC) personality inventory. LSD is a type of experimental design that simultaneously permits us to control for between-subject differences, within-subject variability, and transient error. The sample consisted of 246 participants (four experimental groups, assessed twice, 2-3 week interval). The results showed that (1) faking effect size can be largely attributed to transient error and (2) the quasi-ipsative FC format shows great resistance to faking behavior. The average effect size (Cohen’s d) for the Big Five was 0.21, 0.12, and 0.09 for observed faking, transient error, and true faking, respectively. On average, 62% of observed faking effect size can be attributed to transient error. To conclude, we discuss the implications of these findings for the research and practice of personnel selection.

Scoring a forced-choice image-based assessment of personality: A comparison of machine learning, regression, and summative approaches

Recent years have seen rapid advancements in the way that personality is measured, resulting in a number of innovative predictive measures being proposed, including using features extracted from videos and social media profiles. In the context of selection, game- and image-based assessments of personality are emerging, which can overcome issues like social desirability bias, lack of engagement and low response rates that are associated with traditional self-report measures. Forced-choice formats, where respondents are asked to rank responses, can also mitigate issues such as acquiescence and social desirability bias. Previously, we reported on the development of a gamified forced-choice image-based assessment of the Big Five personality traits created for use in selection, using Lasso regression for the scoring algorithms. In this study, we compare the machine-learning-based Lasso approach to ordinary least squares regression, as well as the summative approach that is typical of forced-choice formats. We find that the Lasso approach performs best in terms of generalisability and convergent validity, although the other methods have greater discriminate validity. We recommend the use of predictive Lasso regression models for scoring forced-choice image-based measures of personality over the other approaches. Potential further studies are suggested.

The Effectiveness of the Extended Forced-Choice Format as an Alternative to the Likert Scale Format in Improving the Factor Structure of the Psychological Scales: فاعلية استخدام صورة الاختيار القسري المطوّل كأسلوب بديل لصورة "ليكرت" في تحسين البناء العاملي للمقاييس النفسية

The study aimed to examine the relative effectiveness of using the extended forced-choice format as an alternative to the Likert scale format in improving the factor structure of the psychological scales. The neuroticism factor, which is one of the five factors of the Jordanian version of NEO personality Inventory was used as an example of the psychological scales. The Descriptive correlational approach was used to achieve the objectives of the study. The data were collected from (532) available undergraduate male and female students of different academic levels and different majors who are enrolled in different sections and different colleges in four Jordanian Universities, where the selected sections were randomly assigned into two groups (Likert format, and extended forced-choice format). The results of the exploratory as well as the confirmatory factor analyses of the scale in each of its two forms revealed that the extended forced-choice format produced better construct validity indices compared to the Likert scale format, where it was found that the factor structure of the Likert scale format is contaminated; that is, the scale does not measure a unidimensional trait as it is supposed. The results also indicated a better stability index (Cronbach-Alpha) for the extended forced-choice format which was found to be (0.847) compared to the stability index of the Likert scale format which was found to be (0.705). It was recommended to conduct more studies on using the extended forced-choice format as an alternative to the Likert scale format in other psychological scales, especially if the scale includes reversed worded items besides positively worded items.

A closer look at grandiose narcissism: A revised version of the narcissistic personality inventory using a single-stimulus Likert format among Chinese adolescents and adults

Despite its extensive use and profound impact, the Narcissistic Personality Inventory (NPI; Raskin & Terry, 1988) has been continuously criticized on its forced-choice format, questionable contents and unstable structure. This research mainly aimed to (a) untangle main issues of the NPI and develop a revised version (NPI-r; NPI-rb as its brief variant), (b) offer insights into psychometric properties of the Likert-style NPI and the nuanced structure of grandiose narcissism, and (c) provide a closer look at grandiose narcissists by comparing their self- versus informant-report profiles. Bi-factor exploratory structural equation modelling and comparison of nomological networks of the NPI versions were performed with two Chinese samples (N1 = 317 adults, age M = 26.9, SD = 6.1, 48% male; N2 = 269 adolescents, age M = 16.8, SD = 0.7, 49% male, with 142 informants). Results recommended the NPI-r and NPI-rb as better versions with improved psychometric properties and content validity. This research also underscored the unique roles of specific narcissistic traits, and found nomological networks of NPI versions in China consistent with previous evidence using Caucasian samples. Notably, the discrepancy between narcissists' self-report and informant-perceived profiles suggested some caution about the credibility of self-report data in narcissism studies.

Radiotherapy as treatment option in biliary cancer patients: a national survey of AIRO (Italian Association of Radiation Oncology) Gastroenterology Group.

BackgroundThe aim is to find out how many radiation oncology centres treat biliary duct carcinoma (BDC), what treatments they offer and whether they would be interested in developing prospective trials.Materials and methodsA questionnaire was posted to all 220 Italian Radiation Oncology Centres. The survey consisted of 31 eligibility questions in a combination of multiple and forced choice formats addressing the following parameters: characteristics of the centre, numbers of BDC patients treated, treatment options, radiotherapy parameters (target definition, schedule, technique, dose constraints) and interest in developing future randomized trials.ResultsNo major differences emerged in BDC management, whatever the site, and whether it was resectable or not. Discrepancies in routine clinical practice were, however, observed with lack of agreement on expansion margins, dose constraints and treatment schedules for the stereotactic technique and palliative treatments.ConclusionsThe present survey attempted to fill the gaps in the role of radiotherapy in patients with BDC. Since lack of prospective randomized studies and disease rarity have mitigated against an evidence-based approach, patients with BDC should be enrolled in prospective studies. The above-mentioned results should also emphasize the need to combine analysis of treatment results from all Italian centres in order to create predictive models.

Modeling Faking in the Multidimensional Forced-Choice Format: The Faking Mixture Model

The multidimensional forced-choice (MFC) format has been proposed to reduce faking because items within blocks can be matched on desirability. However, the desirability of individual items might not transfer to the item blocks. The aim of this paper is to propose a mixture item response theory model for faking in the MFC format that allows to estimate the fakability of MFC blocks, termed the Faking Mixture model. Given current computing capabilities, within-subject data from both high- and low-stakes contexts are needed to estimate the model. A simulation showed good parameter recovery under various conditions. An empirical validation showed that matching was necessary but not sufficient to create an MFC questionnaire that can reduce faking. The Faking Mixture model can be used to reduce fakability during test construction.

Intelligibility of dysphonic speech in auralized classrooms.

Voice disorders can reduce the speech intelligibility of affected speakers. This study evaluated the effect of noise, voice disorders, and room acoustics on vowel intelligibility, listening easiness, and the listener's reaction time. Three adult females with dysphonia and three adult females with normal voice quality recorded a series of nine vowels of American English in /h/-V-/d/ format (e.g., "had"). The recordings were convolved with two oral-binaural impulse responses acquired from measurements in two classrooms with 0.4 and 3.1 s of reverberation time, respectively. The stimuli were presented in a forced-choice format to 29 college students. The intelligibility and the listening easiness were significantly higher in quiet than in noisy conditions, when the speakers had normal voice quality compared to a dysphonic voice, and in low reverberated environments compared to high reverberated environments. The response time of the listener was significantly longer for speech presented in noisy conditions compared to quiet conditions and when the voice was dysphonic compared with healthy voice quality.

Assessing the effect of questionnaire design on unit and item-nonresponse: evidence from an online experiment

ABSTRACT Questionnaire design choices, such as handling ‘do not know’ answers or using check all that apply questions may impact the level of item- and unit nonresponse. We used experimental data from an online panel and the questions of ESS to examine how applying forced answering, offering DK/NA options and specific question formats (CATA vs. forced choice format) impact the level of nonresponse and how respondent characteristics correlate with these effects. The results suggest that neither forcing respondents nor offering an explicit NR option is advisable, as these options strongly increase nonresponse. Rather a skipping-allowed design is a more reasonable choice. Those who were interested in the topic of the survey were far less sensitive to NR handling. Developing adaptive questionnaire designs based on topic interest can be a promising direction for future research.

Developmental differences in reactivation underlying self-derivation of new knowledge through memory integration

Self-derivation of novel facts through integration of memory content is fundamental to acquiring new knowledge and a means of building a semantic knowledge base. It involves combining memory content acquired across separate episodes of learning to generate new knowledge that was not explicitly taught in either episode. To self-derive, one needs to reactivate earlier learned memory content upon exposure to related content and then integrate the learning episodes. Previous research found developmental differences in the conditions under which integration occurs. Adults spontaneously integrate whereas 7- to 9-year-old children seemingly integrate only upon direct tests that verbally prompt for integration. Yet it is unclear whether children engage in the preliminary process of reactivation prior to the direct tests. To address this gap in the current research, we developed an eye-tracking paradigm and tested whether adults and 7- to 9-year-old children engage in the process of reactivation prior to direct tests. The direct tests verbally prompted for integration of memory content requiring self-derivation through both open-ended and forced-choice formats. Both adults and children engaged in reactivation prior to the direct tests. The extent of their reactivation predicted their performance on the direct tests. However, adults showed stronger evidence of reactivation and performed better than children on the direct tests. This work contributes to understandings of developmental differences in the underlying processes involved in the development of new knowledge.

Identifying Faking on Forced-Choice Personality Items Using Mouse Tracking

This research utilizes mouse tracking as a potential behavioral method to examine cognitive processes underlying faking on forced-choice personality inventories. Mouse tracking is a method from social categorization research that captures a variety of metrics related to motor movements, which are linked to cognitive processing. To explore the utility of this method, we examined differences in the mouse tracking metrics of those instructed to respond honestly or to fake. Our findings show that there is a distinguishable difference in the behavioral response of those who are faking when responding to pairs of personality descriptors presented in a forced-choice format compared to those who are responding honestly. Implications and contributions of this study include insights into the cognitive processing that can occur while responding to personality items when respondents are faking and a demonstration of how mouse tracking methods can be used to detect faking.

Reviewing the Structure of Kolb’s Learning Style Inventory From Factor Analysis and Thurstonian Item Response Theory (IRT) Model Approaches

Kolb’s Learning Style Inventory (LSI) continues to generate a great debate among researchers, given the contradictory evidence resulting from its psychometric properties. One primary criticism focuses on the artificiality of the results derived from its internal structure because of the ipsative nature of the forced-choice format. This study seeks to contribute to the resolution of this debate. A short version of Kolb’s LSI with a forced-choice format and an additional inventory scored on a Likert scale was completed by a sample of students of the University Católica del Norte in Antofagasta, Chile. The data obtained from the two forms of the reduced version of the LSI were compared using principal component analysis, confirmatory factor analysis, and the Thurstonian Item Response Theory model. The results support the hypothesis of the existence of four learning mode dimensions. However, they do not support the existence of the learning styles as proposed by Kolb, indicating that said reports are the product of the artificial structure generated by the ipsative forced-choice format .