Effects Of Response Format Research Articles

The Effect of Response Formats on Response Style Strength

Abstract: Many researchers use self-report data to examine abilities, personalities, or attitudes. At the same time, there is a widespread concern that response styles, such as the tendency to give extreme, midscale, or acquiescent responses, may threaten data quality. As an alternative to post hoc control of response styles using psychometric models, a priori control using specific response formats may be a means to reduce biasing response style effects in self-report data in day-to-day research practice. Previous research has suggested that response styles were less influential in a Drag-and-Drop (DnD) format compared to the traditional Likert-type format. In this article, we further examine the advantage of the DnD format, test its generalizability, and investigate its underlying mechanisms. In two between-participants experiments, we tested different versions of the DnD format against the Likert format. We found no evidence for reduced response style influence in any of the DnD conditions, nor did we find any difference between the conditions in terms of the validity of the measures to external criteria. We conclude that adaptations of response formats, such as the DnD format, may be promising, but require more thorough examination before recommending them as a means to reduce response style influence in psychological measurement.

Effects of response format on achievement and aptitude assessment results: multi-level random effects meta-analyses.

Psychological achievement and aptitude tests are fundamental elements of the everyday school, academic and professional lives of students, instructors, job applicants, researchers and policymakers. In line with growing demands for fair psychological assessment tools, we aimed to identify psychometric features of tests, test situations and test-taker characteristics that may contribute to the emergence of test bias. Multi-level random effects meta-analyses were conducted to estimate mean effect sizes for differences and relations between scores from achievement or aptitude measures with open-ended (OE) versus closed-ended (CE) response formats. Results from 102 primary studies with 392 effect sizes revealed positive relations between CE and OE assessments (mean r = 0.67, 95% CI [0.57; 0.76]), with negative pooled effect sizes for the difference between the two response formats (mean d av = -0.65; 95% CI [-0.78; -0.53]). Significantly higher scores were obtained on CE exams. Stem-equivalency of items, low-stakes test situations, written short answer OE question types, studies conducted outside the United States and before the year 2000, and test-takers' achievement motivation and sex were at least partially associated with smaller differences and/or larger relations between scores from OE and CE formats. Limitations and the results' implications for practitioners in achievement and aptitude testing are discussed.

Effect of response format on tendency of increasing middle category response

Effect of response format on tendency of increasing middle category response

Assessing L2 listening at a Japanese university: Effects of input type and response format

Large-scale and classroom-based tests of L2 listening comprehension typically employ audio-only listening texts and a multiple-choice response format. However, the effect of these test task characteristics on test-taker performance remains elusive, and begs the question as to whether they are the best option in terms of construct validity. Furthermore, the interaction between these test task characteristics and their potential joint effect on performance has not been sufficiently explored. To address this gap, a study was conducted at a Japanese university which investigated the effect of input type (audio-only vs. video) and response format (multiple-choice vs. short answer) on L2 listening test performance. Participants were divided into four groups to take an academic listening test with one of four combinations of input and response format: (1) audio-only with multiple-choice questions; (2) video with multiple-choice questions; (3) audio-only with short answer questions; and (4) video with short answer questions. Results of a 2 x 2 factorial ANOVA revealed a statistically significant effect of response format on test-taker performance. No significant effect for input type was found and no significant interaction among the variables was detected. Results suggest that visual input and audiovisual literacy need to be more clearly articulated within the construct definition of academic listening, if they are to be included at all.

Effects of Response Format on Psychometric Properties and Fairness of a Matrices Test: Multiple Choice vs. Free Response

Reasoning is regarded to be an essential facet of fundamental cognitive abilities. As examinee characteristics may affect performance in Reasoning tests, concern about maintaining fairness is expressed. The purpose of the current study was to examine effects of response format on psychometric properties and fairness of a matrices test according to examinee´s sex, risk propensity, and test anxiety. A total of 433 German-speaking pupils (aged 14 to 20) were randomly assigned to either a multiple choice or a free response version of the same 25-item test. Data analysis yielded Rasch-homogeneous 23-item versions, with higher reliability, but lower criterion validity for the free response test. No interactions between response format and sex, test anxiety, or risk propensity were revealed, but a significant main effect of sex: men out-performed women in reasoning irrespective of response format. Results are discussed with reference to attributes of the test situation and sample characteristics.

Likert, slider, or text? reassurances about response format effects

Few experiments in the domain of addictive behaviors address how response format might impact self-reports of drug use, drug problems, or their correlates. In a preregistered study, participants (N = 1,546) reported on their drinking habits, problematic drinking behaviors, cigarette smoking, Big Five traits, and some additional health behaviors using either a Likert scale, slider, or free-response text entry format. Response format exerted little observable effect on mean values despite the large sample size. Exploratory analyses revealed that the magnitude of the association between frequency of drinking and problematic alcohol use was substantially and reliably greater when participants responded on a slider scale (vs. Likert or text entry), and that responses to items inquiring about longer-term estimates (e.g. number of days per year one exercises) were smaller when participants reported in the text-entry format. Likert-style measures appeared easiest to use and generated few problematic associations with other variables. Although variation in response format therefore does not appear to pose a large threat to the absolute value of many variables (with the possible exception of text entry deflating some types of responses), associations among variables may inflate when participants use sliders instead of Likert-type scales or text entry.

Effect of response format on the correlation of survey items

Effect of response format on the correlation of survey items

Phonological span in children and adults: Does response format matter?

In contrast to classical phonological span tasks, which require verbal recall, those used in self-reliant, group-administrable working memory measurement contain a visuospatial response format. As a consequence, these tasks involve recoding, executive, and visual search requirements in addition to encoding and storage processes. To examine age-dependent effects of these special requirements, tablet-based word and digit span tasks with a visuospatial response format were compared to corresponding tasks with a verbal response format in first and second graders, third and fourth graders, and adults (N = 105). Whereas children’s performance was impaired by the visuospatial response format, adults’ performance was not. Whether the arrangement of stimuli in the visuospatial response format—either variable or fixed—has differential effects on performance in children and adults (N = 434) was examined in a second experiment. No substantial differences between variable and fixed response format were found in any of the age groups. Possible underlying mechanisms for age-dependent effects of response format and the role of executive functions and visual search in automatized phonological span tasks are discussed.

To bin or not to bin? A comparison of symptom frequency response formats in the assessment of health-related quality of life.

The goal of this study is to compare three different types of retrospective frequency response formats on the Healthy Days Symptoms Module (HDSM). Responses are compared in terms of intra-individual consistency, psychometric value, and participant feedback about each type of response format. Respondents each completed three versions of the HDSM, where items were framed to elicit an open-ended frequency, a fixed choice frequency, or a vague quantifier response. Traditional reliability statistics were used to evaluate intra-individual consistency. Differential item functioning (DIF) was used to test for response format effects, and item response theory (IRT) scale scores and standard errors were computed across the three forms to compare psychometric value. Linear mixed modeling was used to examine the associations of IRT scale scores across response formats with respondent characteristics. People are largely consistent in how they respond to items about their health, regardless of the response format, and no DIF was detected between response formats. The IRT scores computed from the "# of days" frequency response formats tend to have better measurement precision than those from vague quantifiers. Open-ended frequencies capture a greater span of individual differences for people reporting fewer symptoms; however, little measurement precision is lost in collapsing the frequencies into categories. Both the open-ended and fixed choice frequency response formats offer more measurement precision than vague quantifiers. While the open-ended frequency response format may capture more individual differences, respondents tend to report more difficulty with exact frequency recall, and thus, prefer the fixed choice frequency format.

Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test

The Cognitive Reflection Test, measuring intuition inhibition and cognitive reflection, has become extremely popular because it reliably predicts reasoning performance, decision-making, and beliefs. Across studies, the response format of CRT items sometimes differs, based on the assumed construct equivalence of tests with open-ended versus multiple-choice items (the equivalence hypothesis). Evidence and theoretical reasons, however, suggest that the cognitive processes measured by these response formats and their associated performances might differ (the nonequivalence hypothesis). We tested the two hypotheses experimentally by assessing the performance in tests with different response formats and by comparing their predictive and construct validity. In a between-subjects experiment (n = 452), participants answered stem-equivalent CRT items in an open-ended, a two-option, or a four-option response format and then completed tasks on belief bias, denominator neglect, and paranormal beliefs (benchmark indicators of predictive validity), as well as on actively open-minded thinking and numeracy (benchmark indicators of construct validity). We found no significant differences between the three response formats in the numbers of correct responses, the numbers of intuitive responses (with the exception of the two-option version, which had a higher number than the other tests), and the correlational patterns of the indicators of predictive and construct validity. All three test versions were similarly reliable, but the multiple-choice formats were completed more quickly. We speculate that the specific nature of the CRT items helps build construct equivalence among the different response formats. We recommend using the validated multiple-choice version of the CRT presented here, particularly the four-option CRT, for practical and methodological reasons. Supplementary materials and data are available at https://osf.io/mzhyc/ .

Encounter norms among visitors at a national park in Turkey

This article examined response format effects on norm prevalence (i.e., the percent of individuals who could specify a norm). Data were collected in Dilek Peninsula Büyük Menderes Delta National Park, in Turkey. Respondents were randomly assigned to one of two conditions. In the “semi-open” response format, respondents (n = 458) “wrote a number” for an acceptable number of visitor encounters. In the “closed” format, respondents (n = 459) “circled a number” of acceptable encounters, along a range of possible responses. Results showed that encounter norm prevalence was significantly and consistently higher for the closed format of the survey, as compared to the semi-open version. In addition, among those reporting a norm, the average tolerance levels were statistically higher in the semi-open format. Implications for applying the normative approach across different cultures are discussed.

Assessing Boundary Conditions of the Testing Effect: On the Relative Efficacy of Covert vs. Overt Retrieval.

Repeated testing during learning often improves later memory, which is often referred to as the testing effect. To clarify its boundary conditions, we examined whether the testing effect was selectively affected by covert (retrieved but not articulated) or overt (retrieved and articulated) response format. In Experiments 1 and 2, we compared immediate (5 min) and delayed (1 week) cued recall for paired associates following study-only, covert, and overt conditions, including two types of overt articulation (typing and writing). A clear testing effect was observed in both experiments, but with no selective effects of response format. In Experiments 3 and 4, we compared covert and overt retrieval under blocked and random list orders. The effect sizes were small in both experiments, but there was a significant effect of response format, with overt retrieval showing better final recall performance than covert retrieval. There were no significant effects of blocked vs. random list orders with respect to the testing effect produced. Taken together, these findings suggest that, under specific circumstances, overt retrieval may lead to a greater testing effect than that of covert retrieval, but because of small effect sizes, it appears that the testing effect is mainly the result of retrieval processes and that articulation has fairly little to add to its magnitude in a paired-associates learning paradigm.

Assessing Response Format Effects on the Scaling of Marketing Stimuli

Multi-item rating scales are the accepted solution for achieving reliable and valid measures in the social sciences. Issues not fully resolved include the optimal number of response categories, choice of semantic rating versus Likert form, and the appropriateness of mixing positively and negatively expressed items. While there is considerable empirical research on these issues, it addresses the scaling of respondents and is yet to produce consensus as to the most appropriate practice. In marketing, multi-item scales are not only used to scale consumer respondents, they are used to scale marketing stimuli. This article examines these response format issues when the primary objective is to scale marketing stimuli rather than consumers using generalisability theory criteria for data quality. G-study website assessment data using different response formats are used to compare their effects on the observed variance components and G-coefficients for websites. Conclusions are drawn for the most appropriate response format to use in marketing studies that scale marketing stimuli.

La medida de la comprensión emocional con el modelo de Rasch

<div class="page" title="Page 1"><div class="section"><div class="layoutArea"><div class="column"><p><span>El objetivo fue construir ítems de comprensión emocional desde la perspectiva del Modelo de Rasch, comparando mediante un experimento aleatorizado los formatos de respuesta verbal/imagen. Participaron 204 sujetos de población general. Se contrastó el efecto del formato (verbal/imagen) y del género en las medidas Rasch de comprensión de emociones de los participantes y se analizó el efecto de la distancia social (cerca/lejos) sobre la dificultad Rasch de los ítems. El formato de respuesta tuvo un efecto significativo sobre las medidas independientemente del género: el formato verbal resultó más fácil que el de imagen. El efecto del género fue significativo, favorable a las mujeres. Los ítems que describían situaciones con receptores cercanos resultaron más fáciles que los que incluían receptores lejanos. </span></p></div></div></div></div>

Response Format Effects in Encounter Norm Questions

ABSTRACTSurvey response format has been shown to influence norm prevalence (percentage reporting a norm) and the numerical value of the reported norm. This article summarizes an experiment where respondents were randomly assigned to one of two conditions. In the semi-open response format treatment, respondents (n = 817) “wrote in a number” for an acceptable number of visitor encounters. In the closed format treatment, individuals (n = 826) “circled a number” of acceptable encounters along a range of possible responses. Hypothesis 1 predicted that norm prevalence would be higher for the closed format than the semi-open. Hypothesis 2 predicted that the mean tolerance level would differ in the two treatments. Results supported hypothesis 1. The percent reporting a norm was statistically higher in the closed versus the semi-open treatment. Results failed to support hypothesis 2. The average tolerance levels for the closed and semi-open formats were statistically equivalent.

The Effect of Response Format on Test Performance: Problems and Possible Solution (An Alternative Grammaticality Judgement Rating Scale)

The Effect of Response Format on Test Performance: Problems and Possible Solution (An Alternative Grammaticality Judgement Rating Scale) Naif Alsaedi Abstract Intuitional data elicited by means of grammaticality judgement (GJ) tasks are affected by a diverse number of linguistic and non-linguistic factors including the type of the measurement scales and the response formats used (see Schütze, 1996; Sorace, 1996). The problems associated with the rating scales vary according to the type of the rating scale used - whether absolute or comparative. Such problems pose a serious challenge to linguists to find an alternative rating scale that can overcome these methodological problems and pitfalls. The only way to do so is by using a new rating scale which allows sharp lines to be drawn between the learner's certainty, doubt and lack of knowledge reflected in his or her judgements. The four-point scale in its new format proposed in this article (i.e., clearly correct, clearly incorrect, possibly incorrect and do not know) managed to map the territory between the three possibilities that capture a learner's feelings towards any given sentence. What is unique about this rating scale compared to others commonly used in grammaticality judgments (i.e., Coppetiers, 1987, Schachter and Yip, 1990; Schachter, 1990; Gass, 1994) is how it works and how the data obtained by means of this scale can be marked and scored. The purpose of the present article is to introduce this scale and to discuss from a logical point of view the extent to which it can produce reliable and valid data that reflect second language (L2) learner’s interlanguage knowledge. Full Text: PDF DOI: 10.15640/jehd.v5n4a21

The Effect of Response Format on the Psychometric Properties of the Narcissistic Personality Inventory: Consequences for Item Meaning and Factor Structure.

The Narcissistic Personality Inventory (NPI) is currently the most widely used measure of narcissism in social/personality psychology. It is also relatively unique because it uses a forced-choice response format. We investigate the consequences of changing the NPI's response format for item meaning and factor structure. Participants were randomly assigned to one of three conditions: 40 forced-choice items (n = 2,754), 80 single-stimulus dichotomous items (i.e., separate true/false responses for each item; n = 2,275), or 80 single-stimulus rating scale items (i.e., 5-point Likert-type response scales for each item; n = 2,156). Analyses suggested that the "narcissistic" and "nonnarcissistic" response options from the Entitlement and Superiority subscales refer to independent personality dimensions rather than high and low levels of the same attribute. In addition, factor analyses revealed that although the Leadership dimension was evident across formats, dimensions with entitlement and superiority were not as robust. Implications for continued use of the NPI are discussed.

Effect of response format for clinical vignettes on reporting quality of physician practice

BackgroundClinical vignettes have been used widely to compare quality of clinical care and to assess variation in practice, but the effect of different response formats has not been extensively evaluated. Our objective was to compare three clinical vignette-based survey response formats – open-ended questionnaire (A), closed-ended (multiple-choice) questionnaire with deceptive response items mixed with correct items (B), and closed-ended questionnaire with only correct items (C) – in rheumatologists' pre-treatment assessment for tumor-necrosis-factor (TNF) blocker therapy.MethodsStudy design: Prospective randomized study. Setting: Rheumatologists attending the 2004 French Society of Rheumatology meeting. Physicians were given a vignette describing the history of a fictitious woman with active rheumatoid arthritis, who was a candidate for therapy with TNF blocking agents, and then were randomized to receive questionnaire A, B, or C, each containing the same four questions but with different response formats, that asked about their pretreatment assessment. Measurements: Long (recommended items) and short (mandatory items) checklists were developed for pretreatment assessment for TNF-blocker therapy, and scores were expressed on the basis of responses to questionnaires A, B, and C as the percentage of respondents correctly choosing explicit items on these checklists. Statistical analysis: Comparison of the selected items using pairwise Chi-square tests with Bonferonni correction for variables with statistically significant differences.ResultsData for all surveys distributed (114 As, 118 Bs, and 118 Cs) were complete and available for analysis. The percentage of questionnaire A, B, and C respondents for whom data was correctly complete for the short checklist was 50.4%, 84.0% and 95.0%, respectively, and was 0%, 5.0% and 5.9%, respectively, for the long version. As an example, 65.8%, 85.7% and 95.8% of the respondents of A, B, and C questionnaires, respectively, correctly identified the need for tuberculin skin test (p < 0.0001).ConclusionIn evaluating clinical practice with use of a clinical vignette, a multiple-choice format rather than an open-ended format overestimates physician performance. The insertion of deceptive response items mixed with correct items in closed-ended (multiple-choice) questionnaire failed to avoid this overestimation.

Effect of response format on endorsement of eating disordered attitudes and behaviors

The present study was designed to compare response rates on a standard self-report questionnaire that was nominally anonymous to an unmatched count questionnaire that allowed for true anonymity in responding. Four hundred and fifty-four college students were asked about several topics, including attitudes towards weight and shape, dieting, and eating disordered behavior using one of two response formats; either a standard questionnaire in true-false format or an unmatched count questionnaire that did not require participants to directly answer sensitive questions. Both males and females had significantly different rates of endorsement between the two methods of assessment on the majority of the eating-related questions. Response format and degree of anonymity affect endorsement of eating-related thoughts and behaviors. Understanding response bias is critical to determining accurate rates of eating disordered thoughts and behaviors.

Cultural Variability in the Effects of Question Design Features on Respondent Comprehension of Health Surveys

The aim is to identify the characteristics of common health survey questions that may be associated with cross-cultural variability in question comprehension. Interviews with respondents representing four cultural subgroups in the United States were analyzed through behavior coding to identify survey question characteristics that may be associated with cross-cultural variability in comprehension. Using survey responses as the unit of analysis, nested within survey respondents and survey questions, hierarchical linear modeling was used to examine the effects of four questionnaire design features on cultural variations in comprehension difficulties. Respondent culture was found to moderate the effects of response format, question length, and reading levels. Several question design strategies that reduce overall comprehension difficulty appear to also increase cross-cultural disparities.