Improving the Measurement of Gender in Surveys: Effects of Categorical Versus Open-Ended Response Formats on Measurement and Data Quality Among College Students

Abstract

Abstract While researchers have some recommendations for measuring gender identity in surveys based on research and other sources that are summarized in a series of working group and panel reports, we continue to refine our understanding and practices. Gender identity is usually measured in surveys using a categorical selection response format with a small number of response options (e.g., “female,” “male,” “nonbinary”) and an open text response field to capture additional responses (“not listed, please tell us”). There is limited research guiding researchers on the use of other response formats. This study reports results from a between-subjects experiment embedded in a campus climate survey about university students’ attitudes about their campus and their behaviors and experiences related to inclusion and belonging at a large Midwestern university in 2021. Over 13,000 students were asked “What is your gender?” and subsequently randomly assigned to respond using either a categorical selection response format or an open response format (i.e., a place to specify their gender with no response options listed). We examine the distribution of responses, item nonresponse, response times, and concurrent validity (in terms of the association between gender and relevant survey outcomes) across the two response formats. Findings indicate the categorical selection response format is preferred for this population. While results show similar distributions in the categorization of responses across the formats and similar relationships with other survey outcomes, the selection format is associated with less item nonresponse and shorter response times.