Interviewer-respondent Interaction Research Articles

Behind the Scenes: A Qualitative Investigation of Interviewers' Performance in EQ-5D Valuation Studies.

The EuroQol Valuation Technology (EQ-VT) protocol is currently employed by the valuation studies of the EQ family of instruments worldwide. To date, all the evidence in support of the quality control (QC) originates from quantitative indicators. We aimed to explore interviewers' conversational patterns in EQ-VT interviews, beyond quantitative QC indicators, and toprovide a preliminary exploration of how the interaction between interviewer and respondent impacts data quality. Two researchers transcribed and independently coded 24 video-recorded interviews from the Italian EQ-5D-5L valuation study, adopting the conversational analysis framework. The analysis identified positive and negative 'patterns' of conversational practice. These were categorized into themes and sub-themes and were used to score a random sample of 42 video-recorded interviews conducted at different time points by seven interviewers. The conversational analysis identified 20 positive and 14 negative interview patterns, which were grouped into two main themes (i.e., task execution and communication skills). Positive items included appending questions that stimulated respondents' engagement, providing different explanations for an unclear aspect, supporting the participant with useful information for completing the tasks, and increasing the interview's coherence by confirming the respondent answers. Negative patterns included moving forward in the exercise without making sure that the respondent understood, trying to force an answer from the respondent, speaking too fast, and providing incomplete or incorrect explanations of the task. Most interviewers exhibited a moderate increase in positive patterns or a decrease in negative patterns over time. A certain degree of consistency between the quantitative QC results and the qualitative scoring deriving from the interviewer-respondent interaction was observed, with the best and worst performers of the qualitative scoring showing good and bad scores on key QC items, respectively. The identified positive and negative patterns may be useful to inform the training material of EQ-VT studies worldwide and complement the existing QC process.

Questioning Identity: How a Diverse Set of Respondents Answer Standard Questions About Ethnicity and Race

Ethnoracial identity refers to the racial and ethnic categories that people use to classify themselves and others. How it is measured in surveys has implications for understanding inequalities. Yet how people self-identify may not conform to the categories standardized survey questions use to measure ethnicity and race, leading to potential measurement error. In interviewer-administered surveys, answers to survey questions are achieved through interviewer–respondent interaction. An analysis of interviewer–respondent interaction can illuminate whether, when, how, and why respondents experience problems with questions. In this study, we examine how indicators of interviewer–respondent interactional problems vary across ethnoracial groups when respondents answer questions about ethnicity and race. Further, we explore how interviewers respond in the presence of these interactional problems. Data are provided by the 2013–2014 Voices Heard Survey, a computer-assisted telephone survey designed to measure perceptions of participating in medical research among an ethnoracially diverse sample of respondents.

Gender-responsive remote data collection, analysis, and use during crisis: UN Women’s Rapid Gender Assessments (RGAs)

At a time when timely and relevant data was needed to understand the gendered impacts of COVID-19, the standard face-to-face data collection was limited due to lockdowns and other restrictions. Data producers, both from national and international organizations, had to explore remote methodologies such as phone- or web-based surveys. While differences in women’s and men’s situations must always be considered in all stages of any survey design, added attention should be given to remote data collection wherein there may be immediate gender bias in sampling selection, interviewer-respondent interaction, and handling respondent burden. This paper presents UN Women’s experience in intentionally applying a gender-responsive approach in remotely conducting 78 rapid gender assessment surveys (RGAs) across regions. Based on the learnings, recommendations will be provided to further improve remote data collection such as highlighting the importance of identification and monitoring of use cases, gender mainstreaming in statistical operations, and underscoring the role of national statistics offices in responding to gender data gaps during crisis.

The Carryover Effects of Preceding Interviewer–Respondent Interaction on Responses in Audio Computer-Assisted Self-Interviewing (ACASI)

Abstract Audio computer-assisted self-interviewing (ACASI) has been widely used to collect sensitive information from respondents in face-to-face interviews. Interviewers ask questions that are not sensitive or only moderately sensitive and then allow respondents to self-administer more sensitive questions, listening to audio recordings of the questions and typically entering their responses directly into the same device that the interviewer has used. According to the conventional thinking, ACASI is taken as independent of the face-to-face interaction that almost always precedes it. Presumably as a result of this presumed independence, the respondents’ prior interaction with the interviewer is rarely considered when assessing the quality of ACASI responses. There is no body of existing research that has experimentally investigated how the preceding interviewer–respondent interaction may create sufficient social presence to affect responses in the subsequent ACASI module. The study reported here, a laboratory experiment with eight professional interviewers and 125 respondents, explores the carryover effects of preceding interactions between interviewer and respondent on responses in the subsequent ACASI. We evaluated the impact of the similarity of the live and recorded interviewer’s voice for each respondent as well as respondents’ rapport with interviewers in the preceding interview. We did not find significant main effects of vocal similarity on disclosure in ACASI. However, we found significant interaction effects between vocal similarity and respondents’ rapport ratings in the preceding interview on disclosure in ACASI. When the ACASI voice was similar to the interviewer’s voice in the preceding interaction, respondent-rated rapport led to more disclosure but, when the ACASI voice is clearly different from the interviewer’s voice, respondent-rated rapport in the prior interaction did not affect disclosure.

The effect of emphasis in telephone survey questions on survey measurement quality

ABSTRACT Questionnaire design texts commonly recommend emphasizing important words, including capitalization or underlining, to promote their processing by the respondent. In self-administered surveys, respondents can see the emphasis, but in an interviewer-administered survey, emphasis has to be communicated to respondents through audible signals. We report the results of experiments in two US telephone surveys in which telephone survey questions were presented to interviewers either with or without emphasis. We examine whether emphasis changes substantive answers to survey questions, whether interviewers actually engage in verbal emphasis behaviors, and whether emphasis changes the interviewer-respondent interaction. We find surprisingly little effect of the question emphasis on any outcome, with the primary effects on vocal intonation and the interviewer-respondent interaction. Thus, there is no evidence here to suggest that questionnaire designers should use emphasis in interviewer-administered questionnaires to improve data quality. As the first study on this topic, we suggest many opportunities for future research.

Item Location, the Interviewer–Respondent Interaction, and Responses to Battery Questions in Telephone Surveys

Survey researchers often ask a series of attitudinal questions with a common question stem and response options, known as battery questions. Interviewers have substantial latitude in deciding how to administer these items, including whether to reread the common question stem on items after the first one or to probe respondents’ answers. Despite the ubiquity of use of these items, there is virtually no research on whether respondent and interviewer behaviors on battery questions differ over items in a battery or whether interview behaviors are associated with answers to these questions. This article uses a nationally representative telephone survey with audio-recorded interviews and randomized placement of items within four different batteries to examine interviewer and respondent behaviors and respondent answers in battery questions. Using cross-classified random-effects models, the authors find strong evidence that there is more interviewer–respondent interaction on items asked earlier in the battery. In addition, interviewer and respondent behaviors are associated with both substantive and nonsubstantive answers provided to battery items, especially if the interviewer decided to reread or probe with the response options. These results suggest that survey designers should follow recommendations to randomize battery items and consider the importance of standardization of question administration when designing battery questions.

Interviewer-respondent interaction and rapport in PIAAC

PurposeThis paper aims to investigate small-scale, qualitative observations of interviewer–respondent interaction in the Organisation for Economic Co-operation and Development’s Programme for the International Assessment of Adult Competencies (PIAAC).Design/methodology/approachThe paper uses video-ethnographic methods to document talk and gesture in assessment in Slovenian household settings. It presents an in-depth case study of interaction in a single testing situation.FindingsObserving interaction in assessment captures data on assessment performance that is not available in quantitative analysis of assessment response processes. The character of interviewer–respondent interaction and rapport is shaped by the cognitive demands of assessment and the distinctive ecological setting of the household.Research limitations/implicationsObservational data on assessment response processes and interaction in real-life assessments can be integrated into and synthesized with other sources of “process data”.Practical implicationsAssessment programs such as PIAAC should consider the significance of the household setting on assessment quality and observations of interaction in assessment as a valid source of paradata.Social implicationsThere is a place for small-scale observational studies of assessment to inform public understanding of assessment quality and validity.Originality/valueThe paper provides qualitative insights into the significance of interaction and “interviewer effects” in household assessment settings.

Interviewer–respondent Interactions in Conversational and Standardized Interviewing

Standardized interviewing (SI) and conversational interviewing are two approaches to collect survey data that differ in how interviewers address respondent confusion. This article examines interviewer–respondent interactions that occur during these two techniques, focusing on requests for and provisions of clarification. The data derive from an experimental study in Germany, where the face-to-face interviews were audio-recorded. A sample of 111 interviews was coded in detail. We find that conversational interviewers do make use of the ability to clarify respondent confusion. Although the technique improved response accuracy in the main study compared to SI, conversational interviewers seem to provide clarifications even when there is no evidence of respondent confusion. This may lengthen administration time and potentially increase data collection costs. Conversational interviewers also employ neutral probes, which are generally associated with standardized interviews, at an unexpectedly high rate. We conclude with suggestions for practice and directions for future research.

Christine Guionnet et Sophie Rétif (dir.), Exploiter les difficultés méthodologiques. Une ressource pour l’analyse en sciences sociales , Presses universitaires de Rennes, 2015, 175 p.

Christine guionnet and Sophie Retif, eds., Exploiter les difficultes methodologiques. Une ressource pour l'analyse en sciences sociales [Making use of methodological difficulties: a resource for social science analysis], Presses universitaires de Rennes, 2015, 175 p.Now that social science researchers are expected to demonstrate clear epistemological thinking, they may be tempted to try to achieve a kind of perfect axiological neutrality through rigorous application of methods and techniques. And in such a context, any methodological difficulties encountered in the various research stages may be experienced as stumbling blocks and lead to feelings of discomfort and illegitimacy. The result is that everyone - students and experienced researchers alike - tend to smooth such difficulties. To counter that tendency, to lay bare the difficulties researchers may encounter and show how they can be analysed and used to fuel thinking on research subjects, Christine Guionnet formed a group of researchers from the Centre for research on policy action in Europe (CRAPE) and the Nantes Centre for sociology. This work delivers their thinking. In emphasizing the heuristic value of the difficulties, the work is both a pedagogical argument for gaining proper perspective on the obstacles researchers encounter in their fieldwork and indeed rehabilitating them, and an invitation to researchers to integrate analysis of the meaning of such difficulties more systematically into their scientific work. In this respect, it is for both experienced researchers and beginners.Most of the book's seven chapters discuss fieldwork experiences and difficulties. In the first contribution, Beatrice Damian-Gaillard and Mathieu Trachman present the difficulties they had to cope with during two surveys on pornography use in France. Problems envisaged at the outset - here, entering a perceived as fairly opaque and inaccessible and the need to justify studying a illegitimate subject in an academic context - may prove quite different from the ones encountered in the field. For the female researcher, the main problem turned out to be sexualisation of the survey relationship, whereas the male researcher was compelled to objectify the effects of his male socialization, specifically in connection with female sexuality. The dialogue between the two researchers shows how their personal characteristics - particularly sex and age - influenced their relationship to respondents and clarifies the identities respondents ascribed to them in the as subjects and objects of desire.Chapter 2, by Sami Zegnani, examines how working in a presumably close to the researcher affects that researcher while discussing the assumptions that such proximity elicit not only in respondents but also other researchers. The sociologist in question, originally from a sensitive or rough neighbourhood, was studying three groups with roots in neighbourhoods of the same type (street youth, Salafi Muslims, and rap musicians). He shows that the distantiation requirement that followed from his presumed insider status proved largely irrelevant. Without denying the advantages he enjoyed due to his origins and local embeddedness and ties, he details all the strategies he had to develop for adapting to the and respondents, especially when it came to winning respondents' trust while maintaining distance and circulating within relatively closed groups. He seeks to demystify the widespread idea that 'native' ethnographers fit naturally into the field (p. 69) and to move beyond the insider/ outsider binary opposition so as to analyse more effectively what our relationship to the may tell us about the phenomena under study.The issue of variation in interviewer-respondent interaction by personal characteristics of all parties is likewise the subject of Sylvie Ollitrault's contribution. A political scientist, she reviews her twenty years of research on French environmentalist movements, offering a diachronic perspective on how research difficulties changed over the course of her career. …

Using Data Mining to Predict the Occurrence of Respondent Retrieval Strategies in Calendar Interviewing: The Quality of Retrospective Reports

Abstract Determining which verbal behaviors of interviewers and respondents are dependent on one another is a complex problem that can be facilitated via data-mining approaches. Data are derived from the interviews of 153 respondents of the Panel Study of Income Dynamics (PSID) who were interviewed about their life-course histories. Behavioral sequences of interviewer-respondent interactions that were most predictive of respondents spontaneously using parallel, timing, duration, and sequential retrieval strategies in their generation of answers were examined. We also examined which behavioral sequences were predictive of retrospective reporting data quality as shown by correspondence between calendar responses with responses collected in prior waves of the PSID. The verbal behaviors of immediately preceding interviewer and respondent turns of speech were assessed in terms of their co-occurrence with each respondent retrieval strategy. Interviewers’ use of parallel probes is associated with poorer data quality, whereas interviewers’ use of timing and duration probes, especially in tandem, is associated with better data quality. Respondents’ use of timing and duration strategies is also associated with better data quality and both strategies are facilitated by interviewer timing probes. Data mining alongside regression techniques is valuable to examine which interviewer-respondent interactions will benefit data quality.

Bystanders, noise, and distractions in face-to-face surveys in Africa and Latin America

Survey researchers are increasingly concerned that the presence of other people (bystanders) may affect data quality in structured, face-to-face survey interviews. In this article, we study bystanders using data from 15,309 face-to-face surveys about technology from Ghana, Nigeria, Uganda, Brazil and Guatemala. Our analysis (1) describes the frequency of bystander presence (spouses, parents, other family and non-family members), (2) investigates the correlates of bystander presence, and (3) studies whether bystanders affect response distributions, ‘don’t know’ responses, and satisficing behavior. We also consider the influence of other environmental factors – noise and distractions – on interviewer–respondent interaction and satisficing. The results are encouraging for the field of survey research. Bystanders did not affect response distributions, the number of ‘don’t know’ responses, or satisficing behavior. Although noise negatively affected interviewer–respondent interactions, noise did not affect data quality. Our research suggests that for less sensitive topics, bystanders have limited influence, and also that respondents and interviewers are resilient to challenges such as noisy environments.

Using Interviewer-respondent Interaction Coding as a Manipulation Check on Interviewer Behavior in Persuading CATI Respondents

This paper shows how interaction coding of interviewer-respondent interactions was used to perform manipulation checks of CATI interviewer behavior in experimental studies. An experiment in which interviewers were instructed to persuade potential respondents by means of a personal style or a formal style showed no significant effects of the persuasion style on survey participation. By means of interviewer-respondent interaction analysis, we studied the interviews in more depth focusing on the compliance of interviewers with the instructions. First, we found that many respondents immediately complied, but when respondents were reluctant, using any form of persuasion was better than none. Second, interviewers also had success in gaining cooperation when they referred to an argument that they had not been instructed to use. In conclusion, we assume that interviewers using arguments in which they were trained develop too much of an unauthentic routine in expressing these arguments, whereas using arguments outside instructions are likely to be expressed in a more natural, spontaneous way and are therefore more convincing. In addition, this study shows that it is useful to include behavior coding as a manipulation check in experiments involving interviewer behavior.

Introduction to Special Issue of Survey Practice on Interviewer-Respondent Interaction

Although survey research has witnessed a surge in the use of self-administered web and mail surveys over the past decades, interviewer-administered surveys continue to be an important part of the data collection landscape, particularly when high response rates are critical, samples are complex, and measurement is demanding. Thus, there is continued need for research concerning the interviewer’s role in both measurement and recruitment. This special issue of Survey Practice features several articles that highlight some current issues and recent findings about interviewer-respondent interaction coding (often referred to as “behavior coding”) that have immediate and practical recommendations for writing and evaluating survey questions, training interviewers, and recruiting sample members.

Felicitometric hermeneutics: interpreting quality of life measurements

The use of quality of life (QOL) outcomes in clinical trials is increasing as a number of practical, ethical, methodological, and regulatory reasons for their use have become apparent. It is important, then, that QOL measurements and differences between QOL scores be readily interpretable. We study interpretation in two contexts: when determining QOL and when basing decisions on QOL differences. We consider both clinical situations involving individual patients and research contexts, e.g., randomized clinical trials, involving groups of patients. We note the ethical importance of such understanding: proper interpretation and communication facilitate health care decision making. Communication that facilitates interpretation is of moral significance since better communication can attenuate ethical problems and inform choices. Much of what is communication worthy about QOL assessments is determined by the particular QOL instrument used in the assessment and how it is administered. In practice, these choices will be driven by the purpose of the assessment, but, it is argued, to maximize understanding, we should combine the information garnered from traditional standardized QOL instruments, from individualized QOL assessments, and from a recently proposed dialogic paradigm, where QOL is determined by shared conversation regarding the interpretation of texts. And, while some studies can surely succeed using abbreviated methods of administration (e.g., postal surveys may suffice for certain purposes), we will focus on methods of administration involving interviewer-respondent interaction. We suggest that during the QOL elicitation process, interviewer and respondent should engage in a two-way conversation in order to achieve a shared understanding of the "answers" to QOL "questions" and, finally, to reach a shared interpretation of the individual's QOL.

Evaluation of the convergence between “self-reporters” and “proxies” in a disability questionnaire by means of behaviour coding method

Household surveys often require including proxy reporters to obtain information about other household members who cannot be interviewed. The participation of proxies can undermine survey data quality due to the fact that proxies must respond to questions thinking about other people. The objectives of the present study were to analyze the behaviour of proxy reporters and evaluate the convergence between the answers given by proxies and self-reporters by means of behaviour coding. This improves the evaluation of convergence, since only adequate (i.e., interpretable) answers given by both types of informant are taken into account. Responses to a disability questionnaire employed by an official statistical institute were analyzed. The questionnaire includes 11 questions about different limitations related to everyday activities. 16 self-reporter and 16 proxies formed 16 couples whose members lived together and supported a direct family relation. The results show a high percentage (52%) of convergence between both types of informant, although fluctuating across the questions and the couples. Proxies showed relatively more adequate behaviour during the interaction than self-reporters. From this we conclude that proxies can be considered at least as good informants as self-reporters from an interviewer-respondent interaction perspective. Future research should address the impact of proxy responses on survey validity.

‘‘It Is Time Computers Do Clever Things!’’: The Impact of Dependent Interviewing on Interviewer Burden

Despite the increased interest in assessing the effects of dependent interviewing (DI) on data quality, the impact of DI on interviewer burden and interviewer—respondent interaction remains currently unexplored and undocumented. Drawing on Japec’s work, this article attempts to evaluate the impact of DI on interviewer burden and describes the mechanisms by which DI affects perceived interviewer burden and ultimately data quality. We use a diverse set of data including qualitative data collected from a survey carried out in 2006 on the British Household Panel Survey (BHPS) interviewers. Three main findings stand out: DI is usually welcomed by interviewers; under condition of no change in respondent circumstances, DI eases interviewer burden; the mechanisms by which DI eases interviewer burden are complex as interviewer burden factors are often strictly related to respondent burden; and there are strong interrelationships among the different factors that constitute interviewer burden.

Analysis of Quality of Proxy Questions in Health Surveys by Behavior Coding

The aim of this study is to show how to analyze the quality of questions for proxy informants by means of behavior coding. Proxy questions can undermine survey data quality because of the fact that proxies respond to questions on behalf of other people. Behavior coding can improve questions by analyzing interviewer-respondent interactions. Twenty-nine proxies participated in the pretesting of a disability questionnaire. The questionnaire includes 11 questions related to daily-life limitations as a result of health problems. Interviewer-proxy interactions were coded and analyzed by means of Sequence Viewer program. The percentages, from a methodological perspective, of ideal "question-and-answer" sequences varied from 28% to 76% throughout the 11 questions analyzed. The results obtained pointed out the necessity of reviewing some of the proxy questions analyzed. Behavior coding can improve the quality of proxy questions in health surveys when proxy informants are surveyed.

Interviewer laughs: Shared laughter and asymmetries in employment interviews

Routinely, in a corpus of videotaped employment interviews, the interviewer invites laughter. The interviewee laughs along. The interviewer may produce a next laughable or briefly topicalize the laughable materials, but the interviewee does not do so. Rather, the interviewee will wait for the interviewer to take the lead in returning them to the business of the interview. The asymmetries evident in the sequential organization of these shared laughs show participant orientation to respective institutional roles. These asymmetries are consistent with those identified in previous research examining doctor–patient and survey interviewer–respondent interactions. While laughing together might appear to reduce power distance and bring participants together, the organization and distribution of these shared laughter instances reflect and reinscribe the hegemony of the roles of interviewer and interviewee.

"I didn't write the questions!" - Negotiating telephone-survey questions on birth timing

This paper examines interviewer-respondent interaction in the collection of demographic data where interviewer and respondents speak the same first language. Conversation analysis (CA) or the analysis of talk in interaction makes transparent the interaction between an interviewer and 25 respondents on a question about pregnancy and birth timing in an Australian telephone survey Negotiating the Life Course. The analysis focuses on the troubles that occur and the work that interviewers do to fit respondents answers to the survey researchers categories. Interviewers are shown to act as mediators in difficult interaction with responses often distorted by question format the imperative of achieving an allowed response and the need to keep the respondent in the survey. The analysis suggests that conversational resources could be used constructively to ensure a better fit between questions and responses. (authors)

A Note on Sample Size for Behavior Coding Pretests

Behavior coding is a widely used method for pretesting survey questionnaires. It is based on the idea that certain types of interviewer-respondent interactions indicate question flaws if the number of such interactions exceeds some specified percentage of the pretest interviews. In this article, we examine statistical approaches for assessing the reliability of different observed percentages of problematic interactions.