Verbal Reporting Research Articles

Methods matter: Your measures of explicit and implicit processes in visuomotor adaptation affect your results.

Visuomotor rotations are frequently used to study the different processes underlying motor adaptation. Explicit aiming strategies and implicit recalibration are two of these processes. Various methods, which differ in their underlying assumptions, have been used to dissociate the two processes. Direct methods, such as verbal reports, assume explicit knowledge to be verbalizable, where indirect methods, such as the exclusion, assume that explicit knowledge is controllable. The goal of this study was thus to directly compare verbal reporting with exclusion in two different conditions: during consistent reporting and during intermittent reporting. Our results show that our two conditions lead to a dissociation between the measures. In the consistent reporting group, all measures showed similar results. However, in the intermittent reporting group, verbal reporting showed more explicit re-aiming and less implicit adaptation than exclusion. Curiously, when exclusion was measured again, after the end of learning, the differences were no longer apparent. We suspect this may reflect selective decay in implicit adaptation, as has been reported previously. All told, our results clearly indicate that methods of measurement can affect the amount of explicit re-aiming and implicit adaptation that is measured. Since it has been previously shown that both explicit re-aiming and implicit adaptation have multiple components, discrepancies between these different methods may arise because different measures reflect different components.

A case study of the use of verbal reports for talent identification purposes in soccer: A Messi affair!

Using a two-study approach, the main purpose of this case study was to explore the use of a verbal reporting methodology to better understand the thought processes of soccer talent scouts during an in-situ talent identification environment. Study 1 developed a standardized coding-scheme to examine verbal cognitions during a single soccer game. Study 2 then utilized this methodology to examine two full-time recruitment staff trained in the use of concurrent verbal reporting before undertaking a live, in-game task. Participants also participated in a debrief interview following the game. The findings of the two studies suggest that developing a verbal reporting protocol is viable, however when applied in a live-game environment it is problematic. Future research should therefore consider a modified version of this task to further explore the cognitions of scouts whilst observing and identifying potential talent.

Verbal reports in the reading processes of language learners: A methodological review

This methodological review highlights the trends in empirical studies where a methodological construct (i.e. verbal reports) intersects with content (i.e. literacy research). Specifically, we synthesise research on language learners’ reading in which verbal reports were deployed as a methodological tool. Questioning the long‐standing assumption that verbal report methods validated in first‐language verbal report reading studies necessarily constitute a basis for validation of second‐language verbal report reading studies, we consider the broader educational frameworks within which studies are embedded. In our synthesis of language learners’ verbal report literacy research published between 2000 and 2015, we attend to the social, demographic and geographic realities characteristic of studies reviewed and of the participants involved. Our findings demonstrate the following: (a) tendency to report quantitative information regardless of the type of verbal reporting method and the component of reading explored; (b) predominance of independent concurrent methods that emphasised the reading product; (c) predominance of integrated verbal reports (i.e. concurrent and other forms of reporting) in sociocultural studies that reflected the reading process; (d) concerns about validity in studies premised on cognitivist models of verbal reports; (e) a tendency to use solely concurrent verbal reports in quantitative studies; (f) high reliance on integrated concurrent methods in qualitative studies; and (g) preponderance of qualitative‐to‐quantitative versus a qualitative‐to‐quantitative‐to‐qualitative verbal reporting paradigms across studies. Based on these findings, we make several recommendations to be considered when verbal reports are used to study language learners’ reading processes.

Source evaluation of domain experts and novices during Web search

AbstractNowadays, almost everyone uses the World Wide Web (WWW) to search for information of any kind. In education, students frequently use the WWW for selecting information to accomplish assignments such as writing an essay or preparing a presentation. The evaluation of sources and information is an important sub‐skill in this process. But many students have not yet optimally developed this skill. On the basis of verbal reports, eye‐tracking data and navigation logs, this study investigated how novices in the domain of psychology evaluate Internet sources as compared to domain experts. In addition, two different verbal reporting techniques, namely thinking aloud and cued retrospective reporting, were compared in order to examine students' evaluation behaviour. Results revealed that domain expertise has an impact on individuals' evaluation behaviour during Web search, such that domain experts showed a more sophisticated use of evaluation criteria to judge the reliability of sources and information and selected more reliable information than domain novices. Furthermore, the different verbal reporting techniques did not lead to different conclusions on criteria use in relation to domain expertise, although in general more utterances concerning evaluation of sources and information were expressed during cued retrospective reporting.Lay DescriptionWhat is already known about this topic: When searching the Web, students assess the reliability of information and sources in a spontaneous manner. Students tend to use superficial criteria when judging the trustworthiness of web‐based information and sources. Little is known about the differences between domain experts and novices in terms of the ways they spontaneously evaluate web‐based information and sources. What this paper adds: Domain knowledge influences evaluation behaviour, with greater domain knowledge leading to more reliable selection of sources. A comparison of techniques for measuring evaluation behaviour when searching the Web for information reveals that different techniques, such as thinking aloud and cued retrospective reporting, have different pros and cons whose implications depend on the research questions being addressed. Implications for practice and/or policy: In formal education, more attention should be paid to equipping students with the requisite skills in searching for and evaluating web‐based information. Embedded process‐oriented instructional designs are needed that foster the appropriate evaluation behaviours.

Investigating mediation styles of second language listener verbal reports

Abstract This study investigated verbal report styles for eliciting strategies data from second language listeners. It examined outcomes from three different mediation (prompting) styles, one style unprompted and the two others prompted, after the learners were first provided with low-prescriptive instructions on how to complete the report. Also, the unprompted style was additionally examined after the provision of more-prescriptive instructions to observe the effect of this greater learner guidance. Theoretically, the core of the study examined two competing cognitive perspectives on verbal reporting. One, from an information processing perspective, is that verbal reports elicit the best insight into individuals’ strategic processes when prompts are kept to a minimum. The other perspective, a constructivist one, advocates the use of prompts in the form of researcher questions, mainly to help guide the report. Seventeen Taiwanese EFL learners participated in the study, with data gathered from each through a verbal report followed by a semi-structured interview. It was found that researcher prompting was both strongly favored by the learners and clearly elicited the best data for second language listener strategies research. The results also indicated that unprompted reports were little more effective when preceded by more-prescriptive instructions.

運動計測と言語報告に基づく身体スキル獲得に関する実験的検討

運動計測と言語報告に基づく身体スキル獲得に関する実験的検討

The electronic self report assessment and intervention for cancer: promoting patient verbal reporting of symptom and quality of life issues in a randomized controlled trial.

BackgroundThe electronic self report assessment - cancer (ESRA-C), has been shown to reduce symptom distress during cancer therapy The purpose of this analysis was to evaluate aspects of how the ESRA-C intervention may have resulted in lower symptom distress (SD).MethodsPatients at two cancer centers were randomized to ESRA-C assessment only (control) or the Web-based ESRA-C intervention delivered to patients’ homes or to a tablet in clinic. The intervention allowed patients to self-monitor symptom and quality of life (SxQOL) between visits, receive self-care education and coaching to report SxQOL to clinicians. Summaries of assessments were delivered to clinicians in both groups. Audio-recordings of clinic visits made 6 weeks after treatment initiation were coded for discussions of 26 SxQOL issues, focusing on patients’/caregivers’ coached verbal reports of SxQOL severity, pattern, alleviating/aggravating factors and requests for help. Among issues identified as problematic, two measures were defined for each patient: the percent SxQOL reported that included a coached statement, and an index of verbalized coached statements per SxQOL. The Wilcoxon rank test was used to compare measures between groups. Clinician responses to problematic SxQOL were compared. A mediation analysis was conducted, exploring the effect of verbal reports on SD outcomes.Results517 (256 intervention) clinic visits were audio-recorded. General discussion of problematic SxQOL was similar in both groups. Control group patients reported a median 75% of problematic SxQOL using any specific coached statement compared to a median 85% in the intervention group (p = .0009). The median report index of coached statements was 0.25 for the control group and 0.31 for the intervention group (p = 0.008). Fatigue, pain and physical function issues were reported significantly more often in the intervention group (all p < .05). Clinicians' verbalized responses did not differ between groups. Patients' verbal reports did not mediate final SD outcomes (p = .41).ConclusionsAdding electronically-delivered, self-care instructions and communication coaching to ESRA-C promoted specific patient descriptions of problematic SxQOL issues compared with ESRA-C assessment alone. However, clinician verbal responses were no different and subsequent symptom distress group differences were not mediated by the patients' reports.Trial registrationNCT00852852; 26 Feb 2009

TAPing into the interpreting process: Using participant reports to inform the interpreting process

This article presents the results of a Canadian study that examined the correlation of verbal reporting processes and the quality of interpretation. Two types of verbal reports, Think Aloud Protocols (TAPs) and Stimulated Recalls (SRs), were collected and analyzed to explore how TAPS and SRs might reflect the quality of interpreting provided in educational settings and therefore be potential tools for improving interpreting education strategies. Twelve interpreters working in educational contexts were recruited to participate in a multi-stage research process. Each interpreter was asked to perform a Think Aloud while viewing a sample of classroom discourse in preparation for interpreting it. Each interpreter then provided an interpretation, followed by a post-interpreting Stimulated Recall review of the interpretation. The standardized samples chosen were based on videotaped authentic classroom instruction and represented classes at the elementary, middle school and high school levels. A Deaf child was described for each level of interpreting so that the interpreters could target their interpretation. The results showed that those interpreters who demonstrated higher order cognitive thinking skills and attended to teacher intent and student language preferences provided more effective interpreting than the interpreters who focused primarily on linguistic choices and interpreting decisions. The findings have implications for interpreters, interpreter educators and mentors, and teachers working with interpreters and Deaf students in mediated learning environments. By exploring the ways in which attention to discourse features and teacher-student needs could be heightened, interpreters could enhance the quality of interpretation provided to Deaf learners. Finally, questions for future research and implications for collaboration across countries are discussed.

Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods.

Since its establishment, psychology has struggled to find valid methods for studying thoughts and subjective experiences. Thirty years ago, Ericsson and Simon (1980) proposed that participants can give concurrent verbal expression to their thoughts (think aloud) while completing tasks without changing objectively measurable performance (accuracy). In contrast, directed requests for concurrent verbal reports, such as explanations or directions to describe particular kinds of information, were predicted to change thought processes as a consequence of the need to generate this information, thus altering performance. By comparing performance of concurrent verbal reporting conditions with their matching silent control condition, Ericsson and Simon found several studies demonstrating that directed verbalization was associated with changes in performance. In contrast, the lack of effects of thinking aloud was merely suggested by a handful of experimental studies. In this article, Ericsson and Simon's model is tested by a meta-analysis of 94 studies comparing performance while giving concurrent verbalizations to a matching condition without verbalization. Findings based on nearly 3,500 participants show that the "think-aloud" effect size is indistinguishable from zero (r = -.03) and that this procedure remains nonreactive even after statistically controlling additional factors such as task type (primarily visual or nonvisual). In contrast, procedures that entail describing or explaining thoughts and actions are significantly reactive, leading to higher performance than silent control conditions. All verbal reporting procedures tend to increase times to complete tasks. These results suggest that think-aloud should be distinguished from other methods in future studies. Theoretical and practical implications are discussed.

Supplemental Material for Do Procedures for Verbal Reporting of Thinking Have to Be Reactive? A Meta-Analysis and Recommendations for Best Reporting Methods

Supplemental Material for Do Procedures for Verbal Reporting of Thinking Have to Be Reactive? A Meta-Analysis and Recommendations for Best Reporting Methods

Observing the observer: non-intrusive verbalisations using the Concurrent Observer Narrative Technique

This paper presents the application of, and a discussion about, a new method of information acquisition, the Concurrent Observer Narrative Technique (CONT), that requires an expert to narrate the work of experienced colleagues. The method is explored through a case study of instructors in a training simulation environment. The method involves multiple expert users of the system equally matched in experience; one or more of which perform the task at hand, namely running the training session, whilst another, not engaged in the training activity, provides verbal reports whilst observing task progression. All are situated in the work environment concurrently. The method is discussed in terms of the richness and detail of the data obtained from the reports and in terms of some advantages and disadvantages over other verbal reporting methods, specifically collegial verbalisation (Jansson et al. in Cogn Tech Work 8:41–49, 2006) and Elicitation By Critiquing (Miller et al. in Cogn Tech Work 8: 90–102, 2006). It is argued that the CONT provides a relatively easy, inexpensive, and non-intrusive method that can supply a wealth of information that does not require further interpretation and that the method is of particular applicability in domains where the subject matter expert is unable to provide concurrent verbalisations themselves and where the subject matter is too complex to be understood by a lay person.

Cognitions associated with nurse performance: A comparison of concurrent and retrospective verbal reports of nurse performance in a simulated task environment

Background Cognitions represent the computations associated with human thought processes. Verbal protocols offer a method by which to record these processes. While concurrent and retrospective verbal reporting of cognitions have been used extensively within nursing and other domains, the use of this methodology in nursing has been characterized by inconsistencies in its application. Objectives The primary objective of this study was to describe and compare the content of concurrent and retrospective verbal reports provided by nurses during and after administering care in a simulated task environment. Design The study utilized a laboratory based quasi-experimental research design. Setting The study was based in a simulation laboratory designed specifically for use in studies designed to measure nursing performance. The laboratory integrated extensive instrumentation that facilitated the comprehensive audio and video recording of participant actions. Participants The participants ( N = 15) were recruited from a College of Nursing at a large university in the Southeastern United States. Methods Research participants were asked to admit a patient experiencing an acute exacerbation of congestive heart failure in a simulated task environment, during which they were required to prioritize and provide care. Participants were trained in the method for providing verbal reports of thoughts, and concurrent and retrospective report data were collected during and after the simulation exercise. The data were then coded for the purposes of descriptive analysis. Results The results indicated that the concurrent verbal reports provide a more complete representation of the cognitions of research participants providing care in a simulated task environment. However, the results reflect that additional unique data is present in the retrospective reports, exclusive of the concurrent reports. Conclusions The findings support the utility of concurrent and retrospective verbal reports as a method of gathering data in studies that address nursing performance in a clinical context.

Verbal Reports in L2 Speech Production Research

As there seems to be some confusion concerning the various research methods belonging to the category of verbal reports, I first define thinkaloud (procedures), introspection, and retrospection. For think-alouds (or concurrent verbalisations), researchers instruct their subjects to verbalise directly only the thoughts entering their attention while performing the task (Ericsson & Simon, 1993). Thus, this type of verbal reporting is concurrent with the execution of a specific task, and the participants provide information while it is still available to them-that is, while it remains in short-term memory (STM). For example, study participants can be asked to think aloud while translating a sentence from their L1 to their L2 and to report the sequence of thoughts related to the selection of appropriate words and grammatical structures while doing the translation. For introspection, subjects are not only requested to verbalise but also to describe or explain their thoughts (Ericsson & Simon, 1993). Introspection therefore entails the activation of information that is not stored

Introspection and verbal reports on cognitive processes—Two approaches to the study of thinking: A response to Howe

K. ANDERS ERICSSON and ROBERT J. CRUTCHER Department of Psychology, University of Colorado at Boulder, Boulder, CO 80309-0345, U.S.A. Much confusion has resulted from referring to all verbal descriptions and reports of thinking as introspection. In this paper, we argue that introspection on cognitive processes and verbal reports on cognitive processes are two distinct approaches to the study of thinking. This distinction is important as the introspective approach has continually generated controversy, whereas the verbal report approach has enjoyed general acceptance throughout the history of studying thought. The introspective approach relies on detailed descriptions and analyses of thinking performed by skilled observers. The major problems with introspection have been: (1) whether subjects can give detailed descriptions of thoughts-the nature of thinking is too dynamic to make such descriptions possible; (2) whether the reported characteristics are valid-in introspection, subjects’ reports have a privileged status inconsistent with traditional science based on reliable inter-subjective observations; and (3) whether the act of introspection can be performed without changing the process of thinking-efforts to uncover specific information about a thought transform the mental state corresponding to that thought. More recently, however, the technique of verbal reporting in studying mental processes has enjoyed increased popularity. Whereas introspection implicitly trusted subjects’ reports, the verbal report approach treats reports as another type of data to be accounted for by theory. Furthermore, while the introspectionists often disagreed over what information subjects could report and extensively trained their subjects to report specific types of information, investigators collecting verbal reports have tried to discover what types of information naive subjects can reliably report when given simple instructions to report their thoughts. While the earlier introspective techniques were plagued by methodological problems, the current verbal report techniques have been put on a secure scientific foundation by careful analysis of the different types of reports and the conditions under which valid reports may be elicited. With the renewed interest in consciousness and mind, many investigators are inclined to return to the controversial approach of introspection. For example, in his paper, Howe proposed the readoption of introspection as a valid method of studying mental processes. In our judgment Howe bases his arguments on a broad definition of introspection that includes our more narrow definition of

Effect of Eliciting Verbal Reports of Thinking on Critical Thinking Test Performance

Verbal reports of examinees' thinking on multiple‐choice critical thinking test items can provide useful validation data only if the verbal reporting does not change the course of examinees' thinking and performance. Using a completely randomized factorial design, 343 senior high school students were divided into five groups. In four of the groups, different procedures were used to elicit students' thinking as they worked through Part A of a critical thinking test of observation appraisal (Norris &amp; King, 1983). In the control group, students took the same test in paper‐and‐pencil format. There were no significant differences in test performance among the five groups nor in the quality of thinking among the four groups from whom verbal reports of thinking were elicited. These results are evidence that verbal reports of thinking can meet one of the necessary conditions of useful validation data—namely, that collecting the data does not alter examinees' thinking and performance. Some analyses found significant interviewer main effects and sex‐by‐interviewer and elicitation‐level‐by‐interviewer‐by‐sex‐by‐grade interaction effects. Analysis of these interactions suggested that the role of the interviewer might limit the generality of the technique.

Representation of familiar places in children and adults: Verbal reports as a method of studying environmental knowledge

This research investigated children's representations of familiar places by means of verbal reporting. Eight-year-old children's representation of the entrance hall and the courtyard of their school was examined under three conditions: free recall; description from memory, intended for a person not acquainted with the place; and direct description, given while looking at the place. Children's descriptions from memory were then compared with those given by a group of teachers in the school. Both contents and organization of verbal reportings were analyzed. Quantitative results showed a difference in recall between places only in children; furthermore, both children and adults mainly remembered the aspects of places which are constant constraints to actions. The importance of considering some qualitative-linguistic indexes was assessed by results showing, for instance, differences in children's discourse organization across conditions, the relevance of cognitive more than physical boundaries of places, and age differences in localizing items.

On the Use of Verbal Reports in Reading Research

In 1917, E. L. Thorndike focused attention on the processes, particularly the reasoning processes, involved in reading. More than 50 years later, following the hiatus imposed by behaviorism, Simons (1971) reemphasized that research should focus more on the reading process than on the product of reading if progress were to be made toward understanding the nature of comprehension. However, observing and understanding the comprehension process seems more easily proposed than actualized. We have generally inferred the process from products, such as differential performance on different comprehension questions, perhaps collected under different conditions. Researchers interested in studying the cognitive processes involved in reading have recently shown interest in the use of verbal reports to gather data on aspects of the reading process (Bridge & Winograd, 1982; Brown & Day, 1983; Garner, 1982; Johnston & Afflerbach, 1983; Kavale & Schreiner, 1979; Marr, 1983; Olshavsky, 1976-77). This interest is actually a continuation of the long, if sporadic, history of verbal report data in reading research (Huey, 1908; McCallister, 1930; Piekarz, 1954; Strang, 1964, 1970). Verbal reporting, especially introspection, had been an important component of psychological investigation, including reading research, prior to the rise of behaviorism