Cognitive Task Analysis Methods Research Articles

Eliciting Knowledge from Helicopter Pilots: Recommendations for Revising the ACTA Method for Helicopter Landing Tasks

The task of landing helicopters on ships is cognitively complex and is bounded by several limitations for safe operation. To design technologies to support helicopter pilots in shipboard operations, a better understanding of cognitive processes underlying helicopter piloting in shipboard landing maneuvers is required. Limitations in prior work on ship-based helicopter pilots motivate more studies to better understand how pilots compensate for difficulties associated with ship motion, air turbulence among other factors. We report a preliminary study that used the Applied Cognitive Task Analysis (ACTA) method. We interviewed three participants who have ground-based helicopter piloting experience. The results indicate that ACTA can be used in eliciting knowledge from helicopter pilots. However, we have identified several caveats with the use of the method for this application. The revised method will be used to interview helicopter pilots with shipboard landing experience. Our study has identified a number of cognitive elements in the task which are associated with task management, divided attention, and noticing anomalies.

A situation awareness analysis scheme to identify deficiencies of complex man-machine interactions

This paper presents an analysis scheme which aims to support a systematic study of required situation awareness (RSA). This scheme supports identification of deficiencies of support for situation awareness (SA). Information needs are goal and task dependent and can be defined using existing cognitive task analysis (CTA) methods. RSA, however, is a subset of information needs and depends on the interaction between goals, tasks, system factors and individual factors. The analysis scheme has helped to identify that the current methods to define RSA do not support a distinction between information needs and RSA. The scheme was trialled in a nautical traffic management context as an extension to existing CTA methods. The research activities necessary to study RSA and to identify deficiencies of current support for SA in nautical traffic management context were applied. The study showed that the research set-up designed through application of the analysis scheme helped to define RSA, and that RSA is considerably context and operator dependent. Future research will focus on the potential for context-aware adaptable interface solutions to allow for RSA dependent information visualisation.

Assessment of Matrix Multiplication Learning with a Rule-Based Analytical Model–A Bayesian Network Representation

<p class="apa">This study explored an alternative assessment procedure to examine learning trajectories of matrix multiplication. It took rule-based analytical and cognitive task analysis methods specifically to break down operation rules for a given matrix multiplication. Based on the analysis results, a hierarchical Bayesian network, an assessment model, comprising of 2 layers of explanatory variables-Matrix Multiplication, Performance and Semantic Explanations; and one layer of evidential variables containing 9 evidential variables-was developed. With the simulating data, 9 students’ Performance and Semantic Explanation evidences were recorded. The results indicated that the hierarchical Bayesian assessment effectively traced and recorded students’ learning trajectories; and assessed students’ learning dynamically and diagnostically.</p>

Examination of Anesthetic Practitioners’ Decisions for the Design of a Cognitive Tool for Airway Management

The goal of this research was to examine the decisions made by anesthesia providers in emergency and routine clinical situations relating to airway management. A key function of anesthesia is to ensure oxygen can be delivered to the lungs via a patient’s airway. Failure to maintain adequate oxygen levels leads to brain damage and death. The anesthetic work environment is complex. Occasionally, stressful situations occur when difficulties in airway management arise. Members of the anesthesia team must then engage in complex cognitive activities such as rapid, collaborative decision-making. It is suggested that cognitive aids may support decision-making in these situations, although this has not yet been evaluated empirically from a Cognitive Systems Engineering perspective. This ongoing research combined two studies using Cognitive Task Analysis methods as part of a Decision-Centered Design process: observations and Critical Decision Method interviews. These will inform subsequent research phases concerning the development and evaluation of design concepts.

A Descriptive Model of Computer Code Trustworthiness

Trust and trustworthiness judgments have been studied in the context of social, business, and romantic relationships as well as between humans and automation. This article extends the prior research to explore how programmers assess code for trustworthiness when asked to reuse existing computer code. We used cognitive task analysis methods to explore experienced programmers’ first-person perspectives on code reuse. We elicited specific cues and strategies used to assess trustworthiness in real-world scenarios. Using qualitative analysis techniques, we grouped cues into three trustworthiness factors: performance, transparency, and reputation. We also identified environmental factors that influence acceptable levels of trust, including customer needs and requirements, organizational resources and constraints, and consequences of failure. We propose a descriptive model based on these findings. These findings have important implications for organizations that intend to reuse, adapt, and extend code over time. Writing code with the factors such as reputation, transparency, and performance in mind will increase the likelihood that it will be trusted in the near term and be reusable in the future. Furthermore, this research provides an important foundation for future studies exploring trusting behaviors, individual differences, and the ability to detect malicious code.

How to recognise a kick: A cognitive task analysis of drillers’ situation awareness during well operations

PurposeThe ability to monitor and detect anomalous changes in an oil well is a fundamental aspect of an offshore driller’s job, so as to maintain well control, the safety of the surrounding rig and minimize risk. Drillers are required to have a high level of situation awareness in order to recognise and interpret indicators that suggest that hazardous hydrocarbons may have entered the well bore (referred to as kick detection), allowing them to take the best actions to deal with the situation. The aim was to use cognitive task analysis methods to identify the expert SA skills required for the complex task of kick detection. MethodAn Applied Cognitive Task Analysis (ACTA) of the SA components required for kick detection was conducted. 1. A technical task description was produced, using material extracted from well control manuals and a five day training course. 2. The SA components required for the kick detection task using the preliminary Drillers’ SA model. 3. The ACTA data were validated by three drilling experts. ResultsThe specific cognitive skills associated with maintaining SA that were required for accurate kick detection were: attending to and recognizing changes in the drilling parameters, understanding their significance as kick indicators in the context of a mental picture of the well state, and anticipating what could result in a kick, necessitating flow checking and/or shutting in actions. LimitationsACTA methods are prone to research bias however actions were taken to minimize this. Practical implicationsTraining and work design recommendations based on the ACTA are outlined for supporting driller SA for effective well control and consequently to improve process safety. The study also illustrates how ACTA methods can be used to examine and support cognition in the workplace.

Adaptive Flexibility

Identifying the cognitive processes underlying tactical decision making is vital for two purposes: (a) reducing risk through improved training and (b) facilitating the public’s attitudes toward the legitimacy of the police and criminal justice system. Despite this, very little research has been conducted into British police decision making involving the use of firearms. This study begins to address this gap by examining the impact that expertise has on British police’s use-of-force decisions during armed confrontations. To do so, the tactical decision-making processes of 12 expert specialized firearms officers and 11 novice authorized firearms officers during armed confrontations were compared through cognitive task analysis methods. Data were coded via categories derived from theory and patterns inductively emergent within the data. The results found expert specialized firearms officers to be more flexible in adaptive responding to situational changes, while novice authorized firearms officers reported a more sequential and linear process of tactical decision making. In identifying the key features of expertise within this environment (“adaptive flexibility”), this study has theoretical and practical implications for the acceleration of authorized firearms officers’ expertise acquisition to bridge the existing gap resulting from a lack of available qualified operational force commanders.

Cognitive Technology.

Cognitive Technology.

Understanding complex clinical reasoning in infectious diseases for improving clinical decision support design.

BackgroundClinical experts’ cognitive mechanisms for managing complexity have implications for the design of future innovative healthcare systems. The purpose of the study is to examine the constituents of decision complexity and explore the cognitive strategies clinicians use to control and adapt to their information environment.MethodsWe used Cognitive Task Analysis (CTA) methods to interview 10 Infectious Disease (ID) experts at the University of Utah and Salt Lake City Veterans Administration Medical Center. Participants were asked to recall a complex, critical and vivid antibiotic-prescribing incident using the Critical Decision Method (CDM), a type of Cognitive Task Analysis (CTA). Using the four iterations of the Critical Decision Method, questions were posed to fully explore the incident, focusing in depth on the clinical components underlying the complexity. Probes were included to assess cognitive and decision strategies used by participants.ResultsThe following three themes emerged as the constituents of decision complexity experienced by the Infectious Diseases experts: 1) the overall clinical picture does not match the pattern, 2) a lack of comprehension of the situation and 3) dealing with social and emotional pressures such as fear and anxiety. All these factors contribute to decision complexity. These factors almost always occurred together, creating unexpected events and uncertainty in clinical reasoning. Five themes emerged in the analyses of how experts deal with the complexity. Expert clinicians frequently used 1) watchful waiting instead of over- prescribing antibiotics, engaged in 2) theory of mind to project and simulate other practitioners’ perspectives, reduced very complex cases into simple 3) heuristics, employed 4) anticipatory thinking to plan and re-plan events and consulted with peers to share knowledge, solicit opinions and 5) seek help on patient cases.ConclusionThe cognitive strategies to deal with decision complexity found in this study have important implications for design future decision support systems for the management of complex patients.Electronic supplementary materialThe online version of this article (doi:10.1186/s12911-015-0221-z) contains supplementary material, which is available to authorized users.

Tornado Warning Decisions Using Phased-Array Radar Data

Abstract The 2012 Phased Array Radar Innovative Sensing Experiment identified how rapidly scanned full-volumetric data captured known mesoscale processes and impacted tornado-warning lead time. Twelve forecasters from nine National Weather Service forecast offices used this rapid-scan phased-array radar (PAR) data to issue tornado warnings on two low-end tornadic and two nontornadic supercell cases. Verification of the tornadic cases revealed that forecasters’ use of PAR data provided a median tornado-warning lead time (TLT) of 20 min. This 20-min TLT exceeded by 6.5 and 9 min, respectively, participants’ forecast office and regions’ median spring season, low-end TLTs (2008–13). Furthermore, polygon-based probability of detection ranged from 0.75 to 1.0 and probability of false alarm for all four cases ranged from 0.0 to 0.5. Similar performance was observed regardless of prior warning experience. Use of a cognitive task analysis method called the recent case walk-through showed that this performance was due to forecasters’ use of rapid volumetric updates. Warning decisions were based upon the intensity, persistence, and important changes in features aloft that are precursors to tornadogenesis. Precursors that triggered forecasters’ decisions to warn occurred within one or two typical Weather Surveillance Radar-1988 Doppler (WSR-88D) scans, indicating PAR’s temporal sampling better matches the time scale at which these precursors evolve.

Cognitive Design of an App Enabling Remote Bases to Receive Unmanned Helicopter Resupply

This paper reports on a research project that combined cognitive task analysis (CTA) methods with innovative design processes to develop a handheld device application enabling a non-aviator to interact with a highly autonomous resupply helicopter. In recent military operations, unmanned helicopters have been used to resupply U.S. Marines at remote forward operating bases (FOBs) and combat outposts (COPs). This use of unmanned systems saves lives by eliminating the need to drive through high-risk areas for routine resupply. The U.S. Navy is investing in research to improve the autonomy of these systems and the design of interfaces to enable a non-aviator Marine to safely and successfully interact with an incoming resupply helicopter using a simple, intuitive handheld device application. In this research, we collected data from multiple stakeholders to develop requirements, use cases, and design storyboards that have been implemented and demonstrated during flight tests in early 2014.

Tailoring Cognitive Task Analysis (CTA) Methods for Use in Healthcare

Cognitive task analysis (CTA) methods are most widely known for their contributions to military, nuclear power plant, and aviation research. In recent years, however, these methods have been adapted and applied with increasing frequency to address issues in healthcare. CTA methods have been used in the context of designing and integrating health information technology, in pursuit of improved patient safety, and as a means to improve education and training. This panel will 1) reflect on strategies for tailoring CTA methods for use in a range of healthcare settings, 2) highlight challenges to conducting CTA in healthcare settings, and 3) highlight important contributions of CTA in addressing challenging issues in healthcare today.

Discussion Panel

Cognitive Task Analysis (CTA) has become part of the standard tool set of cognitive engineering. CTAs are routinely used to understand the cognitive and collaborative demands that contribute to performance problems, the basis of expertise, as well as the opportunities to improve performance through new forms of training, user interfaces, or decision aids. While the need to conduct CTAs has become well established, there is little in the way of available guidance with respect to ‘best practice’ for how to conduct a CTA or how to evaluate the quality of a CTA that has been conducted by others. This is an important gap as the range of consumers of CTAs is expanding to include program managers and regulators who may need to make decisions based on CTA findings. This panel brings together some of the leaders in development and application of CTA methods to address the question: Given the variety of methods available, and the lack of rigid guidance on how to perform a CTA, how does one judge the quality of a CTA?” The goal of the panel is to explore points of consensus with respect to ‘best practice’ in conducting and evaluating a CTA, in spite of differences in particular CTA methods, as well as to draw insights from unique and provocative perspectives.

Situation Awareness and Interruption Handling During Medication Administration.

Medication administration error remains a leading cause of preventable death. A gap exists in understanding attentional dynamics, such as nurse situation awareness (SA) while managing interruptions during medication administration. The aim was to describe SA during medication administration and interruption handling strategies. A cross-sectional, descriptive design was used. Cognitive task analysis (CTA) methods informed analysis of 230 interruptions. Themes were analyzed by SA level. The nature of the stimuli noticed emerged as a Level 1 theme, in contrast to themes of uncertainty, relevance, and expectations (Level 2 themes). Projected or anticipated interventions (Level 3 themes) reflected workload balance between team and patient foregrounds. The prevalence of cognitive time-sharing during the medication administration process was remarkable. Findings substantiated the importance of the concept of SA within nursing as well as the contribution of CTA in understanding the cognitive work of nursing during medication administration.

Applications of Cognitive Transformation Theory

Complex domains require cognitive work for which current approaches to training may be ill-suited. To improve training for cognitive work, Klein and Baxter have proposed Cognitive Transformation Theory (CTT), a learning theory that characterizes sensemaking processes as essential to the development of expertise. The objectives of this research were to compare CTT with the instructional strategies of two expert air traffic control instructors to evaluate the relevance of CTT’s four teaching practices, propose refinements to CTT, and identify potential instructional strategies to serve as guidance for the application of CTT. Data were collected using cognitive task analysis methods, including course observation, artifact examination, and knowledge elicitation with two instructors and seven of their students. Data were coded using categories derived from theory and patterns emergent within the data. Results suggest that many of the instructional strategies used were consistent with the teaching practices of CTT and that learning was aligned with the active sensemaking claims of CTT. An integrated set of instructional strategies and a few refinements to CTT are advanced to further its application to training in complex domains. Although this set of strategies may benefit current training practices, further research is needed to evaluate their effectiveness.

Task analysis method for procedural training curriculum development.

A central venous catheter (CVC) is an important medical tool used in critical care and emergent situations. Integral to proper care in many circumstances, insertion of a CVC introduces the risk of central line-associated blood stream infections and mechanical adverse events; proper training is important for safe CVC insertion. Cognitive task analysis (CTA) methods have been successfully implemented in the medical field to improve the training of postgraduate medical trainees, but can be very time-consuming to complete and require a significant time commitment from many subject matter experts (SMEs). Many medical procedures such as CVC insertion are linear processes with well-documented procedural steps. These linear procedures may not require a traditional CTA to gather the information necessary to create a training curriculum. Accordingly, a novel, streamlined CTA method designed primarily to collect cognitive cues for linear procedures was developed to be used by medical professionals with minimal CTA training. This new CTA methodology required fewer trained personnel, fewer interview sessions, and less time commitment from SMEs than a traditional CTA. Based on this study, a streamlined CTA methodology can be used to efficiently gather cognitive information on linear medical procedures for the creation of resident training curricula and procedural skills assessments.

Trialling the SMART Approach

It is important to develop tools that support sense-making by providing representations that help to capture the externalisation of the thinking process. The paper proposes SMART, a new method for identifying the sense-making processes of experts by combing probes with cognitive task analysis methods. The Data-Frame sense-making model is used as a theoretical frame, and the probes have been developed around the model to elicit experts’ sense-making strategies. However, we found that the SMART probes presently lacked the resolution to capture the experts sense-making and a stronger emphasis of cognitive task analysis methods and observations were required to interpret the findings.

Cognitive Task Analysis–Based Training

Cognitive task analysis (CTA) is enjoying growing popularity in both research and practice as a foundational element of instructional design. However, there exists relatively little research exploring its value as a foundation for training through controlled studies. Furthermore, highly individualized approaches to conducting CTA do not permit broadly generalizable conclusions to be drawn from the findings of individual studies. Thus, examining the magnitude of observed effects across studies from various domains and CTA practitioners is essential for assessing replicable effects. This study reports the findings from a meta-analysis that examines the overall effectiveness of CTA across practitioners and settings in relation to other means for identifying and representing instructional content. Overall, the effect of CTA-based instruction is large (Hedges’s g = 0.871). However, effect sizes vary substantially by both CTA method used and training context. Though limited by a relatively small number of studies, the notable effect size indicates that the information elicited through CTA provides a strong basis for highly effective instruction.

One Work Analysis, Two Domains: A Display Information Requirements Case Study

Work domain analyses can be time consuming, requiring extensive interviews, documentation review, and observations, among other techniques. Given the time and resources required, we examine how to generalize a work domain analysis technique, namely the hybrid Cognitive Task Analysis (hCTA) method across two domains in order to generate a common set of display information requirements. The two domains of interest are field workers troubleshooting low voltage distribution networks and telecommunication problems. Results show that there is a high degree of similarity between the two domains due to their service call nature, particularly in tasking and decision-making. While the primary differences were due to communication protocols and equipment requirements, the basic overall mission goals, functions, phases of operation, decision processes, and situation requirements were very similar. A final design for both domains is proposed based on the joint requirements.

Modeling Cognitive Strategies during Complex Task Performing Process

The purpose of this study is to examine individuals’ computer based complex task performing processes and strategies in order to determine the reasons of failure by cognitive task analysis method and cued retrospective think aloud with eye movement data. Study group was five senior students from Computer Education and Instructional Technologies Department at a state university. A computer based complex task including a logical reasoning process was developed and utilized. Data was collected with cued retrospective think aloud containing a gaze video replay and eye tracking. All the retrospective protocols were transcribed for analysis then a coding schema was developed iteratively from segments. By drawing area of interest fields(AOI) of task panel, eye movement data in specific processes and fields were analyzed by using fixation duration and fixation count metrics. None of the participants completed the task successfully within the given amount of time (max-10 min.). The findings yielded seven cognitive strategies (“information gathering-reviewing”, “identification well-defined instructions”, “cue-seeking”, “using cues”, “assumption”, “trial and error” and “crosschecking”) and their actions obtained from the task performing process. Furthermore, this process is modeled by defining sequences and the relations between the actions and cognitive strategies. It has been revealed that trial and error is the most employed strategy and participants used trial and error without reasoning. Moreover, it has been revealed out that participants mostly failed in choosing and using the right strategy.