Subjective Situation Awareness Research Articles

Investigating the role of flight phase and task difficulty on low-time pilot performance, gaze dynamics and subjective situation awareness during simulated flight.

Gaze behaviour has been used as a proxy for information processing capabilities that underlie complex skill performance in real-world domains such as aviation. These processes are highly influenced by task requirements, expertise and can provide insight into situation awareness (SA). Little research has been done to examine the extent to which gaze behaviour, task performance and SA are impacted by various task manipulations within the confines of early-stage skill development. Accordingly, the current study aimed to understand the impact of task difficulty on landing performance, gaze behaviour and SA across different phases of flight. Twenty-four low-time (<300 hours) pilots completed simulated landing scenarios under visual flight rules conditions. Traditional gaze metrics, entropybased metrics, and blink rate provided meaningful insight about the extent to which information processing is modulated by flight phase and task difficulty. The results also suggested that gaze behavior changes compensated for increased task demands and minimized the impact on task performance. Dynamic gaze analyses were shown to be a robust measure of task difficulty and pilot flight hours. Recommendations for the effective implementation of gaze behaviour metrics and their utility in examining information processing changes are discussed.

Advancing maritime technology: Evaluating situation awareness support systems for engine room monitoring at the future shore control centre

ABSTRACT Remote operation offers promising advantages as a viable transitional phase toward autonomous ships, but at the same time, it introduces new challenges in maintaining situation awareness. However, situation awareness is misunderstood as an essential construct for only navigational tasks, albeit situation awareness issues are recurrently reported during system monitoring, e.g. power plants. Hence, this within-subjects study developed the two situation awareness support systems for engine room monitoring based on information requirements and schema instantiation. Sixteen marine engineers participated in the four simulation experiments, and their objective and subjective situation awareness, fault detection rate, response speed and change blindness were evaluated. The findings describe that presenting a few critical perception elements for comprehension is promising to mitigate interface complexity and enable efficient pattern matching. However, caution is necessary for selecting the means of access to elements of low importance, as it can induce attentional tunnelling and additional cognitive load, which can devastate the situation awareness and fault-detecting performance when compounded by high cognitive load from primary monitoring tasks. This study recommends entailing the current level of situation awareness, cognitive load and stress into the support system design to determine the extent and timing of assistance for varying operational demands.

The Level of Measurement of Subjective Situation Awareness and Its Dimensions in the Situation Awareness Rating Technique (SART)

Situation awareness (SA), a measure of how well a person understands the situation, is frequently used to evaluate the safety and effectiveness of critical systems that depend on human behavior. While there are objective ways of measuring SA, subjective assessments, such as the SA rating technique (SART), are still widely used. However, it is not clear what the level of measurement is for SART-measured SA or its constituent dimensions This is a significant gap because the level of measurement determines what mathematics and statistics can be meaningfully used to synthesize and evaluate measures. This research uses a previously developed method for determining the level of measurement of psychometric ratings to evaluate the level of measurement of SART and its elements. Results show that all of the dimensions of SA can be treated as interval in most situations, but that each is on a separate interval scale. This result casts doubt on the validity of the formula SART uses to compute SA from its subcomponents. We ultimately discuss our results and explore future research directions.

Using eye movements to evaluate the effectiveness of the situation awareness rating technique scale in measuring situation awareness for smart manufacturing

Physiological indicators, including eye-tracking measures, may provide insight into human internal states in many domains, such as smart manufacturing. The Situation Awareness Rating Technique (SART) scale has been criticised for not assessing situation awareness (SA) accurately. In this study, we investigated the precision of the SART scale for assessing SA by comparing the scores to eye movement data. Thirty participants were recruited to complete a process plant monitoring task. Participants’ eye movements and SART scores were recorded and analysed. Our results moderately supported the idea that the SART scale did not accurately measure SA. We found that four dimensions in the SART scale need to be revised to reflect real SA, which may partially solve the divergence between objective and subjective SA measurements. Moreover, these findings provided solutions for designing a revised SART scale to measure the internal states of operators for smart manufacturing. Practitioner summary: Situation awareness (SA) is a critical component of decision-making and performance for smart manufacturing. The present study examines the relationships among eye movement, the SART scale, and SA for smart manufacturing in a refinery control room.

Influence of Estimated Mission Data on Air Mission Commander Situation Awareness During Datalink Loss

AbstractThis article presents the impact of displaying accurate or inaccurate position estimates on air mission commanders’ (AMC) situation awareness (SA) during datalink interruptions between AMC and unmanned aerial vehicles (UAVs). The lack of current information about UAV state and position during datalink interruptions is suspected to negatively influence the controlling AMC's SA. We assume that the AMC's SA can be supported by displaying a state estimate, based on last transmitted data, instead of displaying last known position. An experimental evaluation in a full‐mission research cockpit simulator showed improvements in subjective SA with estimate shown. Objective SA only improved, if the estimate was accurate. Although we expected a negative impact of inaccurate estimates on objective SA (since the user might falsely trust it), none could be observed. Improvements in SA with accurate estimates, but no observable degradations with inaccurate estimates suggest benefits of showing an estimate in any case.

Investigating the validity of subjective workload rating (NASA TLX) and subjective situation awareness rating (SART) for cognitively complex human–machine work

Subjective workload and situation awareness measures, such as the NASA task load index (TLX) and the situational awareness rating technique (SART), are frequently used in human–system evaluation. However, the interpretation of these ratings is debated. In this study, empirical evidence for the measures' theoretical assumptions was investigated by comparing operators' ratings collected immediately after performing a scenario and ratings collected after operators' acquisition through a video review of the scenario, knowledge of actual system states. Eighteen licensed control room operators participated in the simulator study, running 12 relatively challenging scenarios. It was found that the interpretation of TLX items involving introspection remained stable after operators acquired factual scenario knowledge, while the interpretation of items involving the perception of external events, such as situation awareness and performance, depended on the operators' scenario knowledge. The result shows that operators’ ratings could discriminate between mental effort, performance, frustration, and situation awareness. No clear evidence for the SART index as a measure of situation awareness was found. Instead, a subjective situation awareness measure developed for this study was distinct from workload and related to operator performance, showing that this type of measure warrants future investigation of its validity. The study findings help in developing measurement procedures and interpreting subjective measures. Finally, the study reveals that informing operators about the scenario can provide useful subjective ratings of situation awareness and performance. Future research should include procedures for how to inform participants adequately and efficiently in subjective assessments.

Impact of Heads-up Displays on Small Unmanned Aircraft System Operator Situation Awareness and Performance: A Simulated Study

ABSTRACT The purpose of the current study was to examine the effect of two interface types on operator situation awareness (SA), performance, and workload for small Unmanned Aircraft System (sUAS) search and rescue operations in a simulated computer-based task. The first interface type was a simulated, traditional, heads-down display (HDD) in which system information was presented separately from the view of the aircraft and the environment. The second interface type was a simulated, heads-up display (HUDs) in which the system information was overlaid on the environmental view. Results revealed a significant effect of interface type on SA, with the simulated HUD resulting in significantly higher objective SA. Interface type was found to have no significant effect on workload, performance, or subjective SA. Additionally, interactions between the order of display presentation and operator experience level were explored. This paper presents associated methods, results, and implications.

The Use of Automatic Object Detection Signals from Smart Cameras to Enhance Bridge Operators’ Situation Awareness

Errors in bridge/lock operations can have fatal consequences, which means adequate situation awareness (SA) is crucial during operators’ inspection process. Although operators from Rijkswaterstaat are well-trained and use carefully designed systems, accident reports revealed that SA is not always sufficient. Rijkswaterstaat considers smart cameras to support operators during their inspection process. Smart cameras have high detection rates, but they are not yet able to detect all relevant circumstances. This study therefore investigates whether reliable but partially inaccurate smart cameras enhance operators’ SA. Performance, eye movements and subjective SA were measured during an experiment. Results indicate that the applied (simulated) signals enhanced operators’ performance, which suggests SA was higher when assisted by smart cameras. Dwell time and time to first fixation revealed that signals were effective in gaining operators’ attention, and operators still actively searched for road users without awaiting signals. Subjective SA was not different when smart cameras were used.

The Divergence of Objective and Subjective Situation Awareness: A Meta-Analysis

A review of 37 studies that included both objective and subjective measures of situation awareness (SA) was conducted. Objective and subjective measures of SA were found to diverge across a wide range of measurement techniques. Reasons for these differences include a lack of meta-awareness about one’s own SA, poor SA/confidence calibration, and confounds with workload among some measures. A model that shows how objective and subjective SA combine to affect performance is presented.

Proceed with caution: using verbal protocol analysis to measure situation awareness

Verbal protocol analysis (VPA) is often used to elicit information about the cognitive processes of operators as it provides rich data and can be used in naturalistic settings. Recently VPA has been used to investigate the acquisition and maintenance of situation awareness (SA), and to make comparisons between groups despite a lack of research regarding the efficacy of using VPA for this purpose. This train simulator experiment investigated whether VPA can effectively measure SA. Novice and expert participants were recorded on an audio device while talking aloud throughout the trials and their verbalisations were transcribed verbatim. A coding scheme developed from the transcripts was used to code the verbalisations. Results did not support the use of VPA as a measure of SA but did show that VPA detected differences in SA errors. Potential reasons for the conflicting findings between this experiment and those conducted by other researchers are discussed.Practitioner summary: This paper examined the validity of verbal protocol analysis (VPA) as a situation awareness (SA) measure. A repeated measures experiment was conducted using a train simulator. Normal VPA did not detect changes in SA but a measure of errors did. Caution should be used when using VPA to measure SA.Abbreviations: LETSSA: low-event task subjective situation awareness technique; SA: situation awareness; SAGAT: situation awareness global assessment technique; TPD: train performance display; VPA: verbal protocol analysis

Human factor issues during remote ship monitoring tasks: An ecological lesson for system design in a distributed context

In recent years there has been increasing research and commercial activities regarding the development of remotely monitored and controlled unmanned ships. Much of this focus is related to the intended migration of operators from ship to shore and the integration of a decision support system to maintain safety of navigation. For this reason, the centralized context onboard could shift to a distributed context characterized as a ¨ship-shore system¨. Although there is substantial research on situation awareness (SA), most tend to view operators as both physically and cognitively situated in the field in a centralized system. Few are paying attention to how SA might be influenced in distributed working domains and their implications on interface design, particularly possible adaptation of those technologies used in a centralized system. In this paper, we developed a remote supervisory control prototype on top of a fully-fledged ship bridge system to support monitoring and controlling of remote simulated unmanned cargo vessels. Six participants were invited to conduct scenario-based simulation trials as proposed shore based operators. Their objective performance and subjective SA assessment was collected and analyzed. The results suggest human factor issues could remain in systems assembled by assumed reliable technological components. Prominent challenges include psychophysical and perceptual limitation for the operators, decision making latencies and automation bias which is applicable to usability issues of interfaces, deprivation of ship sense and lack of current regulatory oversight. The results have important relevance to socio-technical system and interface design that the design also needs to take the context and work domain constraints into account via an ecological approach. As socio-technical systems become less centralized and more globalized, our study suggest the necessity to incorporate the ecological concerns in design to shape the technological artefacts in a way that can truly support the operators to deal with complexity in the field.

Effects of Touch, Voice, and Multimodal Input, and Task Load on Multiple-UAV Monitoring Performance During Simulated Manned-Unmanned Teaming in a Military Helicopter.

We evaluated three interface input methods for a simulated manned-unmanned teaming (MUM-T) supervisory control system designed for Air Mission Commanders (AMCs) in Black Hawk helicopters. A key component of the U.S. Army's vision for unmanned aerial vehicles (UAVs) is to integrate UAVs into manned missions, called MUM-T (Department of Defense, 2010). One application of MUM-T is to provide the AMC of a team of Black Hawk helicopters control of multiple UAVs, offering advanced reconnaissance and real-time intelligence of flight routes and landing zones. Participants supervised a (simulated) team of two helicopters and three UAVs while traveling toward a landing zone to deploy ground troops. Participants classified aerial photographs collected by UAVs, monitored instrument warnings, and responded to radio communications. We manipulated interface input modality (touch, voice, multimodal) and task load (number of photographs). Compared with voice, touch and multimodal control resulted in better performance on all tasks and resulted in lower subjective workload and greater subjective situation awareness, ps < .05. Participants with higher spatial ability classified more aerial photographs ( r = .75) and exhibited shorter response times to instrument warnings ( r = -.58) than participants with lower spatial ability. Touchscreen and multimodal control were superior to voice control in a supervisory control task that involved monitoring visual displays and communicating on radio channels. Although voice control is often considered a more natural and less physically demanding input method, caution is needed when designing visual displays for users sharing common communication channels.

Using Situation Awareness and Workload to Predict Performance in Submarine Track Management: A Multilevel Approach.

Examine the extent to which subjective workload and situation awareness (SA) can predict variance in performance at the between- and within-person levels of analysis in a simulated submarine track management task. SA and workload are crucial constructs in human factors that are conceptualized as states that change within individuals over time. Thus, a change in an individual's subjective workload or SA over the course of performing a task should be predictive of their subsequent performance (within-person effects). However, there is little empirical evidence for this. Participants monitored displays to track the behaviors of contacts in relationship to their own ship (Ownship) and landmarks. The Situational Awareness Global Assessment Technique measured SA, and the Air Traffic Workload Input Technique measured subjective workload. When a participant's subjective workload rating increased, their subsequent performance decreased, but there was no evidence for within-person effects of SA on performance. We replicated prior between-person level effects of SA; participants with higher SA performed better than those with lower SA. Change in an individual's subjective workload rating (but not SA) was predictive of their subsequent performance. Because an increase in SA should increase the extent to which operators hold the knowledge required to perform subsequent tasks, further research is required to examine SA effects on performance at the within-person level. Adapting automation is more likely to produce optimal outcomes if based on measurement of operator states that predict future task performance, such as workload.

The Low-Event Task Subjective Situation Awareness (LETSSA) technique: Development and evaluation of a new subjective measure of situation awareness

Situation awareness (SA) is an important component of an individual's ability to function in a complex environment. As such, it is essential to have effective measures of an individual's SA. The most widely used subjective measure of SA is the Situation Awareness Rating Technique [SART]. However, SART has been criticised for not predicting performance or objective SA, and being highly correlated with workload. This paper describes the development and testing of a new subjective measure of SA, the Low-Event Task Subjective Situation Awareness (LETSSA) measure. To evaluate LETSSA a train simulator study was conducted with 23 novice and 26 expert freight train drivers. LETSSA was able to detect differences in manipulated SA and was comparable to an established objective SA measure (SAGAT). LETSSA was significantly associated with performance but not significantly associated with workload. While further validation is required, LETSSA shows promise as an effective subjective measure of SA.

The perception of team engagement reduces stress induced situation awareness overconfidence and risk-taking

Much research has examined how stress restricts objective Situation Awareness (SA). Little research, however, has focused on SA overconfidence, the notion that an individual may grasp a situation when in fact they do not. Even less SA research has examined the motivational and emotional states of individuals operating in teams in stressful environments. Expanding on recent data suggesting that stress creates SA overconfidence, not simply SA loss, the present experiment manipulated stress levels and the perception of team engagement, which is thought to be a positive motivational state of task-related well-being. Teams of Soldiers were tested in a virtual combat scenario testing shared risk-taking, objective (i.e., collaborative ability to answer SA probes), and subjective SA. Results indicated that the mere perception of above average team engagement reduced stress induced SA overconfidence and risk-taking of teams. These results suggest simple, virtually costless strategies for improving elements of SA that may impact the behavior of teams and potentially improve their decision-making.

Effects of Touch, Voice, and Multimodal Input on Multiple-UAV Monitoring During Simulated Manned-Unmanned Teaming in a Military Helicopter

A key component of the U.S. Army’s vision for future unmanned aerial vehicle (UAV) operations is to integrate UAVs into manned missions, an effort called manned-unmanned teaming (MUM-T; Department of Defense, 2010). One candidate application of MUM-T is to provide the Air Mission Commander (AMC) of a team of Black Hawk helicopters control of multiple UAVs, offering advanced reconnaissance and real-time intelligence of the upcoming flight route and landing zones. One important design decision in the development of a system to support multi-UAV control by an AMC is the selection of the interface used to control the system, for example, through a touchscreen or voice commands. A variety of input methods is feasible from an engineering standpoint, but little is known about the effect of the input interface on AMC performance. The current study evaluated three interface input methods for a MUM-T supervisory control system used by an AMC located in a Black Hawk helicopter. The evaluation was conducted with simulation software developed by General Electric. Eighteen participants supervised a team of two helicopters and three UAVs as they traveled towards a landing zone to deploy ground troops. A primary monitor, located in front of the participant, presented displays used to monitor flight instruments and to supervise the manned and unmanned vehicles that were under the AMC’s control. A secondary monitor, located adjacent to the participant, presented displays used to inspect and classify aerial photographs taken by the UAVs. Participants were responsible for monitoring and responding to instrument warnings, classifying the aerial photographs as either neutral or hostile, and responding to radio communications. We manipulated interface input modality (touch, voice, multimodal) and workload (rate of photographs to classify). Participants completed three blocks of 8.5-minute experimental trials, one for each input modality. Results indicated that touch and multimodal input methods were superior to voice input. Participants were more efficient with touch and multimodal control (compared to voice), evidenced by relatively shorter photograph classification times, a greater percentage of classified photographs, and shorter instrument warning response times. Touch and multimodal input also resulted in a greater percentage of correct responses to communication task queries, lower subjective workload, greater subjective situation awareness, and higher usability ratings. Multimodal input did not result in significant performance advantages compared to touch alone. Designers should carefully consider the performance tradeoffs when selecting from candidate input methods during system development.

Who benefits from simulator training: Personality and heart rate variability in relation to situation awareness during navigation training

ObjectiveThis study tested the relationship between individual differences and Situation Awareness (SA) during training in a navigation simulator. BackgroundSimulators have become an important tool in the training and education of maritime personnel with a view to improving decisions and performance. There are a few, inconclusive empirical studies that have focused on mapping personality characteristics and their link to the ability to generate and maintain SA during simulator training. MethodThirty-six first-year students from the Royal Norwegian Navy Officer Candidate School participated in the study. SA was measured using both subjective and observer ratings. ResultsThe results indicated that low scores on Neuroticism and high scores on Extraversion and Conscientiousness (resilient personality type) predicted both subjective and observer-rated SA. Furthermore, participants with high SA were able to modulate their Heart Rate Variability (HRV), with suppression of HRV during navigation training as well showing recovery of HRV. ApplicationThe potential applications of this research include the assessment of personality differences as a tool in selecting navigators, and the use of HRV as an objective index of adaptability to environmental demands.

A Baseline Evaluation of Situation Awareness For Electric Power System Operation Supervisors

High levels of situation awareness (SA) are critical for successful power transmission and distribution (T&D) operations. Robust and efficient operations within power T&D require that system operators maintain high levels of situation awareness to fully understand the current state of the complex and dynamically changing power grid system, and to successfully project future system states in order to respond in a timely manner. This paper discusses the results of an evaluation study conducted at an electric power control room facility to form a baseline understanding of the impact that current control room conditions have on system operator SA. An objective measure of operator SA was collected during the study through the use of the Situation Awareness Global Assessment Technique (SAGAT), which was modified to accommodate the real-time operational demands of the control room. The study also gathered subjective SA measures for the system operator position. The insights gleaned from this analysis can provide guidance when generating recommendations for modifying or redesigning control center displays to support operator situation awareness.

The role of metacognition in the relationship between objective and subjective measures of situation awareness

The present paper is concerned with the relationship between subjective and objective measures of situation awareness (SA) within the context of a realistic command and control (C2) scenario. A 2-day scenario involved a joint military–civilian C2 exercise relating to the (crisis) management of a large event held in a metropolitan area. The role of participants comprised both the acquisition and maintenance of SA of each crisis event and the coordination of an appropriate course of action. Two measures of SA were collected at the end of each day–Situation Awareness Rating Technique (SART) and Quantitative Assessment of Situation Awareness (QUASA). No correlation was found between QUASA confidence and accuracy scores. However, SART SA, SART S(upply) and SART U(nderstanding) scores showed a negative correlation with QUASA accuracy scores. These results are interpreted in light of Koriat et al.'s (Koriat, A., et al., 2008. Information-based and experience-based metacognitive judgement: evidence from subjective confidence. In: J. Dunlosky and R.A. Bjork, eds. Handbook of metamemory and memory. New York: Taylor & Francis, 117–135) framework of information- and experience-based metacognitive judgements and practical implications for the subjective measurement of SA are discussed. SA is a critical real-world concept, yet its measurement remains ambiguous. The relationship between subjective and objective SA measures was examined within the context of a realistic C2 scenario. Using theories of metacognition, unique insights were gained into how subjective SA plays a strategic role in the acquisition of SA.

A Field Test of Two Methods for Assessing Infantry Situation Awareness

Two methods for assessing situation awareness (SA) were field tested during infantry exercises. Eight platoons of U.S. Military Academy cadets executed an infantry mission during summer field training exercises. A subjective SA measure, the Mission Awareness Rating Scale (MARS), was given to each platoon leader and one squad leader from each platoon to self-assess both SA and cognitive workload demanded by the tasks. In addition, infantry expert observers rated each platoon and squad leader using the Situation Awareness Behaviorally Anchored Rating Scale (SABARS). Both MARS and SABARS had been validated previously in missions conducted in a virtual environment. In the current field test, both instruments showed evidence of successfully measuring SA. Both instruments show promise for assessing SA in the field, or in other venues where more obtrusive measurement protocols are undesirable.