Situation Awareness Rating Technique Research Articles

Operator situation awareness and physiological states during offshore well control scenarios

Incident reviews of oil spill events (both large and small) suggest that human error is a contributor to 50% of well control incidents (primarily kicks). The purpose of this study was to examine operator situation awareness (SA) and associated physiological load, using heart rate and heart rate variability (HRV), during different simulated offshore well control scenarios (drilling and tripping) and criticality levels (failure presented or not). Ten trained participants completed four scenarios (tripping non-failure, tripping failure, drilling non-failure, and drilling-failure) in an experimental session, lasting ∼6 h. Measures were obtained for each scenario, including speed and accuracy of the task performance, composite scores obtained from the Situational Awareness Rating Technique (SART), and operator heart rate and heart rate variability measures. Greater errors were found in kick-related failure events, and drilling scenarios were associated with longer reaction times. Participants perceived lower SA levels during drilling scenarios, and the was observed for lowest SA during drilling failure scenarios. Finally, while physiological responses did not differ significantly for any of the four scenarios, elevated heart rate was observed with drilling and failure-related scenarios. High variability in participant covert and overt responses may increase the challenges associated with classifying high-risk well control scenarios. It is critical that scenario planners understand and recognize the variability in driller situation awareness and associated physiological load when planning for alternative future scenarios.

From Paris Agreement to Action: Enhancing Climate Change Familiarity and Situation Awareness

The Paris Agreement was a monumental stride towards global climate change governance. It unlocked the climate change gridlock, introducing country-subjective commitments and a five-year review mechanism. To support the implementation of the Paris Agreement, we designed the Nzoia WeShareIt climate change game. Game sessions were conducted in June and July 2015, and 35 respondents completed a pre- and post-game situation awareness (SA) questionnaire and an in-game performance measurement system. The questionnaire uses a 10-dimensional situation awareness rating technique (SART). Subsequently, we conducted a factorial MANOVA (multivariate analysis of variance) to assess the interaction effects between familiarity, team, and gender. Results indicate an increase in situation awareness. However, policymakers’ action was not contingent on the increased SA only, there was a significant interaction effect between familiarity and SA, to lead to climate change actions. Therefore, we recommend more emphasis on the role of familiarity in enhancing SA and, subsequently, supporting the implementation to the Paris five-year review country commitments. We also recommend the increased usage of symbols and capacity development of policymakers on connective capacity to enable them to span the climate change boundaries.

Examining the Relationship between Construction Workers’ Visual Attention and Situation Awareness under Fall and Tripping Hazard Conditions: Using Mobile Eye Tracking

The risk of major occupational accidents involving tripping hazards is commonly underestimated with a large number of studies having been conducted to better understand variables that affect situation awareness: the ability to detect, perceive, and comprehend constantly evolving surroundings. An important property that affects situation awareness is the limited capacity of the attentional system. To maintain situation awareness while exposed to tripping hazards, a worker needs to obtain feedforward information about hazards, detect immediate tripping hazards, and visually scan surroundings for any potential environmental hazards. Despite the importance of situation awareness, its relationship with attention remains unknown in the construction industry. To fill this theoretical knowledge gap, this study examines differences in attentional allocation between workers with low and high situation awareness levels while exposed to tripping hazards in a real construction site. Participants were exposed to tripping hazards on a real jobsite while walking along a path in the presence of other workers. Situation awareness was measured using the situation awareness rating technique, and subjects’ eye movements were tracked as direct measures of attention via a wearable mobile eye tracker. Investigating the attentional distribution of subjects by examining fixation-count heat maps and scan paths revealed that as workers with higher situation awareness walked, they periodically looked down and scanned ahead to remain fully aware of the environment and its associated hazards. Furthermore, this study quantitatively compared the differences between the eye-tracking metrics of worker with different situation awareness levels (low versus high) using permutation simulation. The results of the statistical analysis indicate that subjects did not allocate their attention equally to all hazardous areas of interest, and these differences in attentional distribution were modulated by the workers’ level of situation awareness. This study advances theory by presenting one of the first attempts to use mobile eye-tracking technology to examine the role of cognitive processes (i.e., attention) in human error (i.e., failure to identify a hazard) and occupational accidents.

The Low-Event Task Subjective Situation Awareness (LETSSA) technique: Development and evaluation of a new subjective measure of situation awareness

Situation awareness (SA) is an important component of an individual's ability to function in a complex environment. As such, it is essential to have effective measures of an individual's SA. The most widely used subjective measure of SA is the Situation Awareness Rating Technique [SART]. However, SART has been criticised for not predicting performance or objective SA, and being highly correlated with workload. This paper describes the development and testing of a new subjective measure of SA, the Low-Event Task Subjective Situation Awareness (LETSSA) measure. To evaluate LETSSA a train simulator study was conducted with 23 novice and 26 expert freight train drivers. LETSSA was able to detect differences in manipulated SA and was comparable to an established objective SA measure (SAGAT). LETSSA was significantly associated with performance but not significantly associated with workload. While further validation is required, LETSSA shows promise as an effective subjective measure of SA.

Nationwide critical infrastructure monitoring using a common operating picture framework

This paper describes the efforts involved in designing a common operating picture system for monitoring large-scale critical infrastructures. The design leverages the Joint Directors of Laboratories (JDL) data fusion model to enable the integration of different critical infrastructure systems with their dependency relations. The resulting Situational Awareness of Critical Infrastructure and Networks (SACIN) framework offers a platform that provides a common operating picture of a critical infrastructure. A generic data collection component customized to each source system generates events and facilitates JDL level 0 integration. An analysis component collects events and data to produce meaningful information about the current state and future impact estimates in accordance with JDL levels 1 to 3. A brokered architecture supports level 4 control by various components and a JDL level 5 user interface is offered via a web application. Interviews of infrastructure subject matter experts were conducted to obtain the situational awareness requirements. By applying key situational awareness oriented design principles to the situational awareness requirements, a user interface was created for organizing information based on operator situational awareness needs and supporting key cognitive mechanisms that transform data into high levels of situational awareness. Situational awareness measures were used to assess operator performance during critical infrastructure tasks – a “freeze-probe” recall approach (Situational Awareness Global Assessment Technique (SAGAT)), a post-trial subjective rating approach (Situational Awareness Rating Technique (SART)) and the System Usability Scale (SUS). The results indicate that the supply of attentional resources (SART supply) and overall SAGAT score best predict the performance levels of operators.

Situation awareness measures for simulated submarine track management.

The aim of this study was to examine whether the Situation Present Assessment Method (SPAM) and the Situation Awareness Global Assessment Technique (SAGAT) predict incremental variance in performance on a simulated submarine track management task and to measure the potential disruptive effect of these situation awareness (SA) measures. Submarine track managers use various displays to localize and track contacts detected by own-ship sensors. The measurement of SA is crucial for designing effective submarine display interfaces and training programs. Participants monitored a tactical display and sonar bearing-history display to track the cumulative behaviors of contacts in relationship to own-ship position and landmarks. SPAM (or SAGAT) and the Air Traffic Workload Input Technique (ATWIT) were administered during each scenario, and the NASA Task Load Index (NASA-TLX) and Situation Awareness Rating Technique were administered postscenario. SPAM and SAGAT predicted variance in performance after controlling for subjective measures of SA and workload, and SA for past information was a stronger predictor than SA for current/future information. The NASA-TLX predicted performance on some tasks. Only SAGAT predicted variance in performance on all three tasks but marginally increased subjective workload. SPAM, SAGAT, and the NASA-TLX can predict unique variance in submarine track management performance. SAGAT marginally increased subjective workload, but this increase did not lead to any performance decrement. Defense researchers have identified SPAM as an alternative to SAGAT because it would not require field exercises involving submarines to be paused. SPAM was not disruptive, but it is potentially problematic that SPAM did not predict variance in all three performance tasks.

A Quantitative Situational Awareness Model of Pilot

The pilot’s situational awareness (SA) mainly depends on the perception, understanding and prediction of the information obtained from the man-machine display interface, therefore there is an intrinsic link between the display interface design and the level of pilot’s SA. A quantitative SA model, which considers the influence of information importance on SA, the characteristics of human cognition and the Bayesian conditional probability theory, was proposed based on the attention allocation model built previously. This model was expected to be used to predict the pilot’s SA levels under different task conditions. In order to verify the validity of the SA model, 20 volunteers were recruited to perform the instrument supervision tasks in four kinds of task, at the same time, Situation Awareness Global Assessment Technique (SAGTA), 3-dimensional Situational Awareness Rating Technique (3-D SART), and eye movement measurement were adopted for the evaluation of SAs. The experimental results reveal that, the SAGAT is the most effective approach to measure the level 2 SA, showing that the correct rate of SAGAT has the most similar changing trend with the task performance, and the SA model is validated since the changing trend of SA predicted by the model is highly correlated with the measurement indices.

An interface redesign for the feed-water system of the advanced boiling water reactor in a nuclear power plant in Taiwan

A well-designed human-computer interface for the visual display unit in the control room of a complex environment can enhance operator efficiency and, thus, environmental safety. In fact, a cognitive gap often exists between an interface designer and an interface user. Therefore, the issue of the cognitive gap of interface design needs more improvement and investigation. This is an empirical study that presents the application of an ecological interface design (EID) using three cases and demonstrates that an EID framework can support operators in various complex situations. Specifically, it analyzes different levels of automation and emergency condition response at the Lungmen Nuclear Power Plant in Taiwan. A simulated feed-water system was developed involving two interface styles. This study uses the NASA Task Load Index to objectively evaluate the mental workload of the human operators and the Situation Awareness Rating Technique to subjectively assess operator understanding and response, and is a pilot study investigating EID display format use at nuclear power plants in Taiwan. Results suggest the EID-based interface has a remarkable advantage over the original interface in supporting operator performance in the areas of response time and accuracy rate under both normal and emergency situations and provide supporting evidence that an EID-based interface can effectively enhance monitoring tasks in a complex environment.

Modeling the situation awareness by the analysis of cognitive process.

To predict changes of situation awareness (SA) for pilot operating with different display interfaces and tasks, a qualitative analysis and quantitative calculation joint SA model was proposed. Based on the situational awareness model according to the attention allocation built previously, the pilot cognitive process for the situation elements was analyzed according to the ACT-R (Adaptive Control of Thought, Rational) theory, which explained how the SA was produced. To verify the validity of this model, 28 subjects performed an instrument supervision task under different experiment conditions. Situation Awareness Global Assessment Technique (SAGAT), 10-dimensional Situational Awareness Rating Technique (10-D SART), performance measure and eye movement measure were adopted for evaluating SAs under different conditions. Statistical analysis demonstrated that the changing trend of SA calculated by this model was highly correlated with the experimental results. Therefore the situational awareness model can provide a reference for designing new cockpit display interfaces and help reducing human errors.

Comparison of SA Measurement Techniques in a Human-Robot Team Task

Increasingly intelligent and capable robots may lead to human-robot interaction characterized by close collaboration with robots as team members rather than remotely manipulated tools. In this mode of operation, it is important that robot team members contribute to, rather than limit, the situation awareness (SA) of soldiers. An exploratory study was conducted in which the effects of individual differences and mission factors on three measures of SA were compared. The Situational Awareness Rating Technique (SART) was the best metric for capturing elements of SA that were useful in mission performance, although the results suggest that other measures should not be disregarded. Practical implications for the use of SA metrics in human-robot teams are discussed.

Assessing mental workload and situation awareness in the evaluation of computerized procedures in the main control room

Computerized procedure (CP) system has been developed in nuclear power plant (NPP) instrumentation and control (I&C) system. The system may include normal operating procedures (OPs), abnormal operating procedures (AOPs), alarm response procedures (ARPs), surveillance test procedures (STPs) and/or emergency operating procedures (EOPs). While there are many ways to evaluate computerized procedures design, the user's mental workload and situation awareness (SA) are particularly important considerations in the supervisory control of safety-critical systems. Users’ mental workload and situation awareness may be influenced by human factor issues relating to computerized procedures, e.g., level of automation, dealing with (partially) unavailable I&C, switching to back-up system (e.g., paper-based procedures). Some of the positive impacts of CPs on operator performance include the following: tasks can be performed more quickly; overall workload can be reduced; cognitive workload can be minimized; fewer errors may be made in transitioning through or between procedures. However, various challenges have also been identified with CP systems. These should be addressed in the design and implementation of CPs where they are applicable. For example, narrower “field of view” provided by CP systems than with paper-based procedures could reduce crew communications and crewmember awareness of the status and progress through the procedure. Based on a human factors experiment in which each participant monitored and controlled multiple simulated reactors, this study applied the NASA-TLX instrument for assessing mental workload. For the assessment of situation awareness (SA), the present research used the situational awareness rating technique (SART). In support of summarizing the results of user interface evaluation along multiple dimensions (e.g., workload, SA), we propose advantages for the CPs compared to the paper-based procedures.

The role of metacognition in the relationship between objective and subjective measures of situation awareness

The present paper is concerned with the relationship between subjective and objective measures of situation awareness (SA) within the context of a realistic command and control (C2) scenario. A 2-day scenario involved a joint military–civilian C2 exercise relating to the (crisis) management of a large event held in a metropolitan area. The role of participants comprised both the acquisition and maintenance of SA of each crisis event and the coordination of an appropriate course of action. Two measures of SA were collected at the end of each day–Situation Awareness Rating Technique (SART) and Quantitative Assessment of Situation Awareness (QUASA). No correlation was found between QUASA confidence and accuracy scores. However, SART SA, SART S(upply) and SART U(nderstanding) scores showed a negative correlation with QUASA accuracy scores. These results are interpreted in light of Koriat et al.'s (Koriat, A., et al., 2008. Information-based and experience-based metacognitive judgement: evidence from subjective confidence. In: J. Dunlosky and R.A. Bjork, eds. Handbook of metamemory and memory. New York: Taylor & Francis, 117–135) framework of information- and experience-based metacognitive judgements and practical implications for the subjective measurement of SA are discussed. SA is a critical real-world concept, yet its measurement remains ambiguous. The relationship between subjective and objective SA measures was examined within the context of a realistic C2 scenario. Using theories of metacognition, unique insights were gained into how subjective SA plays a strategic role in the acquisition of SA.

A Comparative Analysis of Sagat and Sart for Evaluations of Situation Awareness

Situation awareness (SA) has become an important criterion for systems evaluation efforts. Several measures of SA have been developed, the most widely used among them being the Situation Awareness Global Assessment Technique (SAGAT) and the Situational Awareness Rating Technique (SART). SAGAT provides an objective measure of SA based on queries during freezes in a simulation. SART provides a subjective rating of SA by operators. This paper presents a direct comparison of the two measures which were used within a display evaluation study. It was found that both SART and SAGAT contributed sensitivity and diagnosticity regarding the effects of the display concept. The SART measure was highly correlated with subjective measures of confidence level, a simple subjective SA measure and a subjective performance measure. The SAGAT and SART measures were not correlated with each other. The implications of these findings for the interpretation of subjective SA measures are discussed as well as advantages and disadvantages of both measurement approaches.

Computer Modeling of Operator Mental Workload and Situational Awareness in Simulated Air-to-Ground Combat: An Assessment of Predictive Validity

The predictive validity of computer modeling of operator mental workload and situational awareness during a simulated air-to-ground combat mission was assessed. In Phase 1 of the study, 12 participants completed a series of simulated flights and provided Subjective Workload Assessment Technique (SWAT) ratings of workload and Situational Awareness Rating Technique (SART) ratings of situational awareness. In Phase 2, computer models of the combat mission were constructed using the Micro Saint modeling tool. The visual, auditory, kinesthetic, cognitive, and psychomotor components of workload associated with each task comprising the mission were estimated and used to derive measures of average and peak workload. The results from the flight-simulator data versus the Micro Saint data were similar but not identical, indicating that the computer models were partially but not completely valid predictors of mental workload and situational awareness. The computer modeling appeared to be a more effective predictor of situational awareness than mental workload.

Situation Awareness as a Predictor of Performance for En Route Air Traffic Controllers

Air traffic control instructors controlled simulated traffic while a variety of techniques for determining situation awareness (SA) were implemented. SA was assessed using a self-report measure (the Situation Awareness Rating Technique, or SART), a query method that removed information on the plan-view display (the Situation Awareness Global Assessment Technique, or SAGAT), a query technique that did not have a memory component (the Situation-Present Assessment Method, or SPAM), and the detection of errors integrated into the scenarios (implicit performance). We used these measures of SA together with a measure of workload, NASA Task Load Index (TLX), to predict two different performance measures: (1) an over-the-shoulder, subjective assessment by a subject matter expert (SME), and (2) a count of the number of control actions remaining to be performed at the end of the scenario. The SME evaluation was predicted by workload and the controller’s appreciation of both the present and the future. The remaining-actions count (RAC) was predicted by the controller’s appreciation of the future. In fact, an appreciation of the present led to poorer RAC scores: the better the participant was at answering questions about the present or the better he or she understood the present situation, the larger the number of actions that remained to be performed. The results have implications for the relationships among workload, SA, and performance, and suggest limitations on several of the measures currently proposed as SA techniques. The results confirm that future versus present is an important conceptual difference in ATC. More important, they suggest that a controller who remains overly focused on the present may do so at the expense of the future.

A Test-Battery Approach to Cognitive Engineering: To Meta-Measure or not to Meta-Measure, that is the Question!

This paper describes a task battery approach to system evaluation and cognitive engineering. The limitations of current techniques are discussed. An alternative paradigm using both task-specific measures and design driver measures is suggested. Such high level design driver measures have been referred to as meta-measures (Hardiman et al, 1995). In an experiment conducted to evaluate the utility of using meta-measures within the context of cockpit system assessment, an explanatory display for air combat decision support was tested within a realistic flight simulation. Multiple measures of performance were taken, together with the Situational Awareness Rating Technique (SART) as a meta-measure of Situational Awareness. Results showed that although task-specific measures clearly indicated one candidate design as producing better performance, they allowed little explanation of the underlying mechanism. Concurrent analysis of the meta-measure, however, enabled clarification of the cause of the performance differences, in this case through improved understanding, more information being available, and the information being more task relevant. The potential advantages of this approach for cognitive engineering are discussed.

Multi-Modal Cockpit Warnings: Pictures. Words. or Both?

This paper examines the requirement for, and benefits of, multi-modal information presentation in cockpit warning systems. It also attempts to provide an account of the nature and levels of processing involved when information from one source is integrated with a supposedly redundant additional source to provide a performance gain in choice reaction time (RT) tasks, and its applicability to other cockpit systems. An experiment is described which used warning/caution ‘icons’ (pictorial representations of danger situations) and verbal warning messages, both singly and in combination. The visual icons were generated by RAF aircrew, using an iterative design process, as being meaningful pictorial representations of real-world warnings. Subjects were required to identify whether the situations presented warnings i.e. high priority/immediate action or cautions i.e. low priority/immediate awareness. The results obtained showed a significant decrease in response latencies when correlated bi-modal information was given as compared to the uni-modal conditions. The high level of abstraction of these icons strongly implied that the performance gains occurring must be as the result of the integration of ‘information’ rather than ‘data’. Subjective Situational Awareness Rating Technique (SART) scores also showed that benefits may also be accrued through reduced workload and increased depth of understanding. The results of these experiments are considered in terms of current Information Processing and Neural Network theories and an attempt to provide a cognitive model of this integrality effect is also described.

Workload or Situational Awareness?: TLX vs. SART for Aerospace Systems Design Evaluation

Both workload and situational awareness (SA) have been recognised as important considerations in aerospace system design evaluation. This paper considers the use of subjective techniques for the measurement of workload and SA, namely the NASA TLX workload scale and the Situational Awareness Rating Technique (SART), and compares the diagnosticity and sensitivity of each within a computer-graphics flight simulation environment. Both the 10-dimensional and 3-dimensional implementations of SART were tested. Twelve RAF pilots were asked to rate a videotape of an air combat flight simulation sequence, using the TLX and SART techniques. Task difficulty was varied between the flight sequences used. The experimental group was split according to their flying experience. Task difficulty and experience were analysed as the main independent variables in the experiment. Results showed that although both scales were sensitive to task difficulty, only SART showed any differences between experience conditions. This implies that SART may have some added utility where operator capability or domain expertise is a design variable. Intercorrelations showed a substantial overlap between the two scales on a number of factors. These factors are discussed to attempt to clarify both differences and similarities between the scales, and also the degree of commonality between the concepts of workload and SA. The implications of such differences are considered within the application domain of aerospace systems design and evaluation.