Improved Performance Research Integration Tool Research Articles

Verification and Validation of Cognitive Workload Models for Adaptive Automation Tasks

Technological advances that seek to address future operational challenges abound. While advanced capabilities are being developed, there is an important space for human design considerations, including cognitive workload. One proposed solution to improve cognitive workload management is adaptive automation (AA). This research used a novel, model-based approach to assess the impacts of AA on cognitive workload. This assessment modeled the tasks in NASA’s Multi-attribute Task Battery-II (MATB-II) using the Improved Performance Research Integration Tool (IMPRINT). The effort sought to investigate the relationship of cognitive workload, situation awareness, and performance through three human-in-the-loop studies with 120 participants using MATB-II. The research also attempted to validate cognitive workload models from IMPRINT. The IMPRINT models were representative of the MATB tasks with statistically significant differences between workload conditions, which mirrored the models’ predictions. The results demonstrated that AA system task models can be developed using IMPRINT to provide design recommendations.

Blink Rate as a Measure of Driver Workload during Simulated Driving

Cognitive distraction has been identified as a critical factor for automobile crashes in the United States. Previous research indicates that cognitive tasks such as phone conversations may take attention away from the driving task, degrading driving performance. One continuing issue with cognitive distraction research is establishing a reliable, continuous, online measure of driver workload. Blink rate is a potential candidate for such a measure. The present driving simulator experiment examines whether blink rate is sensitive to variations of driver workload while controlling the vehicle. Participants will navigate simulated environments while performing the auditory 0-and 2-back task with their blink and driving performance data continuously recorded during the drive. Blink and performance data will be compared not only between the different workload conditions, but also to predictions of a human driver performance workload model based on the Improved Performance Research Integration Tool (IMPRINT), developed by the US Army Research Laboratory.

Applying ergonomics within the multi-modelling paradigm with an example from multiple UAV control

This article presents a position statement on using ergonomics in conjunction with the multi-modelling paradigm. Multi-modelling is a computational approach to combine models of systems and components for design and simulation of cyber physical systems and systems of systems. Despite potentially significant benefits in terms of more human-centric system modelling, there is limited evidence of the application of ergonomics within multi-modelling. This article presents the case for applying ergonomics within multi-modelling. We open with an introduction to multi-modelling and benefits, applications and gaps for ergonomics in multi-modelling, and of potentially useful models from ergonomics. We then describe a proof-of-concept implementation of ergonomics within a multi-model of UAV control. This demonstrates that as well as user-centred modelling, this approach supports ergonomics in how we can access rich systems models, and the collaborative value of applying ergonomics theory in systems design. Practitioner Summary: Examines multi-modelling, a computational approach for complex modelling, and the contribution of ergonomics. An autonomous UAV test implementation demonstrates the application of ergonomics knowledge for improving design and evaluation processes, and how multi-modelling can give ergonomics access to rich systems models. Abbreviations: ACT-R: adaptive control of thought—rational; API: application programming interface; CFD: computational fluid dynamics; COTS: commerical off the shelf; CPS: cyber-physical system; CT: continuous time; DE: discrete event; DSE: design space exploration; FME: finite element modeling; FMI: functional mock-up interface; FMU: functional mock-up unit; GOMS: goals, operators, methods, selections; HCI: human-computer interaction; IMPRINT: improved performance research integration tool; INTO-CPS: integrated toolchain for cyber-physical system modeling; KLM: keystroke level model; MPC: model-predictive control; SysML: system markup language; SoS: system of system; UAV: unmanned aerial vehicle

Simulation-Based Evaluation of Adaptive Automation Revoking Strategies on Cognitive Workload and Situation Awareness

Adaptive systems have used a variety of approaches to return the task responsibility back to the user, including restoration of performance or physiological thresholds, leveling of task loads, and user-initiated deactivation. Little attention has been paid to evaluating the relative effectiveness of these various hand-off mechanisms, or the impact that these mechanisms may have on operator workload and situation awareness (SA). This study uses an Improved Performance Research Integration Tool simulation, based upon a human-in-the-loop study, to examine two different revoking strategies (minimum duration and workload threshold) across numerous levels in order to identify impacts on operator workload and SA. This study finds that operator workload and SA are affected by the type of revoking strategy as well as the particular level used for that strategy. For the scenario examined, automation revoking that relied upon a minimum duration of at least 5 s was found to be more effective in reducing workload and increasing SA than using a revoking strategy based upon workload thresholds.

Informing System Design Using Human Performance Modeling

ABSTRACTHumans play a key role in the operation and support of most systems and model‐based systems engineering (MBSE) offers new opportunities to properly consider human capabilities and involvement. This research presents an approach for systems engineers to integrate system models with human performance modeling for early and more effective system design. Unlike analyses using traditional physics‐based models found in most extant MBSE literature, adjusting system parameters for human‐based analyses can greatly impact the design of the system itself. Adjusting a human‐system parameter can lead to design implications including adjustments to task allocation, process and workflow, and interface design. To demonstrate this, a quantitative case‐study approach is used. Starting with a set of Systems Modeling Language (SysML) diagrams, a task analysis is performed to inform an “as is” model of human performance in the Improved Performance Research Integration Tool (IMPRINT). An alternative IMPRINT model is created with varying design parameters and utilized to perform a trade study. Through the analysis, constraints and assumptions placed on the human are verified and the results of varying automation estimated. With current design emphasis in MBSE and model‐based engineering (MBE), there is great opportunity to emphasize human considerations and integrate human performance analysis.

Assessing Continuous Operator Workload With a Hybrid Scaffolded Neuroergonomic Modeling Approach.

We aimed to predict operator workload from neurological data using statistical learning methods to fit neurological-to-state-assessment models. Adaptive systems require real-time mental workload assessment to perform dynamic task allocations or operator augmentation as workload issues arise. Neuroergonomic measures have great potential for informing adaptive systems, and we combine these measures with models of task demand as well as information about critical events and performance to clarify the inherent ambiguity of interpretation. We use machine learning algorithms on electroencephalogram (EEG) input to infer operator workload based upon Improved Performance Research Integration Tool workload model estimates. Cross-participant models predict workload of other participants, statistically distinguishing between 62% of the workload changes. Machine learning models trained from Monte Carlo resampled workload profiles can be used in place of deterministic workload profiles for cross-participant modeling without incurring a significant decrease in machine learning model performance, suggesting that stochastic models can be used when limited training data are available. We employed a novel temporary scaffold of simulation-generated workload profile truth data during the model-fitting process. A continuous workload profile serves as the target to train our statistical machine learning models. Once trained, the workload profile scaffolding is removed and the trained model is used directly on neurophysiological data in future operator state assessments. These modeling techniques demonstrate how to use neuroergonomic methods to develop operator state assessments, which can be employed in adaptive systems.

Performing System Tradeoff Analyses Using Human Performance Modeling

Humans perform critical functions in nearly every system, making them vital to consider during system development. Human Systems Integration (HSI) would ideally permit the human’s impact on system performance to be effectively accounted for during the systems engineering (SE) process, but effective processes are often not applied, especially in the early design phases. Failure to properly account for human capabilities and limitations during system design may lead to unreasonable expectations of the human. The result is a system design that makes unrealistic assumptions about the human, leading to an overestimation of the human’s performance and thus the system’s performance. This research proposes a method of integrating HSI with SE that allows human factors engineers to apply Systems Modeling Language (SysML) and human performance simulation to describe and communicate human and system performance. Using these models, systems engineers can more fully understand the system’s performance to facilitate design decisions that account for the human. A scenario is applied to illustrate the method, in which a system developer seeks to redesign an example system, Vigilant Spirit, by incorporating system automation to improve overall system performance. The example begins by performing a task analysis through physical observation and analysis of human subjects’ data from 12 participants employing Vigilant Spirit. This analysis is depicted in SysML Activity and Sequence Diagrams. A human-in-the-loop experiment is used to study performance and workload effects of humans applying Vigilant Spirit to conduct simulated remotely-piloted aircraft surveillance and tracking missions. The results of the task analysis and human performance data gathered from the experiment are used to build a human performance model in the Improved Performance Research Integration Tool (IMPRINT). IMPRINT allows the analyst to represent a mission in terms of functions and tasks performed by the system and human, and then run a discrete event simulation of the system and human accomplishing the mission to observe the effects of defined variables on performance and workload. The model was validated against performance data from the human-subjects’ experiment. In the scenario, six different scan algorithms, which varied in terms of scan accuracy and speed, were simulated. These algorithms represented different potential system trades as factors such as various technologies and hardware architectures could influence algorithm accuracy and speed. These automation trades were incorporated into the system’s block definition (BDD), requirements, and parametric SysML diagrams. These diagrams were modeled from a systems engineer’s perspective; therefore they originally placed less emphasis on the human. The BDD portrayed the structural aspect of Vigilant Spirit, to include the operator, automation, and system software. The requirements diagram levied a minimum system-level performance requirement. The parametric diagram further defined the performance and specification requirements, along with the automation’s scan settings, through the use of constraints. It was unclear from studying the SysML diagrams which automation setting would produce the best results, or if any could meet the performance requirement. Existing system models were insufficient by themselves to evaluate these trades; thus, IMPRINT was used to perform a trade study to determine the effects of each of the automation options on overall system performance. The results of the trade study revealed that all six automation conditions significantly improved performance scores from the baseline, but only two significantly improved workload. Once the trade study identified the preferred alternative, the results were integrated into existing system diagrams. Originally system-focused, SysML diagrams were updated to reflect the results of the trade analysis. The result is a set of integrated diagrams that accounts for both the system and human, which may then be used to better inform system design. Using human performance- and workload-modeling tools such as IMPRINT to perform tradeoff analyses, human factors engineers can attain data about the human subsystem early in system design. These data may then be integrated into existing SysML diagrams applied by systems engineers. In so doing, additional insights into the whole system can be gained that would not be possible if human factors and systems engineers worked independently. Thus, the human is incorporated into the system’s design and the total system performance may be predicted, achieving a successful HSI process.

Modeling Performance Measures and Self-Ratings of Workload in a Visual Scanning Task

Mental workload is the amount of demand on an individual’s limited mental resources and thus is an important consideration in human factors research. This research focuses on workload from two primary methods of measuring it – self-ratings of workload and performance. An experiment to test workload involved the manipulation of the number of tasks to be performed at once and the time available to respond to the task or tasks. The results show that performance changes by shifting between ceiling, linear decrease, and floor performance as workload increases. SWAT ratings of workload followed the same pattern. We conclude that the IMPRINT (Improved Performance Research Integration Tool; Archer & Adkins, 1999) modeling system should maintain its existing method of modeling self ratings of workload, but that they may make use of a new algorithm based on this data to model performance as workload changes.

Improved Performance Research Integration Tool (IMPRINT): Human Performance Modeling for Improved System Design

Identifying human factors issues before systems are built and demonstrating the importance of those issues to decision makers outside the human factors community is difficult. Modeling and simulation (M&S) has become a critical part of tackling this challenge, but the true impact of M&S is made when the results can be translated into predictions that are important to the decision makers such as future system performance. Through the use of the Improved Performance Research Integration Tool (IMPRINT), analysts are able to quantify the effect of human operator performance on system performance. IMPRINT is a task network modeling tool designed to help assess the influence of the human operator on system performance throughout the system lifecycle. This demonstration will provide a brief overview of IMPRINT and its capabilities and highlight new features that provide enhanced modeling capabilities and better results visualization. It will also feature a look at the new Multimodal Interface Design Support (MIDS) tool plug-in that provides users with analysis specific multimodal design guidelines they could be implemented into the system design to minimize mental overload and improve system performance.

Modeling the Workload-Performance Relationship

Human factors research is often focused on the mental workload that is required to perform a task or set of tasks with the goal of reducing workload to make systems easier to manage. The Improved Performance Research Integration Tool (IMPRINT) includes an algorithm to predict mental workload. The algorithm was developed using subject matter expert ratings of workload tasks. We aimed to enhance this capability by developing algorithms using data from four new studies investigating change in performance as demands on mental resources increase. The results indicate three task types of similar difficulty and one task type of much greater difficulty. We then map these to our hypothesized workload function. Finally, we propose a way forward in modeling performance as a function of workload in IMPRINT.

Modeling Human Performance with Environmental Stressors: A Case Study of the Effect of Vehicle Motion

Human performance modeling tools are used to predict mission performance as a function of human performance. The U.S. Army Research Laboratory has developed a human performance modeling tool, the Improved Performance Research Integration Tool (IMPRINT), for investigation of the impact on a Soldier's performance when the Soldier subjected to environmental stressors such as heat and cold. IMPRINT has the capability to create user-defined stressors to study the stressors' effect on human performance and therefore system performance. This case study used data from literature to create a user-defined stressor in IMPRINT to predict the effect of riding in a moving vehicle on task time and performance. This capability can provide useful information to system designers.

Towards the Shape of Mental Workload

Mental workload is a measure of how much mental effort a person devotes to one or more tasks. In two experiments, we investigated the effect of multiple identical tasks on human performance in terms of both accuracy and response time for a visuo-spatial task set and an auditory task set. The findings showed that participants performed linearly worse on some measures of performance when the number of tasks increased, while other measures showed two distinctive variations on this linear decrease in performance. We discuss these results in terms of their effect on the traditional linear representation of workload in IMPRINT (IMproved Performance Research INtegration Tool, Archer & Adkins, 1999), a task-based human performance modeling system.

Impacting System Design with Human Performance Modeling and Experiment: Another Success Story

In this study, task-network models using the Improved Performance Research Integration Tool were built to predict the mental workload and performance of the crew of a conceptual mounted combat vehicle. The modeling project analyzed which crewmember, if any, should be allocated the functions of controlling robotic assets in addition to standard mission functions similar to present tank crew missions. In order to further investigate issues derived from the modeling project and to verify its analytical results, a simulation experiment was conducted to examine the mental workload and performance of the combined position of robotics operator and gunner. The results of this experiment were consistent with the predictions generated by the modeling analysis. The modeling and experimental data have successfully contributed to modifications in the design concept for the modeled platform as well as its crewmember function allocations.

Human Performance Modeling and Decision Analysis in a Concept Crew Workstation

As part of the effort to evaluate the performance of soldiers operating in the Vetronics Testbed Technology (VTT), a human performance model of operator processes and tasks was developed. The purpose of this effort was to gather insight into operator workload constraints, points of operator overload due to task demands, and key decision-making points and strategies employed by the operators. Operator performance was modeled when the operators were confronted with the task of controlling multiple experimental unmanned ground vehicles (XUV) in addition to performing standard Command and Control (C2) operations. The model was built using the Improved Performance Research Integration Tool (IMPRINT). Development began with a task decomposition of the current VTT system and operator control unit (OCU) and was decomposed to the button-push level of operator interaction with the unit. Next, several operational scenarios were developed to drive the simulated operator actions in the OCU and obtain measures of performance. Finally, a decision-making architecture was implemented in the model to examine points where intelligent agents and system automation could potentially aid in reducing operator cognitive demands.

Tools for Modeling Human Performance in Systems Through Green-Colored Glasses: An Army Perspective

Providing an authoritative representation of human behavior has recently been recognized as essential for modeling and simulation of system performance. Even earlier; however, the Human Research & Engineering Directorate of the U.S. Army Research Laboratory was working to do just that in a tool called IMPRINT, (the Improved Performance Research Integration Tool). IMPRINT enables a quantitative portrayal of human performance. Task network modeling is implemented with embedded data and sound psychological methods in an easy-to-use interface, all of which has been subjected to intensive verification and validation. IMPRINT is being used today to influence system design and acquisition decisions. Several examples of applications to Army systems are discussed, with implications ranging from detailed equipment design to force-on-force effectiveness. Of course, “more research needs to be done,” but in the meantime, human performance modeling is ready for use.