Predicting Operator Performance Research Articles

Exploring Fine-Grained In-Memory Database Performance for Modern CPUs

Modern CPUs keep integrating more cores and large size cache, which is beneficial for in-memory databases to improve parallel processing power and cache locality. While state-of-the-art CPUs have diverse architectures and roadmaps such as large core count and large cache size (AMD x86), moderate core count and cache size (intel x86), large core count and moderate cache size (ARM), exploring in-memory databases performance characteristics for different CPU architectures is important for in-memory database designs and optimizations. In this paper, we develop a fine-grained in-memory database benchmark to evaluate the performance of each operator on different CPUs to explore how CPU hardware architectures influence performance. Different from well known conclusions that more cores and larger cache size can achieve higher performance, we find out that the micro cache architectures play an important role opposite to core count and cache size, the shared monolithic L3 cache with moderate size beats large disaggregated L3 cache. The experiments also show that predicting operator performance on different CPUs is difficult according to diverse CPU architectures and micro cache architectures, and different implementations of each operator are not always high or low with interleaved strong and weak performance regions influenced by CPU hardware architectures. Intel x86 CPUs represent cache-centric processor design, while AMD x86 and ARM CPUs represent computing-centric processor design, the OLAP benchmark experiments of SSB discover that OmniSciDB and OLAP Accelerator with vector-wise processing model performs well on intel x86 CPUs compared to AMD x86 CPUs and the JIT compliant based Hyper prefers to AMD x86 CPUs rather than intel x86 CPUs. The CPU roadmaps of increasing cores or improving cache locality should be considered for in-memory database algorithm design and platform selection.

Gaming experience predicts UAS operator performance and workload in simulated search and rescue missions

Unmanned aircraft systems (UAS) operator training and selection procedures are still being refined to effectively address challenges related to performance, workload, and stress in UAS operation. Research suggests that experience with commercial videogames may test skills relevant to modern UAS operation. This study investigated the ability of videogame experience to predict operator performance, workload, and stress. Forty-nine participants performed 9 trials of a simulated search and rescue mission. It was expected that participants who more frequently played videogames would report lower levels of distress and workload, higher task engagement, and better overall performance. Results showed that gaming experience was negatively correlated with subjective workload and positively correlated with multiple measures of performance. Furthermore, nearly all observed gender-related differences were not present when gaming experience was controlled for. These results have implications for the role of gaming experience in remotely operated systems operator recruitment, selection, and training. Practitioner summary: This study examined how gaming experience influences UAS operator success in simulated search and rescue missions. Participants reported on their experience playing videogames before completing multiple experimental trials on a desktop computer. Results indicated that experience playing videogames significantly impacted performance, workload, and stress.

Wearable technology-based metrics for predicting operator performance during cardiac catheterisation

IntroductionUnobtrusive metrics that can auto-assess performance during clinical procedures are of value. Three approaches to deriving wearable technology-based metrics are explored: (1) eye tracking, (2) psychophysiological measurements [e.g. electrodermal activity (EDA)] and (3) arm and hand movement via accelerometry. We also measure attentional capacity by tasking the operator with an additional task to track an unrelated object during the procedure.MethodsTwo aspects of performance are measured: (1) using eye gaze and psychophysiology metrics and (2) measuring attentional capacity via an additional unrelated task (to monitor a visual stimulus/playing cards). The aim was to identify metrics that can be used to automatically discriminate between levels of performance or at least between novices and experts. The study was conducted using two groups: (1) novice operators and (2) expert operators. Both groups made two attempts at a coronary angiography procedure using a full-physics virtual reality simulator. Participants wore eye tracking glasses and an E4 wearable wristband. Areas of interest were defined to track visual attention on display screens, including: (1) X-ray, (2) vital signs, (3) instruments and (4) the stimulus screen (for measuring attentional capacity).ResultsExperts provided greater dwell time (63% vs 42%, p = 0.03) and fixations (50% vs 34%, p = 0.04) on display screens. They also provided greater dwell time (11% vs 5%, p = 0.006) and fixations (9% vs 4%, p = 0.007) when selecting instruments. The experts’ performance for tracking the unrelated object during the visual stimulus task negatively correlated with total errors (r = − 0.95, p = 0.0009). Experts also had a higher standard deviation of EDA (2.52 µS vs 0.89 µS, p = 0.04).ConclusionsEye tracking metrics may help discriminate between a novice and expert operator, by showing that experts maintain greater visual attention on the display screens. In addition, the visual stimulus study shows that an unrelated task can measure attentional capacity.Trial registration This work is registered through clinicaltrials.gov, a service of the U.S. National Health Institute, and is identified by the trial reference: NCT02928796.

Physiological Measures for Human Performance Analysis in Human-Robot Teamwork: Case of Tele-Exploration

Continuous monitoring of mental workload and situation awareness in operational environments is useful for understanding and prediction of human performance. Such information can be used to develop real-time adaptive systems to enhance human performance. In this paper, we investigate the use of workload- and attention-related physiological measures to predict operator performance and situation awareness in the context of tele-exploration with a small team of robots. A user study is conducted based on a simulated scenario involving visual scanning and manual control tasks with varying levels of task-load. Brain activity and eye movements of the participants are monitored across the experimental tasks using electroencephalogram and eye tracker sensors. The performances of the subjects are evaluated in terms of target detection and situation awareness (primary metrics) as well as reaction time and false detection (secondary metrics). Moreover, individual differences in two specific visual skills, visual search (VS) and multi-object tracking (MOT) are considered as between-subject factors in the experimental design. The main effects of task type and individual differences reveal that VS and MOT skill have significant effects on target detection and situation awareness, respectively. The correlations of physiological measures with the task performance and situation awareness are analyzed. The results suggest that brain-based features (mental workload and distraction) which represent the covert aspect of attention are better suited to predict the secondary performance metrics. On the other hand, glance-based features which represent the overt aspect of attention are shown to be the best predictors of the primary performance metrics.

A theoretical framework for evaluating mental workload resources in human systems design for manufacturing operations

As the nature of manufacturing work is changing, requiring more cognitive demands, there is a need to develop system models for measuring and predicting human performance in repetitive task operations. This paper presents a theoretical framework, which provides a systematic approach for measuring mental workload using a combination of analytical and empirical techniques: human performance modeling with a computer simulation and mathematical modeling, along with physiological, subjective and performance measures. For this study, the Air Force MATB, which is a re-development of the NASA simulation tool, was used to model multitasking in a controlled environment to validate the theoretical framework. The independent variable of task complexity was measured, in the modeling of resource demands for a cleaning-inspection process and a final inspection process, using three dependent variables (subjective, physiological and performance measures) with a total of four responses (NASA-TLX, Workload Profile, fixation duration and human error probability). The results indicate no significant difference among the response variables for each task complexity level, indicating the model accurately represents the operator's workload. Additional analysis shows accurate predication from the model in analyzing workload peaks. Relevance to industryThis theoretical framework is designed to evaluate operator mental workload utilization in the manufacturing domain. This is important because manufacturing work is changing by requiring more multitasking roles, which increases cognitive demand. Therefore, the industry needs models that can predict operator performance and mental workload for improved productivity.

Determination of the Proper Rest Time for a Cyclic Mental Task Using ACT-R Architecture.

Objective Analysis of the effect of mental fatigue on a cognitive task and determination of the right start time for rest breaks in work environments. Background Mental fatigue has been recognized as one of the most important factors influencing individual performance. Subjective and physiological measures are popular methods for analyzing fatigue, but they are restricted to physical experiments. Computational cognitive models are useful for predicting operator performance and can be used for analyzing fatigue in the design phase, particularly in industrial operations and inspections where cognitive tasks are frequent and the effects of mental fatigue are crucial. Method A cyclic mental task is modeled by the ACT-R architecture, and the effect of mental fatigue on response time and error rate is studied. The task includes visual inspections in a production line or control workstation where an operator has to check products' conformity to specifications. Initially, simulated and experimental results are compared using correlation coefficients and paired t test statistics. After validation of the model, the effects are studied by human and simulated results, which are obtained by running 50-minute tests. Results It is revealed that during the last 20 minutes of the tests, the response time increased by 20%, and during the last 12.5 minutes, the error rate increased by 7% on average. Conclusion The proper start time for the rest period can be identified by setting a limit on the error rate or response time. Application The proposed model can be applied early in production planning to decrease the negative effects of mental fatigue by predicting the operator performance. It can also be used for determining the rest breaks in the design phase without an operator in the loop.

Implementing Lumberjacks and Black Swans Into Model-Based Tools to Support Human-Automation Interaction.

The objectives were to (a) implement theoretical perspectives regarding human-automation interaction (HAI) into model-based tools to assist designers in developing systems that support effective performance and (b) conduct validations to assess the ability of the models to predict operator performance. Two key concepts in HAI, the lumberjack analogy and black swan events, have been studied extensively. The lumberjack analogy describes the effects of imperfect automation on operator performance. In routine operations, an increased degree of automation supports performance, but in failure conditions, increased automation results in more significantly impaired performance. Black swans are the rare and unexpected failures of imperfect automation. The lumberjack analogy and black swan concepts have been implemented into three model-based tools that predict operator performance in different systems. These tools include a flight management system, a remotely controlled robotic arm, and an environmental process control system. Each modeling effort included a corresponding validation. In one validation, the software tool was used to compare three flight management system designs, which were ranked in the same order as predicted by subject matter experts. The second validation compared model-predicted operator complacency with empirical performance in the same conditions. The third validation compared model-predicted and empirically determined time to detect and repair faults in four automation conditions. The three model-based tools offer useful ways to predict operator performance in complex systems. The three tools offer ways to predict the effects of different automation designs on operator performance.

Measuring Situation Awareness of Student Air Traffic Controllers with Online Probe Queries: Are We Asking the Right Questions?

ABSTRACTSituation awareness (SA) is a major factor determining operator performance in the current and future airspace systems, so it is not surprising that a considerable body of research on measuring operator SA has been amassed. A commonly used SA measurement method is a probe technique in which the operator is queried about the current state of his/her task environment, either offline (while a computer simulation is paused) or online (while a computer simulation is running.) Although sufficient evidence for the reliability and validity of both techniques is available, one factor that has not been systematically investigated is the efficacy of the questions themselves. We evaluated the types of online probe questions used for predicting the performance of student air traffic controllers in a simulated air traffic management task. These data were obtained from 84 students who completed a 16-week radar internship course in which their situation awareness and performance were assessed at the midpoint and end of the course. We subjected probe question latencies to a principle component analysis (PCA) to explore the variation between questions and grouped them based on the resulting components. The SA question components that emerged from PCA were then used to model the relations between SA factors and performance. The resulting model provides a framework for understanding factors involved in SA, a way to evaluate the validity of individual probe questions as a measure of SA, and a method for determining the efficacy of SA questions in predicting operator performance.

Development, Testing, and Validation of a Model-Based Tool to Predict Operator Responses in Unexpected Workload Transitions

One human factors challenge is predicting operator performance in novel situations. Approaches such as drawing on relevant previous experience, and developing computational models to predict operator performance in complex situations, offer potential methods to address this challenge. A few concerns with modeling operator performance are that models need to realistic, and they need to be tested empirically and validated. In addition, many existing human performance modeling tools are complex and require that an analyst gain significant experience to be able to develop models for meaningful data collection. This paper describes an effort to address these challenges by developing an easy to use model-based tool, using models that were developed from a review of existing human performance literature and targeted experimental studies, and performing an empirical validation of key model predictions.

Using Empirical Research and Human Performance Modeling to Predict Astronaut Performance in Long-Duration Space Missions

The invited symposium session consists of four papers describing the S-PRINT project, sponsored by NASA’s Human Research Program (HRP). The efforts undertaken as part of this project were to develop tools and empirically-based guidelines that support human performance researchers, mission planners, automation designers, and astronauts in long-duration missions. The Space Performance Research Integration Tool (S- PRINT) project addresses aspects of the work environment that are characterized by workload transitions that might occur during long-duration missions. Workload transitions are a potential worst-case scenario, in which under-loaded astronauts experience automation failures that require immediate diagnosis and intervention. The research has focused on the underload situation and on the effects that fatigue has on performance, human performance during overload / multitasking situations, and on automation design factors that affect operator performance. In this symposium, the investigators will present four papers (identified below) that describe different aspects of the research completed to develop the S-PRINT model. The session will show an integrated approach of using empirical research and human performance modeling to predict operator performance in operations that are in the design phase. In part, this symposium will highlight the way that the products from this research can be used to help the above-listed users (a) anticipate and avoid potential problems related to unexpected workload transitions by identifying the expected effects of operator fatigue, automation system design, and task factors on overload performance, and to (b) assure systems can be designed in such a way as to optimize performance in space exploration missions, particularly those that are experienced during longer term missions (6+ months).

Using Modeling and Simulation to Predict Operator Performance and Automation-Induced Complacency With Robotic Automation: A Case Study and Empirical Validation.

The aim of this study was to develop and validate a computational model of the automation complacency effect, as operators work on a robotic arm task, supported by three different degrees of automation. Some computational models of complacency in human-automation interaction exist, but those are formed and validated within the context of fairly simplified monitoring failures. This research extends model validation to a much more complex task, so that system designers can establish, without need for human-in-the-loop (HITL) experimentation, merits and shortcomings of different automation degrees. We developed a realistic simulation of a space-based robotic arm task that could be carried out with three different levels of trajectory visualization and execution automation support. Using this simulation, we performed HITL testing. Complacency was induced via several trials of correctly performing automation and then was assessed on trials when automation failed. Following a cognitive task analysis of the robotic arm operation, we developed a multicomponent model of the robotic operator and his or her reliance on automation, based in part on visual scanning. The comparison of model predictions with empirical results revealed that the model accurately predicted routine performance and predicted the responses to these failures after complacency developed. However, the scanning models do not account for the entire attention allocation effects of complacency. Complacency modeling can provide a useful tool for predicting the effects of different types of imperfect automation. The results from this research suggest that focus should be given to supporting situation awareness in automation development.

Investigation on applicability of information theory to prediction of operator performance in diagnosis tasks at nuclear power plants

This paper investigates the applicability of information theory to predicting operator performance in diagnosis tasks at nuclear power plants (NPPs). Some general descriptions about diagnosis tasks in a main control room (MCR) are provided and information theory conventionally used in the studies of human information processing is briefly reviewed. Then, as an alternative to classical information theory, this paper proposes a method to quantify the cognitive information flow of diagnosis tasks, integrating a stage model (a qualitative approach) with information theory (a quantitative approach). An information flow model is developed based on operating procedures and quantified using Conant's model. Finally, three experiments were conducted to evaluate the effectiveness of the proposed approach to predicting human performance, especially average diagnosis time. The results of two experiments show that subjects' diagnosis time has a proportional relationship to the quantified information flow. The other investigated operator performance variation in terms of accuracy under system-paced information presentation. As a result of the third experiment, subjects showed better performances in a specific range of information rate, compared with in other ranges.

The effect of monocular target blur on simulated telerobotic manipulation

A simulation involving three types of telerobotic tasks that require information about the spatial position of objects is reported. This is similar to the results of psychophysical experiments examining the effect of blur on stereoacuity. It is suggested that other psychophysical experimental results could be used to predict operator performance for other telerobotic tasks. It is demonstrated that refractive errors in the helmet-mounted stereo display, system can affect performance in the three types of telerobotic tasks. The results of two sets of experiments indicate that monocular target blur of two diopters or more degrades stereo display performance to the level of monocular displays. This indicates that moderate levels of visual degradation that affect the operator's stereoacuity may eliminate the performance advantage of stereo displays.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Task Taxonomy: Two Ignored Issues

This paper focuses on two major issues concerning task taxonomies which to date have been ignored. The first issue involves the development of a set of criteria for the evaluation of existing and future task taxonomies. A set of criteria is proposed and several major existing taxonomies are evaluated. The second issue concerns the relation of empirical data, provided by field and laboratory research, to the development of the task taxonomy. Three approaches are cited which seem to encompass the relation between the taxonomy and the empirical data used in the development of task taxonomies. It is suggested that one of these approaches appears to be better suited for the development of a taxonomy to predict operator performance in any system.

A Dynamic Visual Recognition Test for Paced Inspection Tasks

Abstract In many manufacturing organizations, employees are required visually to inspect parts that are moving along a conveyor. Applicants for these inspection tasks are often selected as the result of their performance on static visual acuity tests and rarely on the basis of the recently proposed dynamic acuity tests. However, conventional static and dynamic acuity tests have often proven unsuccessful in predicting operator performance on conveyor-paced inspection tasks because of the physical, psychological, and physiological differences between visual tests and the operating tasks. This research pertains to the development of a two-test battery used to minimize the differences between the selection tests and the paced-inspection tasks.

INDIVIDUAL AND SITUATIONAL VARIABLES AFFECTING HUMAN PERFORMANCE.

Present methods of measuring and predicting operator performance are criticized because of their failure to consider individual and situational variables which mediate that performance.

Pattern Recognition and Display Characteristics

This paper reports experimental results of human operator performance in a visual recognition task. The work began with a method of generating families of complex patterns to simulate certain characteristics of visual sensor displays, such as radar and infrared returns. The experimental effort was directed toward establishing criteria for predicting human operator performance in a map matching task. The operators' task was to recognize which of four patterns presented simultaneously with a reference pattern belonged to the reference pattern family. The measure of performance was the time in seconds taken by the operator to make a selection. Response times were more rapid when the reference pattern was less complex than the comparison than when the reference pattern was the more complex. Analysis of the display characteristics led to the selection of four physical measures to be used in predicting operator performance. These measures?pattern length, pattern density, and two measures of pattern complexity?correlated highly with response time, were not highly intercorrelated, and were applicable to natural sensor returns. The four measures were found to account for a high degree of the total variance. Regression equations were derived which predict performance from known values of the four measures.