Human Interface System Research Articles

Perception-Based Data Reduction for Haptic Force-Feedback Signals Using Velocity-Adaptive Deadbands

In telepresence and teleaction (TPTA) systems, the transmission of haptic signals puts high demands on the applied signal processing and communication procedures. When running a TPTA session across a packet-based communication network (e.g., the Internet), minimizing the end-to-end delay results in packet rates of up to the applied sampling rate of the local control loops at the human system interface and the teleoperator. The perceptual deadband data reduction approach for haptic signals successfully addresses the challenge of high packet rates in networked TPTA systems and satisfies the strict delay constraints. In this paper, we extend the underlying perceptual model of the deadband approach by incorporating psychophysical findings on human force-feedback discrimination during operators' relative hand movements. By applying velocity-dependent perception thresholds to the deadband approach, we observe further improvement in efficiency and performance due to improved adaption to human haptic perception thresholds. The psychophysical experiments conducted reveal improved data reduction performance of our proposed haptic perceptual coding scheme without impairing the user experience. Our results show a high data reduction ability of up to 96% without affecting system transparency or the operator's task performance.

Influence of Varied Human Movement Control on Task Performance and Feeling of Telepresence

One of the main objectives in telerobotics is the development of a telemanipulation system that allows a high task performance to be achieved by simultaneously providing a high degree of telepresence. Specific mechatronic design guidelines and appropriate control algorithms as well as augmented visual, auditory, and haptic feedback systems are typical approaches adopted in this context. This work aims at formulating new design guidelines by incorporating human factors in the development process and analyzing the effects of varied human movement control on task performance and on the feeling of telepresence. While it is well known that humans are able to coordinate and integrate multiple degrees of freedom (DOF), the focus of this work is on how humans utilize rotational degrees of freedom provided by a human-system interface and if and how varied human movement control affects task performance and the feeling of telepresence. For this analysis, a telemanipulation experiment with varying degrees of freedom has been conducted. The results indicate that providing the full range of movement, even though this range is not necessary to accomplish a task, has a beneficial effect on the feeling of telepresence and task performance in terms of measured interaction forces. Further, increasing visual depth cues provided to the human operator also had a positive effect.

The research on extracting the information of human errors in the main control room of nuclear power plants by using Performance Evaluation Matrix

Digitalized nuclear instruments and control systems have been the main stream design for the main control room (MCR) of advanced nuclear power plants (NPPs) nowadays. Digitalization of human–system interface (HSI) could improve human performance and, on the other hand, could reduce operators’ situation awareness as well. It might cause that humans make wrong decision during an emergency unintentionally. Besides, digital HSI relies on computers to integrate system information automatically instead of human operation. It has changed the operator’s role from mainly relating operational activity to mainly relating monitoring. However, if the operators omit or misjudge the information on the video display units and wide display panel, the error of omission and error of commission may occur. This study applies Content Category Analysis (CCA) and Performance Evaluation Matrix (PEM) methods to explore the potential problems in the MCR of advanced NPPs which are caused by human errors. The results show that the potential problems that would probably contribute to the human performance of MCR in the advanced NPPs are multiple accidents, pressure level, number of operators, and other factors such as working environmental. The conclusions of this research may be considered to avoid and prevent human errors in the human factor related researches especially in the nuclear safety field.

Guidance for Human-System Interfaces to Automatic Systems

Automation is ubiquitous in modern complex systems, and commercial nuclear- power plants are no exception. Automation is applied to a wide range of functions, including monitoring and detection, situation assessment, response planning, and response implementation. Automation has become a “team player” supporting personnel in nearly all aspects of system operation. In light of its increasing use and importance in new- and future-plants, guidance is needed to conduct safety reviews of the operator's interface with automation. The objective of this research was to develop such guidance. We first characterized the important HFE aspects of automation, including six dimensions: Levels, functions, processes, modes, flexibility, and reliability. Next, we reviewed literature on the effects of all of these aspects of automation on human performance, and on the design of human-system interfaces (HSIs). Then, we used this technical basis established from the literature to identify general principles for human-automation interaction and to develop review guidelines. The guidelines consist of the following seven topics: Automation displays, interaction and control, automation modes, automation levels, adaptive automation, error tolerance and failure management, and HSI integration.

Mapping Complexity Sources in Nuclear Power Plant Domains

Understanding the sources of complexity in advanced Nuclear Power Plant (NPP) control rooms and their effects on human reliability is critical for ensuring safe performance of both operators and the entire system. New generation control rooms will rely more heavily on automation and computerized Human-System Interfaces (HSI). Without proper management, information representation and required operator-system interaction could challenge operator information processing capabilities. This paper provides an initial step in assessing the sources of complexity in the NPP control rooms and introduces a systems-theoretic descriptive model of these sources of complexity leveraging network theory.

Optimizing human–system interface automation design based on a skill-rule-knowledge framework

This study considers the technological change that has occurred in complex systems within the past 30 years. The role of human operators in controlling and interacting with complex systems following the technological change was also investigated. Modernization of instrumentation and control systems and components leads to a new issue of human–automation interaction, in which human operational performance must be considered in automated systems. The human–automation interaction can differ in its types and levels. A system design issue is usually realized: given these technical capabilities, which system functions should be automated and to what extent? A good automation design can be achieved by making an appropriate human–automation function allocation. To our knowledge, only a few studies have been published on how to achieve appropriate automation design with a systematic procedure. Further, there is a surprising lack of information on examining and validating the influences of levels of automation (LOAs) on instrumentation and control systems in the advanced control room (ACR). The study we present in this paper proposed a systematic framework to help in making an appropriate decision towards types of automation (TOA) and LOAs based on a ‘Skill-Rule-Knowledge’ (SRK) model. From the evaluating results, it was shown that the use of either automatic mode or semiautomatic mode is insufficient to prevent human errors. For preventing the occurrences of human errors and ensuring the safety in ACR, the proposed framework can be valuable for making decisions in human–automation allocation.

日本人とタイ人の顔の人種・性別認知に関する検討

We investigated effect of face perception on race and gender classification by using facial images of racially close groups such as Thai and Japanese. Participants were Japanese and Southeast Asian college students residing in Japan. The purpose of this work is to study human perception in an attempt to establish more efficient human interface systems. In a psychological experiment, the participants from Southeast Asia acquired high recognition scores of approximately 70%, which is higher than those of the Japanese participants. Since the perception rates of the Southeast Asian participants varied by the length of their stay in Japan, it can be suggested that race perception is a learnable ability. For race perception, participants focused mostly on eyes and eyebrows. These results suggest that eyes are a significant factor in race perception. For Gender perception, we found that participants focused mostly on eyes, eyebrows, and mouths.

Review of Kansei Research in Japan

This article gives a historical review of Kansei-based media technologies in Japan. Kansei is a Japanese word, the meaning of which covers sensibility, sentiment, emotion, and feeling. Kansei research started in the field of music, because music is the most acceptable of the arts to computer science. In the 1990s, the applications of Kansei machine vision became widespread in many industrial fields, including electronic production, automobile manufacture, steel-making, the chemical industry, the food industry, and office appliances, among others. Kansei technologies are also applied to human interface systems, including the field of brain science, for human communication.

Mapping and visualization interface for information in disaster

Many rescue robots are developed for search and rescue missions in the disaster area. The rescue robots gather the information in the disaster area using sensors. Gathered information is managed by GIS (Geographic Information System) as an environment map. There are two major problems to gather the information from disaster area by using mobile robots. One is the unstable communication between operator station and disaster area, and the other is the inaccurate estimation of robots' position. In this study, we developed two new human interface systems to gather necessary information preferentially using a virtual 3D environment on demand and adjust the position errors of a robot intuitively. Furthermore simulation is carried out to validate the usability of the proposed interface by comparing with an existing interface.

Human-system interface based on speech recognition: application to a virtual nuclear power plant control desk

This work reports the development of a human-system interface based on automatic speech recognition, and its integration to a virtual nuclear power plant control desk under development. The later is aimed to reproduce a nuclear power plant's control desk using virtual reality technology, for operator training and ergonomic evaluation purposes. Initially, users were able to interact with this virtual control desk only through computer keyboard and mouse. Then, an automatic speech recognition system was developed to serve as a new user interface, so they can navigate and operate this virtual control desk in front of a computer monitor or a projection screen, free from keyboard and mouse, through spoken commands. The automatic speech recognition interface developed is based on a well-known signal processing technique named cepstral analysis, and on artificial neural networks. The speech recognition interface is described, along with its integration with the virtual control desk, and results are presented.

Evaluation of operators’ mental workload of human–system interface automation in the advanced nuclear power plants

It has been expected that the automation of certain tasks in a control room would help decrease operators’ mental workload, enhance situation awareness, and improve the whole system performance. However, there have been too many automation-induced system failures that would warrant a fresh look on the influences of automation. Automation problems include the reduction in the operator's system awareness, an increase in monitoring workload, and the degradation in manual skills. This study evaluates operators’ mental workload and system performance during a human–system interface (HSI) automation in an advanced nuclear power plant (NPP). The reactor shutdown task and alarm reset task simulations were conducted in this study to evaluate operators’ mental workload and performance. The results of this study indicated that for ensuring safe operating in NPPs, the design of automation needs to be carefully implemented. Task characteristics and degrees of automation should be carefully evaluated while designing HSIs. The reactor shutdown tasks studied in this paper suggest that a high level of automation design for the long period and low workload would be sufficient. On the other hand, the degree of automation of alarm reset task does not show a significant difference to the operator's mental workload. In conclusion, the human–system interface automation in advanced NPPs is suggested to be more flexible and needs to be continually improved.

Automation design in advanced control rooms of the modernized nuclear power plants

This study evaluated the effects of levels of automation (LOAs) decisions in advanced control rooms of the modernized nuclear power plants. Following advancements in design of digitalized human–system interfaces (HSIs), the roles of human operators have changed significantly. Negative performance and safety consequences may occur as a result of these changes. These problems are viewed as the out-of-the-loop (OOTL) performance problems. This study conducted an experiment to compare the effects of different LOAs under different operating procedures on operating performance. Experimental results indicated that blended decision-making (level 6 LOA) generates the lowest mental workload. Furthermore, the pattern of SA observed in this study is found better SA at intermediate LOA and poorer SA at low level of automation and full automation. Subjective rating results suggest that LOAs distribute the roles of option generation, and selection between human and/or computer servers which significantly impacts automated system performance. This study provides a direction for the HSI designers in nuclear power plants (NPPs). Additionally, based on results obtained by this study, the user interfaces of PCTRAN system and the alarm reset system should be improved to ensure safe operation of NPPs.

A Hybrid Motion Classification Approach for EMG-Based Human–Robot Interfaces Using Bayesian and Neural Networks

In a human-robot interface, the prediction of motion, which is based on context information of a task, has the potential to improve the robustness and reliability of motion classification to control prosthetic devices or human-assisting manipulators. This paper proposes a task model using a Bayesian network (BN) for motion prediction. Given information of the previous motion, this task model is able to predict occurrence probabilities of the motions concerned in the task. Furthermore, a hybrid motion classification framework has been developed based on the BN motion prediction. Besides the motion prediction, electromyogram (EMG) signals are simultaneously classified by a probabilistic neural network (NN). Then, the motion occurrence probabilities are combined with the NN classifier's outputs to generate motion commands for control. With the proposed motion classification framework, it is expected that classification performance can be enhanced so that motion commands can be more robust and reliable. Experiments have been conducted with four subjects to demonstrate the feasibility of the proposed methods. In these experiments, forearm motions are classified with EMG signals considering a cooking task. Finally, robot manipulation experiments were carried out to verify the proposed human interface system with a task of taking meal. The experimental results indicate that the proposed methods improved the robustness and stability of motion classification.

Robotics in Surgery

Leonardo da Vinci, the master artist, is credited for one of the early concepts of robots in the late 1400’s. His genius propelled the idea of a mechanical device with the capability of automation, repetitive motion and wristed joints. While his idea of a mechanical knight was primitive, the concept led to a landslide of innovations. This concept was subsequently introduced into manufacturing in the early 1800’s when Joseph Jacquard, in France, created a loom to weave repetitive patterns into cloth. This was the first use of programmable machinery in a field of industry. Nevertheless, the application of this robotic technology in surgery occurred only in 1985, when the first truly robotic flexible arm (PUMA 560) was used to orientate a needle for a guided brain biopsy. Soon after, the robot was utilized in different surgical procedures, including transurethral resection of the prostate and hip replacement, changing forever the role of robotics in medicine. The contemporary generation of surgical robots consists of a ‘master-slave’ system. The basic principle involves control of robotic arms by a surgeon sitting at a console, which provides 3D 10x magnified vision, wristed instrumentation, tremor filtration, and motion scaling. The system produces an immersive telerobotic environment ideally suited for surgical precision and reconstructive applications. The introduction of this technology into the modern day operating theatre has revolutionized the very essence of how surgical procedures are approached today. Robotic assisted surgery is now a standard concept in the United States with over 700 robotic surgical systems and is beginning to grow worldwide. While initially, the daVinci robot was designed for cardiac surgery, it was the prostate that led to its ascent. The first robotic prostatectomy was performed in 1999, by Binder and colleagues, and at present, ten years after, is the most common robotic surgery worldwide, with more than 300,000 procedures performed. The device is now growing into other surgical specialties such as Gynecology and GI surgery. Since the approval of the daVinci surgical system for gynecologic surgeries by the FDA in 2005, this technology has been applied to a number of different conditions, including robotic hysterectomy, myomectomy, tubal reanastomoses, pelvic and paraaortic lymph node dissection, and sacrocolpopexy. In the field of general surgery, procedures for which the daVinci surgical system may be of particular value include: esophageal myotomy, paraesophageal hernia repair, gastric bypass, transhiatal esophagectomy, transthoracic esophageal surgery and others. Dr Patel’s personal experience has been in robotic radical prostatectomy over the last seven years. 3000 cases later, he has learned to appreciate the continuing evolution of the technology and the unending bounds of its capability. The ”robot” has evolved into and now is becoming the ”tool” that Leonardo first dreamed in the 1400’s. To realize the future of robotics in medicine fully we will need multidisciplinary collaboration. Physicians and engineers will work collectively to understand the effusive potential of the human-system interface. The potential for enhancing patient care is unrestricted. This issue of The Scandinavian Journal of Surgery is dedicated to robotics in surgery.The respected experts in varying fields of surgery will share their expertise and knowledge in several well written review articles. The possibilities, but also the weaknesses, of the contemporary robotics systems will be elaborated.

A computational model for evaluating the effects of attention, memory, and mental models on situation assessment of nuclear power plant operators

Operators in nuclear power plants have to acquire information from human system interfaces (HSIs) and the environment in order to create, update, and confirm their understanding of a plant state, as failures of situation assessment may cause wrong decisions for process control and finally errors of commission in nuclear power plants. A few computational models that can be used to predict and quantify the situation awareness of operators have been suggested. However, these models do not sufficiently consider human characteristics for nuclear power plant operators. In this paper, we propose a computational model for situation assessment of nuclear power plant operators using a Bayesian network. This model incorporates human factors significantly affecting operators’ situation assessment, such as attention, working memory decay, and mental model. As this proposed model provides quantitative results of situation assessment and diagnostic performance, we expect that this model can be used in the design and evaluation of human system interfaces as well as the prediction of situation awareness errors in the human reliability analysis.

LESSONS LEARNED FROM HALDEN PROJECT RESEARCH ON HUMAN SYSTEM INTERFACES

Innovative Human System Interfaces (HSIs) has been a major topic of research of the international Halden Reactor Project (HRP) for many years. Different design concepts have been addressed and prototypes have been implemented and evaluated in the experimental control room facility of HRP. Many of the concepts go far beyond traditional P&ID type displays, and utilize advanced computer graphics and animations. The paper briefly describes some of the concepts, their advantages and disadvantages experienced through evaluations and feedback from users.

IDENTIFICATION AND EVALUATION OF HUMAN FACTORS ISSUES ASSOCIATED WITH EMERGING NUCLEAR PLANT TECHNOLOGY

This study has identified human performance research issues associated with the implementation of new technology in nuclear power plants (NPPs). To identify the research issues, current industry developments and trends were evaluated in the areas of reactor technology, instrumentation and control technology, human-system integration technology, and human factors engineering (HFE) methods and tools. The issues were prioritized into four categories based on evaluations provided by 14 independent subject matter experts representing vendors, utilities, research organizations and regulators. Twenty issues were categorized into the top priority category. The study also identifies the priority of each issue and the rationale for those in the top priority category. The top priority issues were then organized into research program areas of: New Concepts of Operation using Multi-agent Teams, Human-system Interface Design, Complexity Issues in Advanced Systems, Operating Experience of New and Modernized Plants, and HFE Methods and Tools. The results can serve as input to the development of a long-term strategy and plan for addressing human performance in these areas to support the safe operation of new NPPs.

Development of Human Factors Validation System for the Advanced Control Room of APR1400

A human factors validation system for the main control room (MCR) of Advanced Power Reactor 1400MWe (APR1400) has been developed as it adopts digitalized human-system interfaces (HSIs). The integrated validation system is composed of process/plant models, HSIs, and the human performance evaluation support system (HUPESS). A real-time thermal-hydraulic code, RELAP5 R/T, was used and modified to simulate the dynamic characteristics of the APR1400, and simulation software, 3KeyMaster, was used to model the balance of plant systems. The HSIs developed in this study include all facilities in the APR1400 MCR, such as large display panels, 3 identified operator workstations, and a safety console. In addition, the remote shutdown workstation has been developed. The display systems in the HSIs have been developed using ProcSee, which is a software tool for developing and displaying dynamic graphical user interfaces. This paper describes the configurations of HSIs including display systems, the dynamic models of the APR1400 simulator, the instructor station, and the HUPESS. This paper also presents the results of plant simulation performance tests at transient compared with the results of RELAP5/MOD3.3 calculations. The human factors validation system for the advanced control room of APR1400 provides high degrees of physical, functional, and dynamic fidelities, and can be used in the validation process of the APR1400 HSI design.

Development of Human Factors Validation System for the Advanced Control Room of APR1400

A human factors validation system for the main control room (MCR) of Advanced Power Reactor 1400MWe (APR1400) has been developed as it adopts digitalized human-system interfaces (HSIs). The integrated validation system is composed of process/plant models, HSIs, and the human performance evaluation support system (HUPESS). A real-time thermal-hydraulic code, RELAP5 R/T, was used and modified to simulate the dynamic characteristics of the APR1400, and simulation software, 3KeyMaster, was used to model the balance of plant systems. The HSIs developed in this study include all facilities in the APR1400 MCR, such as large display panels, 3 identified operator workstations, and a safety console. In addition, the remote shutdown workstation has been developed. The display systems in the HSIs have been developed using ProcSee, which is a software tool for developing and displaying dynamic graphical user interfaces. This paper describes the configurations of HSIs including display systems, the dynamic models of the APR1400 simulator, the instructor station, and the HUPESS. This paper also presents the results of plant simulation performance tests at transient compared with the results of RELAP5/MOD3.3 calculations. The human factors validation system for the advanced control room of APR1400 provides high degrees of physical, functional, and dynamic fidelities, and can be used in the validation process of the APR1400 HSI design.

ESTABLISING A NATURAL HRI SYSTEM FOR MOBILE ROBOT THROUGH HUMAN HAND GESTURES

a robust human robot interface system for intelligent robot commands based only on hand gesture is developed. It has a triple-stage face detection scheme and a FLC-Kalman filter to track current user position in a dynamic and cluttered working environment. Through the combined classifier of PCA and BPANN, the commands defined by facial positions and hand gestures are identified by dynamic programming for real-time controls of a mobile robot. The results show that the system accurately perform real-time face detection and tracking robustly at a speed of 8 frames per second.