Industrial Human Machine Interface Research Articles

디지털 적층 제조 장비의 사용자 중심 HMI 디자인 가이드 제안

With the advancement of metal additive manufacturing technology and the digitization of equipment, the need for user-centered design in HMI (Human-Machine Interface) has become increasingly apparent. However, research on design guides and usability test for industrial equipment has been lacking. Therefore, this study applied user-centered design principles to design and manufacture a prototype HMI for metal additive manufacturing equipment, followed by a usability test of the HMI prototype. Based on usability testing and in-depth interview analysis, this study proposes UX/UI design guidelines for user-centered industrial equipment HMI. The usability evaluation highlighted the need to improve the visibility of the system's real-time status, enhance interface intuitiveness for error prevention, and address emergency situations. It is crucial that the interface reflects both the visible and invisible states of the equipment, considering that the operation targets are mechanical equipment but the operating devices are HMI screens. Furthermore, for industrial equipment with numerous complex functions, it is essential to restrict or guide user actions through permission settings. Through the industrial PBF (Powder Bed Fusion) method metal additive manufacturing equipment HMI design guides proposed by this study, we anticipate the development of user-friendly and productivity-enhancing HMI designs.

A User Study for the Evaluation of Adaptive Interaction Systems for Inclusive Industrial Workplaces

In recent years, production systems have become highly sophisticated and complex. As a result, while on the one hand, the least-skilled labor has been partially displaced by machines, high-skilled labor is more required to supervise and control advanced automation systems. In many cases, the complexity of machines implies an increased complexity of human–machine interfaces (HMIs), which are the main point of contact between the operator and the machine. To enable effective use of HMIs and to enable their usage by workers with different knowledge and capabilities, novel design approaches have been proposed. In particular, in this article, we consider the approach developed in the framework of the European research project INCLUSIVE, which aimed at designing industrial HMIs that adapt to the skills and capabilities of human operators. As a case study, we consider an adaptive interaction system for the woodworking industry and present an extensive evaluation carried out in real production environment with shopfloor workers. The effectiveness of the INCLUSIVE approach has been assessed with subjective and objective measurements and compared to that of interaction systems customarily used in industry. Results have shown that users appreciated the INCLUSIVE system and largely preferred it over the customary system. Moreover, with regard to objective performance-related measurements, they performed better when using the INCLUSIVE system since they received tailored guidance during the considered working tasks. Note to Practitioners— This article was motivated by the fact that advanced automation systems are often highly complex for human operators. To address this problem, we focus on the importance of designing the automation system around users and discuss an approach for adaptive automation, called INCLUSIVE. Its main feature is that it adapts the human–machine interface (HMI) according to operator’s skills, capabilities, and current mental fatigue. In this article, we provide an extensive evaluation of the INCLUSIVE system, considering a company producing woodworking machines as a use case. Assessment was carried out in the company shopfloor, considering real workers, and the INCLUSIVE system was compared with the customary HMI running on the company machines. The results of our study suggest that adapting the interaction to operator’s needs allows better working performance while letting workers more satisfied with the use of the system.

Modeling and Calibration of Active Thermal-Infrared Visual System for Industrial HMI

In the industrial application of the human-machine interface (HMI), thermal-infrared cameras can detect objects that are limited by visible-spectrum cameras. It can receive the energy radiated by a target through an infrared detector and obtain the thermal image corresponding to the heat distribution field on the target surface. Because of its special imaging principle, a thermal-infrared camera is not affected by the light source when imaging. Compared to visible-spectrum cameras, thermal imaging cameras can better detect defects with poor contrast but temperature differences or internal defects in products. Therefore, it can be used in many specific industrial inspection applications. However, thermal-infrared imaging has the phenomenon of thermal diffusion, which leads to noisy thermal infrared images and limits its applications in high precision industrial environments. In this paper, we proposed a high precision measurement system for industrial HMI based on thermal-infrared vision. The accurate measurement model of the system was established to deal with the problems caused by imaging noise. The experiments conducted suggest that the proposed model and calibration method is valid for the active thermal-infrared visual system and achieves high precision measurements.

A General Methodology for Adapting Industrial HMIs to Human Operators

Modern production systems are becoming more and more complex to comply with diversified market needs, flexible production, and competitiveness. Despite technological progress, the presence of human operators is still fundamental in production plants, since they have the important role of supervising and monitoring processes, by interacting with such complex machines. The complexity of machines implies an increased complexity of human–machine interfaces (HMIs), which are the main point of contact between the operator and the machine. Thus, HMIs cannot be considered anymore an accessory to the machine and their improvement has become an important part of the design of the whole machines, to enable a nonstressful interaction and make them easy to also use less skilled operators. In this article, we present a general framework for the design of HMIs that adapt to the skills and capabilities of the operator, with the ultimate aim of enabling a smooth and efficient interaction and improving user’s situation awareness. Adaptation is achieved by considering three different levels: perception (i.e., how information is presented), cognition (i.e., what information is presented), and interaction (i.e., how interaction is enabled). For each level, general guidelines for adaptation are provided, thus defining a meta-HMI independent of the application. Finally, some examples of how the proposed adaptation patterns can be applied to the case of procedural and extraordinary maintenance tasks are presented. Note to Practitioners —This article was motivated by the problem of facilitating the interaction of human operators with human–machine interfaces (HMIs) of complex industrial systems. Standard industrial HMIs are static and do not consider the user’s characteristics. As a consequence, least-skilled operators are prevented from their use and/or have poor performance. In this article, we suggest a novel methodology to the design of adaptive industrial HMIs that adapt to the skills and capabilities of operators and compensate their limitations (e.g., due to age or inexperience). In particular, we propose a methodological framework that consists of general rules to accommodate the user’s characteristics. Adaptation is achieved at three different levels: perception (i.e., how information is presented), cognition (i.e., what information is presented), and interaction (i.e., how interaction is enabled). The presented rules are independent of the target application. Nevertheless, we establish a relationship between such design rules and user’s impairments and capabilities and kind of working tasks. Hence, designers of HMIs are called to instantiate them considering the specific requirements and characteristics of the users and the working tasks of the application at hand.

Using Augmented Reality and Internet of Things for Control and Monitoring of Mechatronic Devices

At present, computer networks are no longer used to connect just personal computers. Smaller devices can connect to them even at the level of individual sensors and actuators. This trend is due to the development of modern microcontrollers and singleboard computers which can be easily connected to the global Internet. The result is a new paradigm—the Internet of Things (IoT) as an integral part of the Industry 4.0; without it, the vision of the fourth industrial revolution would not be possible. In the field of digital factories it is a natural successor of the machine-to-machine (M2M) communication. Presently, mechatronic systems in IoT networks are controlled and monitored via industrial HMI (human-machine interface) panels, console, web or mobile applications. Using these conventional control and monitoring methods of mechatronic systems within IoT networks, this method may be fully satisfactory for smaller rooms. Since the list of devices fits on one screen, we can monitor the status and control these devices almost immediately. However, in the case of several rooms or buildings, which is the case of digital factories, ordinary ways of interacting with mechatronic systems become cumbersome. In such case, there is the possibility to apply advanced digital technologies such as extended (computer-generated) reality. Using these technologies, digital (computer-generated) objects can be inserted into the real world. The aim of this article is to describe design and implementation of a new method for control and monitoring of mechatronic systems connected to the IoT network using a selected segment of extended reality to create an innovative form of HMI.

Case Study of the Experience Capturer Evaluation Tool in the Design Process of an Industrial HMI

In the absence of user experience evaluation tools for industrial human–machine interfaces (HMI), a specific tool called eXperience Capturer (XC) has been created. It is a multi-method user-centred tool that evaluates the pragmatic and experiential aspects of employees’ interaction with industrial HMIs during the three phases of experience. In this article, a case study is shown where the XC tool is used in an industrial HMI design process. The results show that evaluation using the XC tool facilitates the creation of a new design that improves the experience of employees during interaction, increasing their autonomy, competence, closeness to the system, safety and stimulation.

NLP-Based Approach for Predicting HMI State Sequences Towards Monitoring Operator Situational Awareness.

A novel approach presented herein transforms the Human Machine Interface (HMI) states, as a pattern of visual feedback states that encompass both operator actions and process states, from a multi-variate time-series to a natural language processing (NLP) modeling domain. The goal of this approach is to predict operator response patterns for time-step window given past HMI state patterns. The NLP approach offers the possibility of encoding (semantic) contextual relations within HMI state patterns. Towards which, a technique for framing raw HMI data for supervised training using sequence-to-sequence (seq2seq) deep-learning machine translation algorithms is presented. In addition, a custom Seq2Seq convolutional neural network (CNN) NLP model based on current state-of-the-art design elements such as attention, is compared against a standard recurrent neural network (RNN) based NLP model. Results demonstrate comparable effectiveness of both the designs of NLP models evaluated for modeling HMI states. RNN NLP models showed higher () forecast accuracy, in general for both in-sample and out-of-sample test datasets. However, custom CNN NLP model showed higher () validation accuracy indicative of less over-fitting with the same amount of available training data. The real-world application of the proposed NLP modeling of industrial HMIs, such as in power generating stations control rooms, aviation (cockpits), and so forth, is towards the realization of a non-intrusive operator situational awareness monitoring framework through prediction of HMI states.

A Model-Driven Mobile HMI Framework (MMHF) for Industrial Control Systems

With the advent of software technologies, over a period of time, the Industrial Control Systems (ICSs) have grown exponentially. Whereas, almost all ICSs comprise Human Machine Interfaces (HMIs), which are the key component for monitoring and controlling complex industrial systems. For decades, traditional HMIs with simple User Interfaces (UIs) remained operational to minimize the complexities and resulting operational costs. However, due to the emergence of smartphone technologies, the perception about user interfaces has been transformed significantly and users now demand same sort of experience with industrial HMIs, as well. There are few industrial solutions, like, ICONICS GraphWorX to support the development of mobile HMI screens. However, such proprietary solutions are quite expensive. Furthermore, the underlying development approaches and source codes are not accessible in public domain. On the other hand, the state-of-the-art approaches for the development of native mobile HMI screens are hard to find in the literature. Consequently, there is dire need of a cost-effective, easy to use, open source framework for the development of native mobile HMI screens. In order to achieve this goal, here we propose, a M odel-driven M obile H MI F ramework ( MMHF ). MMHF comprises, a Unified Modeling Language (UML) Profile for Mobile HMI (UMLPMH) for modeling of HMI screens. MMHF also includes, an open source transformation engine and a M odel D riven M obile-based H MI C ode G enerator (MDMHCG) to automatically transform UMLPMH models into target native mobile HMI implementations. Consequently, MMHF enables simpler way to design the HMI screens using UMLPMH and generates native Mobile HMI Screen implementations automatically using MDMHCG. The empirical evidence of MMHF is demonstrated through three (3) benchmark case studies, which prove that the MMHF is a feasible, cost effective and scalable solution to develop native HMI screens for wide-ranging ICSs.

임베디드 리눅스 기반 산업용 무선 HMI 소프트웨어 모듈 설계 및 구현

Industrial HMI(Human Machine Interface) system is the main element among the factory automation processes and have been used to monitor and control operation and status of machine in factory with PLC. This HMI often brings heavy loads to the system development and difficult decreasing the system because it tends to use a specific system per each manufacturer. Therefore, in this thesis, we have developed an embedded linux-based embedded industrial HMI software modules which can be used for touch panel embedded system to solve these problem. In this module, we have used the Qt/Embedded software component because it can be used by all systems which support C++ compiler without modifying the existing codes. We can design more flexible system and network configuration because we have used the wireless communication module. In this thesis, we implement linux-based HMI software modules which are capable of wireless communication as well as bringing the mobility to the overall system and finally decreasing the system development loads by using the general purpose OS with competitive price.