HMI Systems Research Articles

Industrial process optimization through advanced HMI systems: exploring the integration of IoT and AI

Industrial process optimization through advanced HMI systems: exploring the integration of IoT and AI

Live application programming in the defense industry with the Molecule component framework

At Thales Defense Mission Systems (DMS), software products first go through an industrial prototyping phase. Prototypes are serious applications that we evaluate with our end-users during demonstrations. End-users have a central role in the design process of our products. They often ask for software modifications during demonstrations to experiment new ideas or to focus the existing design on their needs.In this paper, we present how we combined Smalltalk’s live-programming capabilities with software component models to obtain flexible and modular software designs in our context of live prototyping. We present Molecule, an open-source implementation of the Lightweight CORBA Component Model in Pharo. We use Molecule to build HMI systems prototypes, and we benefit from the dynamic run-time modification capabilities of Pharo during demonstrations with our end-users where we explore software designs in a lively way.Molecule is an industrial contribution to Smalltalk, as it capitalizes 20 years of usage and maturation in our prototyping activity. The Molecule framework and tools are now mature, and we started building end-user software used in production at Thales DMS. We present two such end-user software and analyze their component architecture, that are representative of how we (learnt to) build HMI prototypes. Finally, we analyze our technological decisions with regards to the benefits we sought for our industrial activity.

Implementation of a Trust-Based Framework for Substation Defense in the Smart Grid

The Smart Grid is a cyber-integrated power grid that manages electricity generation, transmission, and distribution to consumers and central to its functioning is the substation. However, integrating cyber-infrastructure into the substation has increased its attack surface. Notably, sophisticated attacks such as the PipeDream APT exploit multiple device protocols, such as Modbus, DNP3, and IEC61850. The substation’s constraints pose challenges for implementing security measures such as encryption and intrusion detection systems. To address this, we propose a comprehensive trust-based framework aimed at enhancing substation security. The framework comprises a trust model, a risk posture model, and a trust transferability model. The trust model detects protocol-based attacks on Intelligent Electronic Devices and SCADA HMI systems, while the risk posture model dynamically assesses the substation’s risk posture. The trust transferability model evaluates the feasibility of transferring and integrating a device and its trust capabilities into a different substation. The practical substation emulation involves a Docker-based testbed, employing a multi-agent architecture with a real-time Security Operations Center-influenced dashboard. Assessment involves testing against attacks guided by the MITRE ICS ATT&CK framework. Our framework displays resilience against diverse attacks, identifies malicious behavior, and rewards trustworthy devices.

Security Baseline for Substation Automation Systems.

The use of information technology and the automation of control systems in the energy sector enables a more efficient transmission and distribution of electricity. However, in addition to the many benefits that the deployment of intelligent and largely autonomous systems brings, it also carries risks associated with information and cyber security breaches. Technology systems form a specific and critical communication infrastructure, in which powerful control elements integrating IoT principles and IED devices are present. It also contains intelligent access control systems such as RTU, IDE, HMI, and SCADA systems that provide communication with the data and control center on the outer perimeter. Therefore, the key question is how to comprehensively protect these specialized systems and how to approach security implementation projects in this area. To establish rules, procedures, and techniques to ensure the cyber security of smart grid control systems in the energy sector, it is necessary to understand the security threats and bring appropriate measures to ensure the security of energy distribution. Given the use of a wide range of information and industrial technologies, it is difficult to protect energy distribution systems using standard constraints to protect common IT technologies and business processes. Therefore, as part of a comprehensive approach to cyber security, specifics such as legislative framework, technological constraints, international standards, specialized protocols or company processes, and many others need to be considered. Therefore, the key question is how to comprehensively protect these specialized systems and how to approach security implementation projects in this area. In this article, a basic security concept for control systems of power stations, which are part of the power transmission and distribution system, is presented based on the Smart Grid domain model with emphasis on substation intelligence, according to the Purdue model. The main contribution of the paper is the comprehensive design of mitigation measures divided into mandatory and recommended implementation based on the standards defined within the MITRE ATT&CK matrix specified, concerning the specifications of intelligent distribution substations. The proposed and industry-tested solution is mapped to meet the international security standards ISO 27001 and national legislation reflecting the requirements of NIS2. This ensures that the security requirements will be met when implementing the proposed Security Baseline.

A Standard Set of Courses to Assess the Quality of Driving Off-Road Combat Vehicles

<div class="section abstract"><div class="htmlview paragraph">Making manned and remotely-controlled wheeled and tracked vehicles easier to drive, especially off-road, is of great interest to the U.S. Army. If vehicles are easier to drive (especially closed hatch) or if they are driven autonomously, then drivers could perform additional tasks (e.g., operating weapons or communication systems), leading to reduced crew sizes. Further, poorly driven vehicles are more likely to get stuck, roll over, or encounter mines or improvised explosive devices, whereby the vehicle can no longer perform its mission and crew member safety is jeopardized.</div><div class="htmlview paragraph">HMI technology and systems to support human drivers (e.g., autonomous driving systems, in-vehicle monitors or head-mounted displays, various control devices (including game controllers), navigation and route-planning systems) need to be evaluated, which traditionally occurs in mission-specific (and incomparable) evaluations.</div><div class="htmlview paragraph">To support the use of comparable test conditions, a set of combat-relevant driving courses was developed for usability evaluations. This set of courses has been implemented in simulation (the Detroit Arsenal TBMS/CS simulator) and in the field (Camp Grayling). The 9 courses are: (1) on-road driving, (2) slalom (to avoid obstacles), (3) mogul (to assess rollover propensity), (4) ditches (which are difficult to see), (5) minefield (where the path is narrow), (6) berm drill (a defensive maneuver), (7) urban cover (scampering between buildings), (8) urban drive (narrow alleys), and (9) formation change (e.g., line to column). The courses used at Camp Grayling are documented here and representative driving data are provided to support use of these courses by others. Improvements are also suggested.</div></div>

Implementation of a Hybrid Intelligence System Enabling the Effectiveness Assessment of Interaction Channels Use in HMI.

The article presents a novel idea of Interaction Quality Sensor (IQS), introduced in the complete solution of Hybrid INTelligence (HINT) architecture for intelligent control systems. The proposed system is designed to use and prioritize multiple information channels (speech, images, videos) in order to optimize the information flow efficiency of interaction in HMI systems. The proposed architecture is implemented and validated in a real-world application of training unskilled workers-new employees (with lower competencies and/or a language barrier). With the help of the HINT system, the man-machine communication information channels are deliberately chosen based on IQS readouts to enable an untrained, inexperienced, foreign employee candidate to become a good worker, while not requiring the presence of either an interpreter or an expert during training. The proposed implementation is in line with the labor market trend, which displays significant fluctuations. The HINT system is designed to activate human resources and support organizations/enterprises in the effective assimilation of employees to the tasks performed on the production assembly line. The market need of solving this noticeable problem was caused by a large migration of employees within (and between) enterprises. The research results presented in the work show significant benefits of the methods used, while supporting multilingualism and optimizing the preselection of information channels.

Use of a DNN in Recording and Analysis of Operator Attention in Advanced HMI Systems

The main objective of this research was to propose a smart technology to record and analyse the attention of operators of transportation devices where human–machine interaction occurs. Four simulators were used in this study: General Aviation (GA), Remotely Piloted Aircraft System (RPAS), AS 1600, and Czajka, in which a spatio-temporal trajectory of system operator attention describing the histogram distribution of cockpit instrument observations was sought. Detection of the position of individual instruments in the video stream recorded by the eyetracker was accomplished using a pre-trained Fast R-CNN deep neural network. The training set for the network was constructed using a modified Kanade–Lucas–Tomasi (KLT) algorithm, which was applied to optimise the labelling of the cockpit instruments of each simulator. A deep neural network allows for sustained instrument tracking in situations where classical algorithms stop their work due to introduced noise. A mechanism for the flexible selection of Area Of Interest (AOI) objects that can be tracked in the recorded video stream was used to analyse the recorded attention using a mobile eyetracker. The obtained data allow for further analysis of key skills in the education of operators of such systems. The use of deep neural networks as a detector for selected instrument types has made it possible to universalise the use of this technology for observer attention analysis when applied to a different objects-sets of monitoring and control instruments.

Cobot as the Part of Multi-agent Model of a Manufacturing System

In recent years an upward trend has been observed regarding the use of robots in the industry. It is caused, among other things, by introduction of assumptions that causes minimization of employee’s participation in the manufacturing process. Such approach allows improving quality, to optimize manufacturing costs, to increase productivity and at the same time introduce full supervision over the manufacturing process. However, there are still processes that are performed faster, and more efficiently by a worker, without the need for special training. In such cases, the implementation of the robot is not economically justified. However, it is possible to install a cobot that acts as an assistant in relation to the activities performed by the employee. In the industrial environment, cobots operate on the boundary between digitally control led systems and the domain of factory workers. This paper will discuss the problems relate to the use of agent approach in order to model the cobot as a part of multiagent model of a manufacturing system. An important issue is the interaction and communication with the worker and its consequences for the entire system. It is also necessary to take into account the information provided by HMI systems and signals coming from the appropriate sensors.

Multi-modal user experience evaluation on in-vehicle HMI systems using eye-tracking, facial expression, and finger-tracking for the smart cockpit

Multi-modal user experience evaluation on in-vehicle HMI systems using eye-tracking, facial expression, and finger-tracking for the smart cockpit

Multi-modal user experience evaluation on in-vehicle HMI systems using eye-tracking, facial expression, and finger-tracking for the smart cockpit

Multi-modal user experience evaluation on in-vehicle HMI systems using eye-tracking, facial expression, and finger-tracking for the smart cockpit

Smart Glasses and Augmented Reality: Empowering Maintenance Personnel in Industrial Settings

The efficient maintenance and servicing of industrial equipment are crucial for the smooth operation of manufacturing facilities. However, maintenance personnel often encounter challenges in locating and addressing specific issues, such as equipment jams, within complex systems. Despite the presence of SCADA or HMI systems that can indicate the existence of a fault, identifying the exact location of the issue within the system remains a significant hurdle. This research aims to explore the development and implementation of smart glasses integrated with augmented reality (AR) technology to provide maintenance personnel with real-time, location-based navigation assistance within industrial environments. By leveraging AR, the proposed smart glass software will function as a navigational tool, guiding users directly to the precise location of equipment faults, resembling "Google Maps" for industrial maintenance. This research seeks to investigate the potential impact of AR-assisted navigation on the efficiency and effectiveness of maintenance activities, particularly in complex industrial settings. Furthermore, the study will address the usability, user acceptance, and technical feasibility of integrating AR technology into maintenance processes, ultimately contributing to the advancement of smart maintenance solutions in industrial settings.

Pengembangan Embedded Device Berbasis PLC untuk Simulator Rejection System dengan Penambahan Human Machine Interface

Changes to the simulator rejection system from the previous research were carried out by replacing all sensors, drive motors, PLC systems, and adding HMI systems. The objectives in this research, namely (i) changing and developing a rejection system simulator, creating a ladder-based program structure and configuring HMI systems and (ii) measuring the performance of the simulators. Rejection system simulator is fabricated and reassembled, ladder-based syntax into PLCs and HMI is also configured, and observing the performance is done through the HMI layer. The results of programming is carried out through (i) providing software for PLCs, (ii) programming the PLC system, (iii) compiling and uploading programs from PC to PLC, (iv) configuring PLC and HMI via ethernet, and (v) compiling and uploading the program structure from PC to HMI. The performance for observing the condition of the bottle cap through the HMI is observed when (i) synchronization between the simulator system and the HMI-assisted PLC control, (ii) the reading of the sensors installed on the simulator, and (iii) the rotation process of the rejector arm. Overall, the rejection system simulator with a PLC-based assisted by HMI can be used as a process simulation against the implementation of the rejection system

Modern Human Machine Interface in Industrial Wireless Networks

This paper presents several existing experimental setups on latest industrial systems and provide comparison of advantages and disadvantages of different HMI implementation. It proposes a simple and low cost HMI system for automation by using remote software application. The method does not required specific software package from vendors, hence provide a low cost and easily available solutions for control. We have examined their switching on various wireless technologies including Bluetooth, Infrared, Wi-Fi and Cellular Mobile Systems. The presented HMI provide effective and inexpensive control of automation systems by using remote software packages on the mobile phone, tablet and laptop. This paper presents several experiments on HMIs. The first set of experiments are related existing HMI systems on latest Siemens TIA Portal software and hardware. The second set of experiments presents the developed method of HMI switching and control by using remote communication software; Vectir and Teamviewer over different industrial wireless technologies.

Telugu Speech Recognition on LSF and DNN Techniques

This fast world is running with machine and human interaction. This kind of interaction is not an easy task. For proper interaction between human and machine speech recognition is major area where the machine should understand the speech properly to perform the tasks. So ASR have been developed which improvised the HMIS (“Human Machine Interaction systems”) technology in to the deep level. This research focuses on speech recognition over “Telugu language”, which is used in Telugu HMI systems. This paper uses LSF (linear spectral frequencies) technique for feature extraction and DNN for feature classification which finally produced the effective results. Many other recognition systems also used these techniques but for Telugu language this are the most suitable techniques.

Maintaining HMI and SCADA Systems Through Computer Virtualization

Maintaining HMI and SCADA Systems Through Computer Virtualization

Innovative approach in signal processing of electromyography signals

with its potential implementation for the purpose of HMI systems, where embedded platforms are applied. The method does not involve any traditional, statistical signal processing methods. Materials and methods. The proposed method differs from the traditional signal processing methods due to the no need of using equipment with high-computing power, which results with its wide potential implementation. Signal processing of various bio-signals is currently a very dynamically developing scientific area. The innovation of the proposed solution relies on its simplicity, efficiency and waht’s more – it does not implement any statistical signal processing. Results. The proposed method has prospective implementation for the control purpose in order to improve quality of life for handicapped users. Conducted research was intended for potential application on an embedded system platform, which has caused some significant limits in choosing an appropriate signal processing method. Traditional, sophisticated, statistical signal processing methods were not used for the purpose of this work. Discussion. This paper is a pilot study for the prospective EMG-based control of an artificial hand. The conducted analysis was done in an off-line mode, however further plans on on-line signal processing plans were made. All calculations were done in MATLAB. Conclusions. The paper presents an innovation as no other similar methods were found in literature. The paper also shows

Eye Glance and Head Turn Correspondence during Secondary Task Performance in Simulator Driving

This paper and its findings serve as a fundamental characterization of the interaction between the head and eye as a function of task demand and duration while driving. These findings have potential use in future vehicle active safety, and the design of driver assistance and/or HMI systems and features. Our findings show that head position and eye-point-of regard for certain task locations have dependence and may be suited to characterize task engagement while driving.

Rapid Prototyping of Automotive HMI Systems Utilizing Vector CANape and Mathworks Simulink

Automotive electronics systems are continuously becoming more complicated with an ever increasing list of features. As the level of content increases, so does the interaction between the systems. This additional complexity can translate into a more confusing HMI experience for the driver if it is not designed carefully. This is especially true for features that interact across multiple systems, where each must seamlessly integrate. It becomes clear that a method to rapidly prototype such HMI systems is desirable to enable testing of HMI concepts as early in the design process as possible. This paper will describe one such method that we have successfully utilized incorporating the use of Vector CANape software to quickly create prototype HMI experiences that integrates Mathworks Simulink feature code, touchscreen interaction, user inputs and the vehicle CAN network to simulate an in vehicle HMI experience.

Web Monitoring System and Gateway for Serial Communication PLC

An industrial process requires interacting with the rest of the plant, being able to exchange data with other devices and monitoring systems in order to optimize production, reporting information and providing control capabilities to distant users. Internet, and, especially web browsers are an excellent tool to provide information for remote users, allowing not only monitoring but also controlling the industrial process as an SCADA software or HMI system. The proposed system does not need specific proprietary software and its associated license costs. In this work, a webserver system is implemented under a Freescale microcontroller, acting as a gateway for a simple PLC with single RS232 communication capabilities. The webserver is modular, providing independent access to single I/O PLC locations. Different webpage design can offer different monitoring capabilities by combining the required I/O modules according to the required application without any change in the microcontroller programming. This capability is close to SCADA software or industrial HMI systems where custom screens can be made. This proposal offers a low cost and flexible monitoring solution to old or basic industrial processes controlled by PLC with low communication capabilities. Using a web browser, the system can be monitored from any internet capable device: PC, tablet, smartphone, etc. An example for a pneumatic PID levitation control system is given.

Investigating the Functionality of Different Human Machine Interfaces for Cooperative Adaptive Cruise Control

Cooperative-Adaptive Cruise Control (C-ACC) systems are seen as a potential technology to increase capacity and safety within Intelligent Traffic Systems (ITS). In this research, interaction mediums of the driver for both advisory and autonomous platforms are investigated. Two different advisory and automated frameworks with three different interfaces including a visual dashboard display, a Head Up Display (HUD) and a haptic gas pedal are studied. It is shown that the designed advisory interfaces do not result in significant improvement in velocity control but subjective evaluation indicates a preference for haptic feedback. For the autonomous cooperative cruise control, the designed HUD interface results in about 50% reduction of undesired human intervention which is significant for some conditions. Next versions of the HMI systems will be tested with more subjects, more scenarios and longer test durations to gain more insight into variation and accommodation.