Machine Interface Research Articles

Sensorial Feedback Contribution to the Sense of Embodiment in Brain–Machine Interfaces: A Systematic Review

The sense of embodiment (SoE) is an essential element of human perception that allows individuals to control and perceive the movements of their body parts. Brain–machine interface (BMI) technology can induce SoE in real time, and adding sensory feedback through various modalities has been shown to improve BMI control and elicit SoEe. In this study, we conducted a systematic review to study BMI performance in studies that integrated SoE variables and analyzed the contribution of single or multimodal sensory stimulation. Out of 493 results, only 20 studies analyzed the SoE of humans using BMIs. Analysis of these articles revealed that 40% of the studies relating BMIs with sensory stimulation and SoE primarily focused on manipulating visual stimuli, particularly in terms of coherence (i.e., synchronous vs. asynchronous stimuli) and realism (i.e., humanoid or robotic appearance). However, no study has analyzed the independent contributions of different sensory modalities to SoE and BMI performance. These results suggest that providing a detailed description of the outcomes resulting from independent and combined effects of different sensory modalities on the experience of SoE during BMI control may be relevant for the design of neurorehabilitation programs.

Theoretical Basis of Biomimetic Flexible Piezoelectric Acoustic Sensors for Future Customized Auditory Systems

AbstractFlexible piezoelectric sensors have been spotlighted as an essential human–machine interface (HMI) by obtaining high‐quality data from omnipresent biomechanical inputs. Because human voice is the most intuitive bio‐signal among them, flexible piezoelectric acoustic sensors (f‐PAS) have a potential to shift the paradigm of HMI technologies. Despite the reported outstanding performance such as high sensitivity and speaker recognition accuracy, the theoretical investigation of f‐PAS has been insufficient to realize future customized development, because sensing principles are fundamentally different from commercialized microphones. Here, a theoretical framework of self‐powered f‐PAS by using mechanical and electrical physics is introduced. First of all, the basic theory of f‐PAS is compared with the auditory system of human cochlear. Based on the biomimetic trapezoidal shape, the resonant frequencies are analyzed with various structural and material conditions. In addition, the piezoelectricity of f‐PAS is derived to predict the sensitivity and SNR prior to experiments. To investigate sensor properties under the medium condition that is similar to human ear, the acoustic responses depending on the states of matter are theoretically compared. Finally, the distance limit of f‐PAS is studied with the correlations between piezoelectricity and sound pressure, which would provide novel strategies of functional material design for future applications of f‐PAS.

A spiking neural network with continuous local learning for robust online brain machine interface

Objective. Spiking neural networks (SNNs) are powerful tools that are well suited for brain machine interfaces (BMI) due to their similarity to biological neural systems and computational efficiency. They have shown comparable accuracy to state-of-the-art methods, but current training methods require large amounts of memory, and they cannot be trained on a continuous input stream without pausing periodically to perform backpropagation. An ideal BMI should be capable training continuously without interruption to minimize disruption to the user and adapt to changing neural environments. Approach. We propose a continuous SNN weight update algorithm that can be trained to perform regression learning with no need for storing past spiking events in memory. As a result, the amount of memory needed for training is constant regardless of the input duration. We evaluate the accuracy of the network on recordings of neural data taken from the premotor cortex of a primate performing reaching tasks. Additionally, we evaluate the SNN in a simulated closed loop environment and observe its ability to adapt to sudden changes in the input neural structure. Main results. The continuous learning SNN achieves the same peak correlation () as existing SNN training methods when trained offline on real neural data while reducing the total memory usage by 92%. Additionally, it matches state-of-the-art accuracy in a closed loop environment, demonstrates adaptability when subjected to multiple types of neural input disruptions, and is capable of being trained online without any prior offline training. Significance. This work presents a neural decoding algorithm that can be trained rapidly in a closed loop setting. The algorithm increases the speed of acclimating a new user to the system and also can adapt to sudden changes in neural behavior with minimal disruption to the user.

Real-Time Deep Neurolinguistic Learning Enhances Noninvasive Neural Language Decoding for Brain-Machine Interaction.

Electroencephalogram (EEG)-based brain-machine interface (BMI) has been utilized to help patients regain motor function and has recently been validated for its use in healthy people because of its ability to directly decipher human intentions. In particular, neurolinguistic research using EEGs has been investigated as an intuitive and naturalistic communication tool between humans and machines. In this study, the human mind directly decoded the neural languages based on speech imagery using the proposed deep neurolinguistic learning. Through real-time experiments, we evaluated whether BMI-based cooperative tasks between multiple users could be accomplished using a variety of neural languages. We successfully demonstrated a BMI system that allows a variety of scenarios, such as essential activity, collaborative play, and emotional interaction. This outcome presents a novel BMI frontier that can interact at the level of human-like intelligence in real time and extends the boundaries of the communication paradigm.

Machine learning-coupled tactile recognition with high spatiotemporal resolution based on cross-striped nanocarbon piezoresistive sensor array

Flexible pressure sensor arrays have been playing important roles in various applications of human–machine interface, including robotic tactile sensing, electronic skin, prosthetics, and human–machine interaction. However, it remains challenging to simultaneously achieve high spatial and temporal resolution in developing pressure sensor arrays for tactile sensing with robust function to achieve precise signal recognition. This work presents the development of a flexible high spatiotemporal piezoresistive sensor array (PRSA) by coupling with machine learning algorithms to enhance tactile recognition. The sensor employs cross-striped nanocarbon-polymer composite as an active layer, though screen printing manufacture processes. A miniaturized signal readout circuit and transmission board is developed to achieve high-speed acquisition of distributed pressure signals from the PRSA. Test results indicate that the developed PRSA platform simultaneously possesses the characteristics of high spatial resolution up to 1.5 mm, fast temporal resolution of about 5 ms, and long-term durability with a variation of less than 2%. The PRSA platform also exhibits excellent performance in real-time visualization of multi-point touch, mapping embossed shapes, and tracking motion trajectory. To test the performance of PRSA in recognizing different shapes, we acquired pressure images by pressing the finger-type device coated with PRSA film on different embossed shapes and implementing the T-distributed Stochastic Neighbor Embedding model to visualize the distinction between images of different shapes. Then we adopted a one-layer neural network to quantify the discernibility between images of different shapes. The analysis results show that the PRSA could capture the embossed shapes clearly by one contact with high discernibility up to 98.9%. Collectively, the PRSA as a promising platform demonstrates its promising potential for robotic tactile sensing.

Decoding motor plans using a closed-loop ultrasonic brain–machine interface

Brain–machine interfaces (BMIs) enable people living with chronic paralysis to control computers, robots and more with nothing but thought. Existing BMIs have trade-offs across invasiveness, performance, spatial coverage and spatiotemporal resolution. Functional ultrasound (fUS) neuroimaging is an emerging technology that balances these attributes and may complement existing BMI recording technologies. In this study, we use fUS to demonstrate a successful implementation of a closed-loop ultrasonic BMI. We streamed fUS data from the posterior parietal cortex of two rhesus macaque monkeys while they performed eye and hand movements. After training, the monkeys controlled up to eight movement directions using the BMI. We also developed a method for pretraining the BMI using data from previous sessions. This enabled immediate control on subsequent days, even those that occurred months apart, without requiring extensive recalibration. These findings establish the feasibility of ultrasonic BMIs, paving the way for a new class of less-invasive (epidural) interfaces that generalize across extended time periods and promise to restore function to people with neurological impairments.

A novel equestrian helmet testing method: helmet liner performance in highly realistic simulation

ABSTRACT Objective Employ a novel testing method to assess Multi Directional Impact Protection System (MIPS) helmet technology on rotational velocity and acceleration during head impact. Methods An optimization study was completed utilizing a 50th percentile male Hybrid III anthropomorphic test device (ATD). Helmets included expanded polystyrene foam (EPS) and two different MIPS helmets (MIPS 1, MIPS 2). A 24.38-m-long elevated track with rails and a motorized sled was utilized to replicate a fall from approximately 2.13 m. The sled was set to a speed of 20.92 kph, where a tripping mechanism induced rotation in the ATD from the sled and onto a sand surface. During impact of the ATD with the sand surface, head kinematics were measured using resultant acceleration (peak G’s), duration of impact (ms), and rotational velocity (rad/s). Results A total of three trials for each helmet did not demonstrate a significant difference between the EPS vs. MIPS 1 group with, peak (G’s) for resultant acceleration (p = 0.100), duration (ms) for resultant acceleration, (p = 0.100), peak (G’s) for rotational velocity, (p = 0.700), and duration (ms) for rotational velocity (p = 0.700). Similarly, the EPS vs. MIPS 2 testing demonstrated no significant differences between the MIPS 2 helmet compared to the EPS helmet, with resultant acceleration (p = 0.400), duration acceleration (p = 0.200), rotational velocity (p = 0.400) and duration velocity (p = 0.400). However, when the MIPS helmet data were pooled, and the EPS helmet data were compared, a statistically significant difference in the duration of acceleration was found (p = 0.048). Conclusions Current testing uses a helmeted head form which is dropped or rolled from a prescribed height. These methods discount the loading placed on the neck and head through the angular momentum of the body. Our novel testing method did not find significant differences between the helmet types in diminishing peak rotational forces to the brain; however, our data suggests that MIPS helmet liners may reduce duration of impact. The reduction of acceleration duration could indicate less rotation of the neck, due to the dampening of these forces by the MIPS liners.

The Development Process of Human Machine Interface of Plant Protection System of a Small Modular Reactor

The Plant Protection System (PPS), which consists of Reactor Protection Systems (RPS) and Engineered Safety Actuated Systems (ESFAS), is one of the most important safety systems in nuclear reactors, including Small Modular Reactors (SMRs). The RPS generates a signal to trip the reactor if the measured reactor parameters exceed the trip setpoint, and then the ESFAS is actuated to mitigate the consequences of the accident by minimizing fuel damage and radioactivity release into the environment. Therefore, a comprehensive Human-Machine Interface (HMI) is essential for monitoring and controlling the PPS to ensure its reliability and enhance the operators' situational awareness. This study discusses the development process of the HMI for the digital PPS of an SMR. In this study, various standards, guidance, and design criteria for PPS and HMI are incorporated and applied to ensure that the proposed design meets the required level of reliability. In the first stage, the proposed design is intended for assessing the functionality and reliability of the PPS. Moreover, in the future, it will play an essential role in the design phase of the HMI for the PPS of an SMR in Indonesia.

Comparative Analysis of Vision Transformer Models for Facial Emotion Recognition Using Augmented Balanced Datasets

Facial emotion recognition (FER) has a huge importance in the field of human–machine interface. Given the intricacies of human facial expressions and the inherent variations in images, which are characterized by diverse facial poses and lighting conditions, the task of FER remains a challenging endeavour for computer-based models. Recent advancements have seen vision transformer (ViT) models attain state-of-the-art results across various computer vision tasks, encompassing image classification, object detection, and segmentation. Moreover, one of the most important aspects of creating strong machine learning models is correcting data imbalances. To avoid biased predictions and guarantee reliable findings, it is essential to maintain the distribution equilibrium of the training dataset. In this work, we have chosen two widely used open-source datasets, RAF-DB and FER2013. As well as resolving the imbalance problem, we present a new, balanced dataset, applying data augmentation techniques and cleaning poor-quality images from the FER2013 dataset. We then conduct a comprehensive evaluation of thirteen different ViT models with these three datasets. Our investigation concludes that ViT models present a promising approach for FER tasks. Among these ViT models, Mobile ViT and Tokens-to-Token ViT models appear to be the most effective, followed by PiT and Cross Former models.

Mixed Reality: A Tool for Investigating the Complex Design and Mechanisms of a Mechanically Actuated Digital Pump

Digital hydraulics is a discrete technology that integrates advanced dynamic system controls, digital electronics, and machine learning to enhance fluid power systems’ performance, overall efficiency, and controllability. A mechanically actuated inline three-piston variable displacement digital pump was previously proposed and designed. The inline three-piston pump incorporates complex mechanical and hydraulic subsystems and highly coupled mechanisms. The complexity of the utilized subsystems poses challenges when assessing the viability of the conceptual design. Therefore, this work focuses on designing, developing, and implementing a collaborative virtual platform involving a digitized module showcasing the internal mechanical structure of the digital pump utilizing mixed reality (MR) technology. MR technology is acknowledged as the forthcoming evolution of the human–machine interface in the real–virtual environment utilizing computers and wearables. This technology permits running simulations that examine the complexity of highly coupled systems, like the digital pump, where understanding the physical phenomenon is far too intricate. The developed MR platform permits multiple users to collaborate in a synchronized immersive MR environment to study and analyze the applicability of the pump’s design and the adequacy of the operated mechanisms. The collaborative MR platform was designed and developed on the Unity game engine, employing Microsoft Azure and Photon Unity Networking to set up the synchronized MR environment. The platform involves a fully interactive virtual module on the digital pump design, developed in multiple stages using Microsoft’s Mixed Reality Tool Kit (MRTK) for Unity and deployed in the synchronized MR environment through a HoloLens 2 MR headset. A research study involving 71 participants was carried out at Purdue University. The study’s objective was to explore the impact of the collaborative MR environment on understanding the complexity and operation of the digital pump. It also sought to assess the effectiveness of MR in facilitating collaboration among fluid power stakeholders in a synchronized digital reality setting to study, diagnose, and control their complex systems. Surveys were designed and completed by all 71 participants after experiencing the MR platform. The results indicate that approximately 75% of the participants expressed positive attitudes toward their overall MR platform experience, with particular appreciation for its immersive nature and the synchronized collaborative environment it provided. More than 70% of the participants agreed that the pump’s collaborative MR platform was essential for studying and understanding the complexity and intricacy of the digital pump’s mechanical structure. Overall, the results demonstrate that the MR platform effectively facilitates the visualization of the complex pump’s internal structure, inspection of the assembly of each of the involved subsystems, and testing the applicability of the complicated mechanisms.

Automatic Transfer Switch (ATS) Using Omron CP1L PLC and Human Machine Interface

Airports in aviation services are required to use facilities with the latest technology. This requires a stable and uninterrupted electricity supply to operate correctly. The Automatic Transfer Switch design based on the CX-Designer Human Machine Interface moves the electrical power supply source from PLN to the Generator Set or vice versa. There are two operating modes, namely manual mode and auto mode. In operation, it is also equipped with an interlock function between the PLN contactor and the generator and a PLN phase loss sensor for the R, S and T phases. For auto mode, it uses a timer with a setting for PLN of 12 seconds and for the generator of 15 seconds. The control system uses an Omron CP1L-L20DR-D PLC with CX-Programmer programming for Human Machine Interface with CX-Designer. The Automatic Transfer Switch test was successful in system testing, starting from manual and auto mode, PLN phase loss testing, and timer testing; 100% of the entire hardware and software system ran well.

Passive Electrical Components Based on Cotton Fabric Decorated with Iron Oxides Microfibers: The Influence of Static and Pulsed Magnetic Fields on the Equivalent Electrical Properties.

In this work, environmentally friendly and low-cost passive electrical components (PECs) are manufactured based on composites consisting of cotton fabrics soaked with solutions of silicone oil and different amounts of iron oxides microfibers (μFe). The μFe consists of a mixture of three phases: hematite (α-Fe2O3), maghemite (γ-Fe2O3), and magnetite (Fe3O4). The equivalent electrical capacitance (Cp) and resistance (Rp) of PECs are measured as a function of magnetic flux density B in a static and pulsed magnetic field superimposed on an alternating electric field of frequency 1 kHz. The relative variation in the hysteresis curves for both Cp and Rp are obtained by measuring them in the ascending and then the descending mode of B. We show that all these three quantities are sensibly influenced by the volume fractions of μFe and by the values of B. The main influence on this behavior is attributed to the semiconductor properties of the α-Fe2O3 and γ-Fe2O3 components of the oxide microfibers. In addition, it is found that at B≃ 175 mT, the maximum relative variance of the hysteresis curve is about 3.35% for Cp and 3.18 % for Rp. When a pulsed magnetic field is used, it is shown that Cp and Rp closely follow the variation in the magnetic field. Thus, the resulting electrical properties of PECs, together with the fast response to the application of pulsed magnetic fields, make them useful in the fabrication of various devices, such as electric, magnetic, and deformation fields, or mechanical stress sensors with applications in protection against electromagnetic smog, healthcare monitoring, or for human-machine interfacing.

Volitional activation of remote place representations with a hippocampal brain-machine interface.

The hippocampus is critical for recollecting and imagining experiences. This is believed to involve voluntarily drawing from hippocampal memory representations of people, events, and places, including maplike representations of familiar environments. However, whether representations in such "cognitive maps" can be volitionally accessed is unknown. We developed a brain-machine interface to test whether rats can do so by controlling their hippocampal activity in a flexible, goal-directed, and model-based manner. We found that rats can efficiently navigate or direct objects to arbitrary goal locations within a virtual reality arena solely by activating and sustaining appropriate hippocampal representations of remote places. This provides insight into the mechanisms underlying episodic memory recall, mental simulation and planning, and imagination and opens up possibilities for high-level neural prosthetics that use hippocampal representations.

The neural basis of mental navigation in rats.

A brain-machine interface demonstrates volitional control of hippocampal activity.

Layer Engineered MXene Empowered Wearable Pressure Sensors for Non‐Invasive Vital Human–Machine Interfacing Healthcare Monitoring

AbstractPressure sensors with high flexibility and sensitivity face significant challenges in meeting the delicate balance and synergy among suitable active sensing electrode materials, substrates, and their device geometry design. In this contribution, layer‐engineered delaminated Ti‐MXene (DL‐Ti3C2Tx) is introduced, which has relatively wider interlayer spacing through intercalated large organic molecules and accordion‐like open internal microstructure than the narrower pristine Ti3C2Tx MXene (Ti‐MXene), graphene/carbon nanotube's interlayer spacing suitably fulfill the high sensitivity and flexibility requirement through accessible electronic pathways under the external pressure. Notably, a milder in‐situ ambient condition etching is performed to eliminate the associated safety risks for a flexible personal healthcare monitoring pressure sensor. DL‐Ti3C2Tx MXene‐empowered, flexible pressure sensor demonstrates a broad range of sensitivities up to a very high‐pressure of 20.8 kPa at a sensitivity of 242.3 kPa−1 with a fast response and recovery time (<300 ms). A twofold increase in pressure sensitivity performance of DL‐Ti3C2Tx MXene than that of Ti‐MXene, graphene can be attributed to the engineered wider interlayer distance among the delaminated DL‐Ti3C2Tx MXene layers causing a facile interlayer atomic movements, contacts, and reversible compressibility. The current economical, scalable DL‐Ti3C2Tx MXene flexible pressure sensor can provide future safe personal healthcare artificial intelligence with real‐time tracking ability.

Local eye-net: An attention based deep learning architecture for localization of eyes

Development of human machine interface has become a necessity for modern day machines to catalyze more autonomy and more efficiency. Gaze driven human intervention is an effective and convenient option for creating an interface to alleviate human errors. Facial landmark detection is very crucial for designing a robust gaze detection system. Regression based methods capacitate good spatial localization of the landmarks corresponding to different parts of the faces. But there are still scope of improvements which have been addressed by incorporating attention. In this paper, we have proposed a deep coarse-to-fine architecture called LocalEyenet for localization of only the eye regions that can be trained end-to-end. The model architecture, build on stacked hourglass backbone, learns the self-attention in feature maps which aids in preserving global as well as local spatial dependencies in face image. We have incorporated deep layer aggregation in each hourglass to minimize the loss of attention over the depth of architecture. Our model shows good generalization ability in cross-dataset evaluation and in real-time localization of eyes.

Deep Learning-Based Hand Gesture Recognition System and Design of a Human–Machine Interface

Deep Learning-Based Hand Gesture Recognition System and Design of a Human–Machine Interface

Recent Developments and Future Directions of Wearable Skin Biosignal Sensors

AbstractThis review article explores the transformative advancements in wearable biosignal sensors powered by machine learning, focusing on four notable biosignals: electrocardiogram (ECG), electromyogram (EMG), electroencephalogram (EEG), and photoplethysmogram (PPG). The integration of machine learning with these biosignals has led to remarkable breakthroughs in various medical monitoring and human–machine interface applications. For ECG, machine learning enables automated heartbeat classification and accurate disease detection, improving cardiac healthcare with early diagnosis and personalized interventions. EMG technology, combined with machine learning, facilitates real‐time prediction and classification of human motions, revolutionizing applications in sports medicine, rehabilitation, prosthetics, and virtual reality interfaces. EEG analysis powered by machine learning goes beyond traditional clinical applications, enabling brain activity understanding in psychology, neurology, and human–computer interaction, and holds promise in brain–computer interfaces. PPG, augmented with machine learning, has shown exceptional progress in diagnosing and monitoring cardiovascular and respiratory disorders, offering non‐invasive and accurate healthcare solutions. These integrated technologies, powered by machine learning, open new avenues for medical monitoring and human–machine interaction, shaping the future of healthcare.

Distinct neural representations during a brain–machine interface and manual reaching task in motor cortex, prefrontal cortex, and striatum

Although brain–machine interfaces (BMIs) are directly controlled by the modulation of a select local population of neurons, distributed networks consisting of cortical and subcortical areas have been implicated in learning and maintaining control. Previous work in rodents has demonstrated the involvement of the striatum in BMI learning. However, the prefrontal cortex has been largely ignored when studying motor BMI control despite its role in action planning, action selection, and learning abstract tasks. Here, we compare local field potentials simultaneously recorded from primary motor cortex (M1), dorsolateral prefrontal cortex (DLPFC), and the caudate nucleus of the striatum (Cd) while nonhuman primates perform a two-dimensional, self-initiated, center-out task under BMI control and manual control. Our results demonstrate the presence of distinct neural representations for BMI and manual control in M1, DLPFC, and Cd. We find that neural activity from DLPFC and M1 best distinguishes control types at the go cue and target acquisition, respectively, while M1 best predicts target-direction at both task events. We also find effective connectivity from DLPFC → M1 throughout both control types and Cd → M1 during BMI control. These results suggest distributed network activity between M1, DLPFC, and Cd during BMI control that is similar yet distinct from manual control.

An Analysis of Factors Influencing Ice Management Performance in an Experimental Marine Simulator and Their Application to Decision Support System Design

Abstract Ice management is essential for maintaining the safety of offshore operations in Arctic regions. We present the combined results of three experiments conducted in a full-mission bridge simulator specially designed for ice management. From a quantitative analysis of the results, we infer the effect of three variables on performance: (1) experience, (2) training, and (3) Decision Support System (DSS). The results confirm that experience and training improve performance for untrained and inexperienced simulator participants. The DSS also improves performance, but with a smaller effect. Qualitative observations using vessel position heat-map diagrams and exit interviews suggested that novice participants using the DSS adopted expert strategies but carried out their tasks more slowly and with less precision. This has important consequences for the design of a future DSS used in training simulators or onboard ships. Potential improvements to the DSS design might include real-time feedback to the user, a redesign of the human–machine interface (HMI), and increasing user input and customization with a human factors focus.