Applications In Human-machine Interaction Research Articles

Prediction of state anxiety by machine learning applied to photoplethysmography data.

BackgroundAs the human behavior is influenced by both cognition and emotion, affective computing plays a central role in human-machine interaction. Algorithms for emotions recognition are usually based on behavioral analysis or on physiological measurements (e.g., heart rate, blood pressure). Among these physiological signals, pulse wave propagation in the circulatory tree can be assessed through photoplethysmography (PPG), a non-invasive optical technique. Since pulse wave characteristics are influenced by the cardiovascular status, which is affected by the autonomic nervous activity and hence by the psychophysiological state, PPG might encode information about emotional conditions. The capability of a multivariate data-driven approach to estimate state anxiety (SA) of healthy participants from PPG features acquired on the brachial and radial artery was investigated.MethodsThe machine learning method was based on General Linear Model and supervised learning. PPG was measured employing a custom-made system and SA of the participants was assessed through the State-Trait Anxiety Inventory (STAI-Y) test.ResultsA leave-one-out cross-validation framework showed a good correlation between STAI-Y score and the SA predicted by the machine learning algorithm (r = 0.81; p = 1.87∙10−9). The preliminary results suggested that PPG can be a promising tool for emotions recognition, convenient for human-machine interaction applications.

A multilevel fusion network for 3D object detection

3D object detection is an important yet challenging problem in a myriad of vision, robotics, and human–machine interaction applications. Given an RGB-D image, the task is to infer the class labels and the 3D bounding boxes of the objects in the image. While the previous studies have made remarkable progress over the past decade, how to effectively exploit the feature fusion with neural networks for boosting 3D object detection performance remains an open problem. This paper proposes a multilevel fusion network (MFN) model to detect 3D objects in RGB-D images. The MFN model contains two streams of neural networks which respectively extracts the RGB and depth features with cascaded convolutional modules. To effectively exploit the information of 3D objects, a multilevel fusion mechanism is adopted to fuse the convolutional RGB and depth features at multiple levels. To train the network, we propose a new weighted loss function by encoding the difference of geometric attributes on 3D bounding box regression. Since the original depth data is full of noisy holes, we also develop an adaptive filtering algorithm to restore and correct the depth images. We test the proposed model on challenging RGB-D datasets. The experimental results on the datasets prove the strength and advantage of the proposed model.

Analysis of Momentous Fragmentary Formants in Talaqi-like Neoteric Assessment of Quran Recitation using MFCC Miniature Features of Quranic Syllables

The use of technological speech recognition systems with a variety of approaches and techniques has grown rapidly in a variety of human-machine interaction applications. Further to this, a computerized assessment system to identify errors in reading the Qur'an can be developed to practice the advantages of technology that exist today. Based on Quranic syllable utterances, which contain Tajweed rules that generally consist of Makhraj (articulation process), Sifaat (letter features or pronunciation) and Harakat (pronunciation extension), this paper attempts to present the technological capabilities in realizing Quranic recitation assessment. The transformation of the digital signal of the Quranic voice with the identification of reading errors (based on the Law of Tajweed) is the main focus of this paper. This involves many stages in the process related to the representation of Quranic syllable-based Recitation Speech Signal (QRSS), feature extraction, non-phonetic transcription Quranic Recitation Acoustic Model (QRAM), and threshold classification processes. MFCC-Formants are used in a miniature state that are hybridized with three bands in representing QRSS combined vowels and consonants. A human-guided threshold classification approach is used to assess recitation based on Quranic syllables and threshold classification performance for the low, medium, and high band groups with performances of 87.27%, 86.86%and 86.33%, respectively.

Pollen-Shaped Hierarchical Structure for Pressure Sensors with High Sensitivity in an Ultrabroad Linear Response Range

High sensitivity and linear response over a wide sensing range are important in flexible pressure sensors for their practical applications in biomimetic electronics and human-machine interactions. Previous studies regarding flexible pressure sensors have primarily focused on their high sensitivity, whereas they generally exhibit a narrow linear sensing range. In this article, a hierarchical structure with conical secondary features is reported, and its role in enhancing the linear sensing range of piezoresistive pressure sensors is demonstrated. We find that the conical secondary features on the hierarchical structure significantly improve the linear relationship between the contact area and applied force over a broad range. This advantage endows the sensor with a wide linear sensing range. To obtain this type of hierarchical structure, pollen grains of wild chrysanthemum are exploited as templates, and the prepared sensor presents a high sensitivity of 3.5 kPa-1 over an ultrawide response range of 0-218 kPa with good linearity via a coefficient of determination (R2) of 0.997. Furthermore, owing to the simple and scalable process, a sensor array with high density is fabricated to map the spatial pressure distribution and simulate an electronic skin to detect Braille characteristics.

Wirelessly powered multi-functional wearable humidity sensor based on RGO-WS2 heterojunctions

High-performance multifunctional sensors that are powered wirelessly could find applications in athlete training, health monitoring, and human-machine interactions. In this paper, a humidity sensor based on a reduced graphene oxide (RGO) and tungsten disulfide (WS2) heterojunction is proposed. The sensor exhibits high performance in regard to humidity sensing. The wireless humidity sensor has a fast response time (0.56 s) and recovery time (2.26 s), which can be attributed to the heterojunction and the synergistic effect. The sensor also has a wide working range between 5%RH to 95 %RH. The flexible humidity sensor can be used to monitor the skin humidity and evaluate the metabolism of the human body. Moreover, in situ Raman spectrum was used to prove the electron transfer between the water molecule and RGO-WS2 (RGWS) at different humidity. Thus, this work could have broad applications in various fields such as personal health, physical training, and psychological state monitoring.

A robust stretchable pressure sensor for electronic skins

As an essential role in electronic skins, flexible pressure sensors have attracted wide attention for their applications in human-machine interaction and intelligent prosthetic, etc. Whereas, in general, it is difficult for most flexible pressure sensors to acquire high accuracy, especially under multi-modal interference (temperature, humidity and deformation). Herein, a novel stretchable flexible pressure sensor was presented by direct writing, which was capable of weakening the strain effect, fending off humidity interference and restraining thermal influence by using a double layer raised island structure substrate, a covering layer and a Wheatstone bridge-configuration of the sensing element. The sensor kept robust under multi-modal interference of strain about 0–20%, temperature of 30–80 °C and humidity in 20–90%, owning high and stable sensitivity of 2.6 kPa −1 –14.05 kPa −1 under 0–74 kPa. Two demonstrations of grabbing a bottle with different temperatures and a kid's catching behavior on a opisthenar of a carer verified that such sensor possessed a great potential in many non-structured applications and will be a promising candidate for stretchable flexible pressure sensors in future. • A robust pressure sensor which can restrain the interference of deformation, temperature and humidity was presented. • A double layer raised island structure substrate was used to weaken the strain effect. • A Wheatstone bridge configuration of four same sensing elements was elaborated to eliminate the thermal impact. • Two applications including grabbing a bottle and a kid's catching behavior have been demonstrated.

NIR-responsive multi-healing HMPAM/dextran/AgNWs hydrogel sensor with recoverable mechanics and conductivity for human-machine interaction

Conductive and self-healing hydrogel sensor is perspective in human-machine interaction applications. However, the design of ideal self-healing hydrogels are always challenging. Herein, by introducing disulfide modified Ag nanowires (AgNWs), we show a novel self-healing hydrogel strain sensor with superior mechanics, conductivity, antibacterial property, and firstly realizing of self-healing with both recovery of mechanics and sensing properties. We demonstrate that the covalent and reversible non-covalent hydrophobic blocks in hydrophobic modified polyacrylamide (HMPAM) achieves the basic self-healing network; dextran with plentiful hydroxyl groups synergistic helps the self-healing by hydrogen bonds; disulfide on the AgNWs surface forms a NIR-responsive and dynamic Ag-S coordination bridge between HMPAM and AgNWs. The resulted hydrogel sensor exhibits comprehensive electromechanical properties, and precisely monitors human motion and subtle electromyography (EMG) signals. Importantly, we firstly achieved the recovery of sensing properties on human motion detection and EMG signal detection after self-healing. This work provides a promising exploration to manufacture bionic strain sensors for potential applications in wearable electronics.

Microstructured flexible capacitive sensor with high sensitivity based on carbon fiber-filled conductive silicon rubber

Flexible sensors with high sensitivity and stability have been tremendously attractive in recent years. In this study, flexible capacitive sensors (FCSs) with microstructured-surface were fabricated, where electrode layers (e-layers) and dielectric layers (d-layers) of the FCSs were sprayed with carbon fiber (CF) filled-polydimethylsiloxane (PDMS) and pure PDMS on substrate with surface roughness, respectively. Sensitivity of the fabricated FCSs to compressive load was evaluated. Influence of the surface microstructure of d- and e-layers on the sensitivity was studied through in-situ microscopic observations and theoretical capacitance calculations. Results showed that the roughness of the surface microstructure did have a correlation with the sensitivity. Excellent sensing performance, including a high sensitivity (∼0.82 kPa−1), ultrafast response (∼0.2 s) and relaxation (∼0.3 s) time, ultralow detection limit (∼1.2 Pa), and excellent stability (104 loading/unloading cycles) over a wide pressure range (0−50 kPa) appeared on the FCS with its surface microstructure size of 48.6 μm in roughness. The high sensitivity of the fabricated FCS was dominantly resulted from the closing of air gap between e- and d-layers. A 3 × 3 FCS array capable of distinguishing space pressure distribution and a mouse glove capable of inputting computer information were finally fabricated which demonstrated a prospect of the FCS in micro-load response and human-machine interaction applications.

Conformal Devices for Thermal Sensing and Heating in Biomedical and Human–Machine Interaction Applications

Thermal ablation has been adopted as one of the most common cancer treatment approaches in medical surgery. By increasing the temperature (>50 °C) on the cells, the cells are destroyed because of denaturation. Herein, an ultrathin Archimedean spiral pattern heater/sensor technology is introduced which can perform ablation by attaching conformally onto the organs for precise heating and temperature sensing. In the heater mode, the heater temperature is linearly proportional to the input joule heating power up to 400 mW. In the sensor mode, the temperature of the conformal metal wire is also linearly related to the resistance by the temperature coefficient of resistance (TCR). The conformal heater to perform ex vivo ablation on the porcine liver is utilized. By further integrating the devices with robotic palm and perform heat‐and‐sense interactions, a human–machine interface (HMI) apparatus is demonstrated which can be potentially applied in surgical robots or other tactile stimulation systems.

Skin-conformal BaTiO3/ecoflex-based piezoelectric nanogenerator for self-powered human motion monitoring

Skin-conformability and lead-free piezo-materials are important for piezoelectric nanogenerator (PENG)-based wearable electronics. Here, a skin-conformal BaTiO3/Ecoflex-based PENG was fabricated by incorporating BaTiO3 particles into an Ecoflex matrix. Due to the inherent elongation of Ecoflex, appropriate thickness of the BTO&Ecoflex film, and optimized polarization, the fabricated device exhibited suitable skin-conformability and high output performance. The output voltage/current reached 8.9 V/49.7 nA with a load resistance of 200 MΩ (effective area: 1.0 cm × 1.0 cm). Due to its suitable skin-conformability and high output performance, the device can be attached to an elbow as an active sensor for the monitoring of joints. In this way, it is promising for medical research and human-machine interaction (HMI) applications. This study is significant for the development of skin-conformal lead-free PENGs.

Ultrahigh-Sensitive Finlike Double-Sided E-Skin for Force Direction Detection.

Flexible pressure sensing is required for the excellent sensing performance and dexterous manipulation of the measured objects in their potential applications. Particularly, the ability to measure and discriminate the direction of force, contact surface, and contact location in real time is crucial for robotics with tactile feedback. Herein, a three-dimensional elastic porous carbon nanotube (CNT) sponge is synthesized by chemical vapor deposition, which is successfully applied in the piezoresistive sensor. In situ scanning electron microscopy study intuitively illustrates the characteristics that the microfibers of the CNT sponge distort and contact with each other under an external force. As a result, new conductive paths are created at the contact points between the CNT microfibers, which provides a basic sensing principle for a piezoresistive sensor. The CNT sponge-based sensor has an ultrahigh sensitivity in a wide pressure range (0-4 kPa for 4015.8 kPa-1), a rapid response time of 120 ms, and excellent durability over 5000 cycles. Moreover, a finlike flexible double-sided electronic skin (e-skin) is fabricated by a simple method to achieve force direction detection, which has potential applications in intelligent wearable devices and human-machine interaction.

A Marked Point Process Filtering Approach for Tracking Sympathetic Arousal From Skin Conductance

Human emotion represents a complex neural process within the brain. The ability to automatically recognize emotions from physiological signals has the potential to impact humanity in multiple ways through applications in human-machine interaction, remote health monitoring, smart living environments and entertainment. We present a marked point process-based Bayesian filtering approach to track sympathetic arousal from skin conductance features. The rate at which individual skin conductance responses (SCRs) occur and their respective amplitudes encode important information regarding an individual's psychological arousal level. We develop a state-space model relating a latent neuropsychological arousal state to the rate at which neural impulses underlying SCRs occur and the impulse amplitudes. We simultaneously estimate both arousal and the state-space model parameters within an expectation-maximization framework. We evaluate our method on both simulated and experimental data. Results on simulated data indicate the method's ability to accurately estimate an unobserved state from marked point process observations. The experimental data include studies involving mental stress artificially-induced in laboratory environments, real-world driver stress and Pavlovian fear conditioning. Results on experimental data outperform previous Bayesian filtering approaches in terms of a lower sensitivity to small impulses and avoiding overfitting. The filter is thus able to estimate emotional arousal from skin conductance features and is a promising approach for everyday emotion recognition applications.

Piezotronic and piezo-phototronic devices based on the third generation semiconductors

<p indent=0mm>Compared with the first-generation semiconductors (such as silicon, germanium) and the second-generation semiconductors (such as gallium arsenide, indium antimonide), the third-generation semiconductor materials represented by silicon carbide (SiC), zinc oxide (ZnO), gallium nitride (GaN) and cadmium sulfide (CdS) usually possess wider band gap, higher thermal conductivity, bigger electron saturation rate and better radiation resistance properties, and thus draw intensive attentions in high temperature and high frequency applications in recent years. Most of the third-generation semiconductors are wurtzite structures, which have piezoelectric effects due to their lack of symmetry in certain directions. This feature serves as a good bridge of transferring mechanical stress signals between the flexible semiconductor electronic devices and the surrounding environment or the host (e.g., the human body). Conventional piezoelectric effects are mainly found in barium titanate and lead zirconate titanate type perovskite materials, but such materials do not have semiconductor properties, thus limiting their use in electronics and optoelectronic devices. Our group has pioneered a new research field, i.e., piezoelectric nanogenerator, by utilizing third-generation semiconductor nanowires (such as ZnO nanowires) under a dynamic force. The piezoelectric potential generated by the dynamic strain on the nanowire can drive electrons to flow in the external load circuit, which is the basic principle of the piezoelectric nanogenerator. Piezoelectric nanogenerator based on ZnO nanowires was firstly proposed in 2006. When a uniform strain is applied on a non-centrosymmetric semiconductor, a piezo-potential will be induced in the semiconductor, accompanied with static piezoelectric charges distributed on the surfaces. This piezoelectric effect is commonly existed in the third generation semiconductors. The strain-induced piezo-potential and piezoelectric polarization charges can also statically and/or dynamically tune the transportation and photoelectric processes of carriers at the interfaces or junctions. The three ways coupling among piezoelectric, photoexcitation and semiconducting properties coined two new research fields, that is, the piezotronics and piezo-phototronics. In order to systematically explain the coupling properties of piezoelectric and semiconductor transport properties in such materials, our group first coined the term piezotronics in a paper published in 2007. The piezotronics is about the modulation of the metal-semiconductor interface barrier via piezoelectric potential acting as a “gate” voltage, which has been used to explain the transistor behavior observed in metal-ZnO-metal structures and the strain-gated diode effect of metal-ZnO structures. The basic core of piezo-phototronics is to regulate the generation, separation, transport and/or recombination of photogenerated carriers at interfaces or heterojunctions by piezoelectric polarization charges, which was first proposed by our group in 2010. The performance of many optoelectronic devices can be effectively enhanced by piezo-phototronic effect. Over the past decade, these two areas have received extensive attentions and made great progresses in basic science and device applications. The combinations of piezotronics and piezo-phototronics with the current popular electronics, optoelectronics and spintronics have great significances, which will bring in revolutionized impacts for future sensor networks, artificial intelligence, micro-nano energy and human-machine interaction applications. This paper gives a brief review of the experimental progress in device application in these two fields in recent years, and looks forward to the future development of these two subjects.

Soft dry electroophthalmogram electrodes for human machine interaction.

Current soft surface electrodes have attracted more and more attentions owing to their potential applications in biological signal monitoring, human-machine interaction (HMI) and Internet of Things (IoT). The paper presents that soft dry electrode based on polydimethylsiloxane-carbon black (PDMS-CB) conductive polymer is designed and fabricated to continuous, long-term, stable electroophthalmogram (EOG) signal recordings for HMI applications. The features corresponding to the different eye motions are extracted from the EOG data via the soft dry electrodes. Linear discriminant analysis (LDA) recognition algorithms are proposed to recognize eye motion behaviors for controlling the motion of the mobile robots. Experiment results have been demonstrated that LDA recognition algorithm achieves a relatively high recognition accuracy of 90.63% for recognizing four eye movements ('Up', 'Down', 'Right', and 'Left'). The control commands are generated with different eye motions and transmitted to the mobile robot through WiFi communication unit, which the mobile robot is successfully controlled. The soft dry electrodes have the potential in a comfortable, simple, wearable and wireless control of rehabilitation devices.

Application and Prospect of Human-Machine Interaction in Power System

Poser system provide many scenarios the innovation of human-machine interaction technology. Four advanced human-machine interaction technologies in power system are introduced and the necessity of human-machine interaction in power system are summarized from four aspects. Development Course of Human-Machine Interaction in Power System are divided into five generations based on technical characteristics. Four technological trends which should be focused on are given.

A simple and cost-effective method for improving the sensitivity of flexible strain sensors based on conductive polymer composites

Conductive polymer composites (CPCs) based flexible strain sensors requires high performance in terms of sensitivity and stretchability towards potential applications in health monitoring, motion detection and human-machine interaction. In this work, a simple and cost-effective method was proposed to improve the sensitivity yet maintain the stretchability of CPCs based strain sensor. Compared with conventional multi-wall carbon nanotubes/polydimethylsiloxane (MWCNTs/PDMS) flexible strain sensors, silicone fluid was introduced to configure the conductive networks as well as the mechanical properties. It was found that silicone fluid can increase the sensitivity of strain sensor by 10–30 times without reducing the stretchability, which was attributed to the non-covalently functionalized surface of the carbon nanotubes leading to higher ratio of the tunneling resistance to overall resistance of CPCs based flexible strain sensor. Furthermore, the synergetic effect between silicone fluid and CNTs was investigated through conductive network reconfiguration and electromechanical test. The proposed efficient and effective flexible strain sensor preparation approach provides useful insights for high-performance flexible strain sensor design and application.

Ultralow-Cost, Highly Sensitive, and Flexible Pressure Sensors Based on Carbon Black and Airlaid Paper for Wearable Electronics.

Flexible pressure sensors have attracted considerable attention because of their potential applications in healthcare monitoring and human-machine interactions. However, the complicated fabrication process and the cos of sensing materials limit their widespread applications in practice. Herein, a flexible pressure sensor with outstanding performances is presented through an extremely simple and cost-efficient fabrication process. The sensing materials of the sensor are based on low-cost carbon black (CB)@airlaid paper (AP) composites, which are just prepared by drop-casting CB solutions onto APs. Through simply stacking multiple CB@APs with an irregular surface and a fiber-network structure, the obtained pressure sensor demonstrates an ultrahigh sensitivity of 51.23 kPa-1 and an ultralow detection limit of 1 Pa. Additionally, the sensor exhibits fast response time, wide working range, good stability, as well as excellent flexibility and biocompatibility. All the comprehensive and superior performances endow the sensor with abilities to precisely detect weak air flow, wrist pulse, phonation, and wrist bending in real time. In addition, an array electronic skin integrated with multiple CB@AP sensors has been designed to identify spatial pressure distribution and pressure magnitude. Through a biomimetic structure inspired by blooming flowers, a sensor with the open-petal structure has been designed to recognize the wind direction. Therefore, our study, which demonstrates a flexible pressure sensor with low cost, simple preparation, and superior performances, will open up for the exploration of cost-efficient pressure sensors in wearable devices.

Multifunctional Electronic Textiles Using Silver Nanowire Composites.

Textiles represent an appealing platform for continuous wearable applications due to the exceptional combination of compliance, water vapor permeability, and comfortableness for long-term wear. We present mechanically and electrically robust integration of nanocomposites with textiles by laser scribing and heat press lamination. The simple and scalable integration technique enables multifunctional E-textiles without compromising the stretchability, wearability, and washability of textiles. The textile-integrated patterns exhibit small line width (135 μm), low sheet resistance (0.2 Ω/sq), low Young's modulus, good washability, and good electromechanical performance up to 50% strain, which is desirable for wearable and user-friendly electronic textiles. To demonstrate the potential utility, we developed an integrated textile patch comprising four dry electrophysiological electrodes, a capacitive strain sensor, and a wireless heater for electrophysiological monitoring, motion tracking, and thermotherapy, respectively. Beyond the applications demonstrated in this paper, the materials and methods presented here pave the way for various other wearable applications in health care, activity tracking, rehabilitation, sports medicine,and human-machine interactions.

Flexible Smart Noncontact Control Systems with Ultrasensitive Humidity Sensors.

The development of noncontact humidity sensors with high sensitivity, rapid response, and a facile fabrication process is urgently desired for advanced noncontact human-machine interaction (HMI) applications. Here, a flexible and transparent humidity sensor based on MoO3 nanosheets is developed with a low-cost and easily manufactured process. The designed humidity sensor exhibits ultrahigh sensitivity, fast response, great stability, and high selectivity, exceeding the state-of-the-art humidity sensors. Furthermore, a wearable moisture analysis system is assembled for real-time monitoring of ambient humidity and human breathing states. Benefiting from the sensitive and rapid response to fingertip humidity, the sensors are successfully applied to both a smart noncontact multistage switch and a novel flexible transparent noncontact screen for smart mobile devices, demonstrating the potential of the MoO3 nanosheets-based humidity sensors in future HMI systems.

Laser-Processed Nature-Inspired Deformable Structures for Breathable and Reusable Electrophysiological Sensors toward Controllable Home Electronic Appliances and Psychophysiological Stress Monitoring.

Physiological monitoring through skin patch stretchable devices has received extensive attention because of their significant findings in many human-machine interaction applications. In this paper, we present novel nature-inspired, kiri-spider, serpentine structural designs to sustain mechanical deformations under complex stress environments. Strain-free mechanical structures involving stable high areal coverage (spiderweb), three-dimensional out-of-plane deformations (kirigami), and two-dimensional (2D) stretchable (2D spring) electrodes demonstrated high levels of mechanical loading under various strains, which were verified through theoretical and experimental studies. Alternative to conventional microfabrication procedures, sensors fabricated by a facile and rapid benchtop programmable laser machine enabled the realization of low-cost, high-throughput manufacture, followed by transferring procedures with a nearly 100% yield. For the first time, we demonstrated laser-processed thin (∼10 μm) flexible filamentary patterns embedded within the solution-processed polyimide to make it compatible with current flexible printed circuit board electronics. A patch-based sensor with thin, breathable, and sticky nature exhibited remarkable water permeability >20 g h-1 m-2 at a thickness of 250 μm. Moreover, the reusability of the sensor patch demonstrated the significance of our patch-based electrophysiological sensor. Furthermore, this wearable sensor was successfully implemented to control human-machine interfaces to operate home electronic appliances and monitor mental stress in a pilot study. These advances in novel mechanical architectures with good sensing performances provide new opportunities in wearable smart sensors.