Applications In Human-machine Interaction Research Articles

Application and Prospect of Human-Machine Interaction in Power System

Poser system provide many scenarios the innovation of human-machine interaction technology. Four advanced human-machine interaction technologies in power system are introduced and the necessity of human-machine interaction in power system are summarized from four aspects. Development Course of Human-Machine Interaction in Power System are divided into five generations based on technical characteristics. Four technological trends which should be focused on are given.

A simple and cost-effective method for improving the sensitivity of flexible strain sensors based on conductive polymer composites

Conductive polymer composites (CPCs) based flexible strain sensors requires high performance in terms of sensitivity and stretchability towards potential applications in health monitoring, motion detection and human-machine interaction. In this work, a simple and cost-effective method was proposed to improve the sensitivity yet maintain the stretchability of CPCs based strain sensor. Compared with conventional multi-wall carbon nanotubes/polydimethylsiloxane (MWCNTs/PDMS) flexible strain sensors, silicone fluid was introduced to configure the conductive networks as well as the mechanical properties. It was found that silicone fluid can increase the sensitivity of strain sensor by 10–30 times without reducing the stretchability, which was attributed to the non-covalently functionalized surface of the carbon nanotubes leading to higher ratio of the tunneling resistance to overall resistance of CPCs based flexible strain sensor. Furthermore, the synergetic effect between silicone fluid and CNTs was investigated through conductive network reconfiguration and electromechanical test. The proposed efficient and effective flexible strain sensor preparation approach provides useful insights for high-performance flexible strain sensor design and application.

Ultralow-Cost, Highly Sensitive, and Flexible Pressure Sensors Based on Carbon Black and Airlaid Paper for Wearable Electronics.

Flexible pressure sensors have attracted considerable attention because of their potential applications in healthcare monitoring and human-machine interactions. However, the complicated fabrication process and the cos of sensing materials limit their widespread applications in practice. Herein, a flexible pressure sensor with outstanding performances is presented through an extremely simple and cost-efficient fabrication process. The sensing materials of the sensor are based on low-cost carbon black (CB)@airlaid paper (AP) composites, which are just prepared by drop-casting CB solutions onto APs. Through simply stacking multiple CB@APs with an irregular surface and a fiber-network structure, the obtained pressure sensor demonstrates an ultrahigh sensitivity of 51.23 kPa-1 and an ultralow detection limit of 1 Pa. Additionally, the sensor exhibits fast response time, wide working range, good stability, as well as excellent flexibility and biocompatibility. All the comprehensive and superior performances endow the sensor with abilities to precisely detect weak air flow, wrist pulse, phonation, and wrist bending in real time. In addition, an array electronic skin integrated with multiple CB@AP sensors has been designed to identify spatial pressure distribution and pressure magnitude. Through a biomimetic structure inspired by blooming flowers, a sensor with the open-petal structure has been designed to recognize the wind direction. Therefore, our study, which demonstrates a flexible pressure sensor with low cost, simple preparation, and superior performances, will open up for the exploration of cost-efficient pressure sensors in wearable devices.

Multifunctional Electronic Textiles Using Silver Nanowire Composites.

Textiles represent an appealing platform for continuous wearable applications due to the exceptional combination of compliance, water vapor permeability, and comfortableness for long-term wear. We present mechanically and electrically robust integration of nanocomposites with textiles by laser scribing and heat press lamination. The simple and scalable integration technique enables multifunctional E-textiles without compromising the stretchability, wearability, and washability of textiles. The textile-integrated patterns exhibit small line width (135 μm), low sheet resistance (0.2 Ω/sq), low Young's modulus, good washability, and good electromechanical performance up to 50% strain, which is desirable for wearable and user-friendly electronic textiles. To demonstrate the potential utility, we developed an integrated textile patch comprising four dry electrophysiological electrodes, a capacitive strain sensor, and a wireless heater for electrophysiological monitoring, motion tracking, and thermotherapy, respectively. Beyond the applications demonstrated in this paper, the materials and methods presented here pave the way for various other wearable applications in health care, activity tracking, rehabilitation, sports medicine,and human-machine interactions.

Flexible Smart Noncontact Control Systems with Ultrasensitive Humidity Sensors.

The development of noncontact humidity sensors with high sensitivity, rapid response, and a facile fabrication process is urgently desired for advanced noncontact human-machine interaction (HMI) applications. Here, a flexible and transparent humidity sensor based on MoO3 nanosheets is developed with a low-cost and easily manufactured process. The designed humidity sensor exhibits ultrahigh sensitivity, fast response, great stability, and high selectivity, exceeding the state-of-the-art humidity sensors. Furthermore, a wearable moisture analysis system is assembled for real-time monitoring of ambient humidity and human breathing states. Benefiting from the sensitive and rapid response to fingertip humidity, the sensors are successfully applied to both a smart noncontact multistage switch and a novel flexible transparent noncontact screen for smart mobile devices, demonstrating the potential of the MoO3 nanosheets-based humidity sensors in future HMI systems.

Laser-Processed Nature-Inspired Deformable Structures for Breathable and Reusable Electrophysiological Sensors toward Controllable Home Electronic Appliances and Psychophysiological Stress Monitoring.

Physiological monitoring through skin patch stretchable devices has received extensive attention because of their significant findings in many human-machine interaction applications. In this paper, we present novel nature-inspired, kiri-spider, serpentine structural designs to sustain mechanical deformations under complex stress environments. Strain-free mechanical structures involving stable high areal coverage (spiderweb), three-dimensional out-of-plane deformations (kirigami), and two-dimensional (2D) stretchable (2D spring) electrodes demonstrated high levels of mechanical loading under various strains, which were verified through theoretical and experimental studies. Alternative to conventional microfabrication procedures, sensors fabricated by a facile and rapid benchtop programmable laser machine enabled the realization of low-cost, high-throughput manufacture, followed by transferring procedures with a nearly 100% yield. For the first time, we demonstrated laser-processed thin (∼10 μm) flexible filamentary patterns embedded within the solution-processed polyimide to make it compatible with current flexible printed circuit board electronics. A patch-based sensor with thin, breathable, and sticky nature exhibited remarkable water permeability >20 g h-1 m-2 at a thickness of 250 μm. Moreover, the reusability of the sensor patch demonstrated the significance of our patch-based electrophysiological sensor. Furthermore, this wearable sensor was successfully implemented to control human-machine interfaces to operate home electronic appliances and monitor mental stress in a pilot study. These advances in novel mechanical architectures with good sensing performances provide new opportunities in wearable smart sensors.

A Soft Three-Axis Force Sensor Based on Radially Symmetric Pneumatic Chambers

In physical human–machine interaction applications, the accurate measurement of interactive forces between the human and the machine plays a significant role. Sensors for this purpose should not only be accurate and reliable, but also be soft enough to guarantee the safe and compliant interaction. In this aspect, pneumatic sensors with soft air chambers have often been utilized as a soft force sensor system. Although the pneumatic-based force sensor provides a good compliance and softness, however, it measures only a lumped force acting on the chamber, because the pneumatic sensor measures the overall pressure change in the air chamber. In this paper, a three-axis force sensor with three air chambers in a radially symmetric pattern is proposed for the measurement of multi-dimensional interaction forces with high softness and compliance. Each air chamber embeds a pneumatic sensor, and the pressure changes in the three air chambers are measured in order to distinguish the direction of the applied force. By decoupling the sensor signals from the three pneumatic sensors, the three-dimensional force components can be calculated accurately. Consequently, the proposed sensor is able to measure the three-dimensional forces while maintaining the great softness and compliance provided by the soft air chambers. The design, the fabrication method, and the verification of the proposed method are introduced in this paper.

Surface electromyography feature extraction via convolutional neural network

Although a large number of surface electromyography (sEMG) features have been proposed to improve hand gesture recognition accuracy, it is still hard to achieve acceptable performance in inter-session and inter-subject tests. To promote the application of sEMG-based human machine interaction, a convolutional neural network based feature extraction approach (CNNFeat) is proposed to improve hand gesture recognition accuracy. A sEMG database is recorded from eight subjects while performing ten hand gestures. Three classic classifiers, including linear discriminant analysis (LDA), support vector machine (SVM) and K nearest neighbor (KNN), are employed to compare the CNNFeat with 25 traditional features. This work concentrates on the analysis of CNNFeat through accuracy, safety index and repeatability index. The experimental results show that CNNFeat outperforms all the tested traditional features in inter-subject test and is listed as the best three features in inter-session test. Besides, it is also found that combining CNNFeat with traditional features can further improve the accuracy by 4.35%, 3.62% and 4.7% for SVM, LDA and KNN, respectively. Additionally, this work also demonstrates that CNNFeat can be potentially enhanced with more data for model training.

Time Derivative Moments Based Feature Extraction Approach for Recognition of Upper Limb Motions Using EMG

Electromyography pattern recognition (EMG-PR) is extensively recognized in human–machine interactive applications such as prosthesis control and rehabilitation devices. Conventional time-domain (TD) features have been shown to produce a decent performance in upper limb movements’ classification. However, performance limitation exists in terms of classification accuracy. Hence, a novel feature set extraction on time derivative moments is proposed to improve the performance of EMG-PR in upper limb motion classification. A standardized and benchmark data base NinaPro (subdatabase2) has been used for examination of the proposed feature set. A total of eight intact subjects (sub1–sub8) have been selected for eight grasping motion's EMG signal collection. For classification of features, three classification techniques (linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), and support vector machine (SVM)) have been applied. The average improvisation with respect to the conventional TD features that was achieved for LDA, QDA, and SVM are 8.08%, 7.95%, and 6.41%, respectively.

Learning Facial Expressions with 3D Mesh Convolutional Neural Network

Making machines understand human expressions enables various useful applications in human-machine interaction. In this article, we present a novel facial expression recognition approach with 3D Mesh Convolutional Neural Networks (3DMCNN) and a visual analytics-guided 3DMCNN design and optimization scheme. From an RGBD camera, we first reconstruct a 3D face model of a subject with facial expressions and then compute the geometric properties of the surface. Instead of using regular Convolutional Neural Networks (CNNs) to learn intensities of the facial images, we convolve the geometric properties on the surface of the 3D model using 3DMCNN. We design a geodesic distance-based convolution method to overcome the difficulties raised from the irregular sampling of the face surface mesh. We further present interactive visual analytics for the purpose of designing and modifying the networks to analyze the learned features and cluster similar nodes in 3DMCNN. By removing low-activity nodes in the network, the performance of the network is greatly improved. We compare our method with the regular CNN-based method by interactively visualizing each layer of the networks and analyze the effectiveness of our method by studying representative cases. Testing on public datasets, our method achieves a higher recognition accuracy than traditional image-based CNN and other 3D CNNs. The proposed framework, including 3DMCNN and interactive visual analytics of the CNN, can be extended to other applications.

A Flexible Wearable Pressure Sensor with Bioinspired Microcrack and Interlocking for Full-Range Human-Machine Interfacing.

Flexible wearable pressure sensors have drawn tremendous interest for various applications in wearable healthcare monitoring, disease diagnostics, and human-machine interaction. However, the limited sensing range (<10%), low sensing sensitivity at small strains, limited mechanical stability at high strains, and complicated fabrication process restrict the extensive applications of these sensors for ultrasensitive full-range healthcare monitoring. Herein, a flexible wearable pressure sensor is presented with a hierarchically microstructured framework combining microcrack and interlocking, bioinspired by the crack-shaped mechanosensory systems of spiders and the wing-locking sensing systems of beetles. The sensor exhibits wide full-range healthcare monitoring under strain deformations of 0.2-80%, fast response/recovery time (22 ms/20 ms), high sensitivity, the ultrasensitive loading sensing of a feather (25 mg), the potential to predict the health of patients with early-stage Parkinson's disease with the imitated static tremor, and excellent reproducibility over 10 000 cycles. Meanwhile, the sensor can be assembled as smart artificial electronic skins (E-skins) for simultaneously mapping the pressure distribution and shape of touching sensing. Furthermore, it can be attached onto the legs of a smart robot and coupled to a wireless transmitter for wirelessly monitoring human-motion interactivities.

Large-Scale Fabrication of High-Performance Ionic Polymer-Metal Composite Flexible Sensors by in Situ Plasma Etching and Magnetron Sputtering.

Flexible electronics has received widespread concern and research. As a most-fundamental step and component, polymer metallization to introduce conductive electrode is crucial in successful establishment and application of flexible and stretchable electronic system. Ionic polymer–metal composite (IPMC) is such an attractive flexible mechanical sensor with significant advantages of passive and space-discriminative capability. Generally, the IPMC sensor is fabricated by the electroless plating method to form structure of ionic polymer membrane sandwiched with two metallic electrodes. In order to obtain high-quality interface adhesion and conductivity between polymer and metal, the plating process for IPMC sensor is usually time-consuming and uncontrollable and has low reproducibility, which make it difficult to use in practice and in large-scale. Here, a manufacturable method and equipment with short processing time and high reproducibility for fabricating IPMC sensors by in situ plasma etching and magnetron sputtering depositing on flexible substrates is developed. First, the new method shortens the fabrication period greatly from 2 weeks to 2 h to obtain IPMC sensors with sizes up to 9 cm × 9 cm or arrays in various patterns. Second, the integrated operation ensures all sample batch stability and performance repeatability. In a typical IPMC sensor, nearly 200 mV potential signal due to ion redistribution induced by bending strain under 1.6% can be produced without any external power supply, which is much higher than the traditional electroless plating sensor. This work verified that the in situ plasma etching and magnetron sputtering deposition could significantly increase the interface and surface conductivity of the flexible devices, resulting in the present high sensitivity as well as linear correlation with strain of the IPMC sensor. Therefore, this introduced method is scalable and believed to be used to metalize flexible substrates with different metals, providing a new route to large-scale fabrication of flexible devices for potential wearable applications in real-time monitoring human motion and human–machine interaction.

AIEM: AI-enabled affective experience management

Nowadays, with rapid development of artificial intelligence technology, the emerging human–machine interaction application researches grow up with machine intelligence, cognitive science and CEM (Customer Experience Management). This paper puts forward a new AIEM (AI-enabled affective experience management) method, blends AI and CEM in the emotion recognition and interactive intelligence application. Besides, in order to create good user experience, AIEM method also strives for the intelligence at various phases of emotion acquisition, emotion recognition, and emotion interaction. This paper introduces the composition and architecture of AIEM from three aspects, i.e. intelligent management of emotion data collection, accuracy management of emotion recognition, and real-time management of emotion interaction. Then we use advanced algorithm and model in two phases of emotion recognition algorithm and emotion computing offloading. Moreover, we select two deep learning algorithms (VGG-Net and Alex-Net) for facial expression recognition and speech emotion recognition, respectively. In the experiment using AIWAC system in real environment, we evaluate the emotion interaction delay in different computing nodes (Cloud and Edge) using AIEM method. Experiment results show that our method can provide intuitive and reasonable user experience management, and select suitable computing nodes for users. Finally, we provide summary and prospect for the future research proposal.

The application of functional oxide thin films in flexible sensor devices

With promising applications in healthcare and human-machine interaction, flexible sensor devices have drawn much attention and have been intensively studied in recent years. Meanwhile, functional oxide thin films, as one key family of inorganic functional materials, have been widely used in the electronic and optoelectronic devices because of their rich electronic, optical, thermal, and mechanical properties. Their application in flexible sensor devices is expected to pave the way for high-performance devices and the design of novel devices. In this review, the key scientific and technical issues that need to be solved to promote the application of functional oxide thin films in flexible sensor devices are summarized and recent research progress in this field is discussed.

3D hybrid porous Mxene-sponge network and its application in piezoresistive sensor

High-performance pressure sensors attract widespread attention due to the potential applications in human-machine interaction and wearable electronics. However, most of flexible pressure sensors need elaborate nanostructure design, and thus their manufacture is consuming and complicated. Therefore, a large-scale and low-cost technology is highly desirable to fabricate flexible pressure sensitive materials with high sensitivity in broad pressure range. Here, an MXene-sponge was fabricated by a simple and efficient dipping-coating process method and was applied in the piezoresistive sensor with insulating Polyvinyl Alcohol (PVA) nanowires as a spacer. The sensor based on MXene-sponge presents high sensitivity for a broad pressure range (147 kPa−1 for less than 5.37 kPa region and 442 kPa−1 for 5.37–18.56 kPa region), a low detection limit of 9 Pa, a rapid response time of 138 ms, and an excellent durability over 10,000 cycles. Human physiological signal (such as respiration, joint movement and pulses) can be real-time monitored by the MXene-sponge sensor. The potential applications of the sensor in measuring pressure distribution and human-machine interaction are also explored.

Oxygen-assisted preparation of mechanoluminescent ZnS:Mn for dynamic pressure mapping

Mechanoluminescent materials that convert mechanical stimuli to light emission have attracted extensive attention for potential applications in human-machine interactions. Here, we report a simple and available novel approach for the oxygen-assisted preparation of ZnS:Mn particles by solid-state reaction at atmospheric pressure without the formation of the corresponding oxides. The existence of O2 has a positive impact on the formation of S vacancies in wurtzite-phase ZnS, leading to the introduction of Mn2+ ion luminescent centers and shallow donor levels, which can improve the electron-hole recombination rate. The O2 ratio and Mn2+ ion doping concentration have significant effects on the luminous efficiency, which is optimal at 1%–20% and 1 at.%–2 at.% respectively. In addition, a device based on the piezo-photonic effect with excellent pressure sensitivity of 0.032 MPa−1 was fabricated, which can map the two-dimensional pressure distribution ranging from 2.2 to 40.6 MPa in situ. This device can be applied to real-time pressure mapping, smart sensor networks, high-level security systems, human-machine interfaces, and artificial skins.

Stretchable Triboelectric-Photonic Smart Skin for Tactile and Gesture Sensing.

Smart skin is expected to be stretchable and tactile for bionic robots as the medium with the ambient environment. Here, a stretchable triboelectric-photonic smart skin (STPS) is reported that enables multidimensional tactile and gesture sensing for a robotic hand. With a grating-structured metal film as the bioinspired skin stripe, the STPS exhibits a tunable aggregation-induced emission in a lateral tensile range of 0-160%. Moreover, the STPS can be used as a triboelectric nanogenerator for vertical pressure sensing with a maximum sensitivity of 34 mV Pa-1 . The pressure sensing characteristics can remain stable in different stretching conditions, which demonstrates a synchronous and independent sensing property for external stimuli with great durability. By integrating on a robotic hand as a conformal covering, the STPS shows multidimensional mechanical sensing abilities for external touch and different gestures with joints bending. This work has first demonstrated a triboelectric-photonic coupled multifunctional sensing terminal, which may have great applications in human-machine interaction, soft robots, and artificial intelligence.

A Highly Stretchable Transparent Self-Powered Triboelectric Tactile Sensor with Metallized Nanofibers for Wearable Electronics.

Recently, the quest for new highly stretchable transparent tactile sensors with large-scale integration and rapid response time continues to be a great impetus to research efforts to expand the promising applications in human-machine interactions, artificial electronic skins, and smart wearable equipment. Here, a self-powered, highly stretchable, and transparent triboelectric tactile sensor with patterned Ag-nanofiber electrodes for detecting and spatially mapping trajectory profiles is reported. The Ag-nanofiber electrodes demonstrate high transparency (>70%), low sheet resistance (1.68-11.1 Ω □-1 ), excellent stretchability, and stability (>100% strain). Based on the electrode patterning and device design, an 8 × 8 triboelectric sensor matrix is fabricated, which works well under high strain owing to the effect of the electrostatic induction. Using cross-locating technology, the device can execute more rapid tactile mapping, with a response time of 70 ms. In addition, the object being detected can be made from any commonly used materials or can even be human hands, indicating that this device has widespread potential in tactile sensing and touchpad technology applications.

Sensitivity-Enhanced Wearable Active Voiceprint Sensor Based on Cellular Polypropylene Piezoelectret.

Wearable active sensors have extensive applications in mobile biosensing and human-machine interaction but require good flexibility, high sensitivity, excellent stability, and self-powered feature. In this work, cellular polypropylene (PP) piezoelectret was chosen as the core material of a sensitivity-enhanced wearable active voiceprint sensor (SWAVS) to realize voiceprint recognition. By virtue of the dipole orientation control method, the air layers in the piezoelectret were efficiently utilized, and the current sensitivity was enhanced (from 1.98 pA/Hz to 5.81 pA/Hz at 115 dB). The SWAVS exhibited the superiorities of high sensitivity, accurate frequency response, and excellent stability. The voiceprint recognition system could make correct reactions to human voices by judging both the password and speaker. This study presented a voiceprint sensor with potential applications in noncontact biometric recognition and safety guarantee systems, promoting the progress of wearable sensor networks.

A Skin Segmentation Algorithm Based on Stacked Autoencoders

A good skin detector that is capable of capturing skin tones under different conditions is important for human–machine interaction applications. In a general situation, skin detectors, such as skin probability maps or Gaussian mixture models, achieve acceptable skin segmentation results. However, the false positive rate increases significantly when the skin tones are in shadow or when skin-like background objects are under similar illumination. In this paper, we propose a novel skin feature learning algorithm based on stacked autoencoders, which are deep neural networks. To overcome the problems encountered in skin segmentation that are caused by different ethnicities and varying illumination conditions, the stacked autoencoders are utilized to learn more discriminative representations of the skin area in both the RGB color space and the HSV color space. Unlike traditional machine learning methods, instead of predicting each pixel individually, our algorithm utilizes blocks to learn the representations and detect the skin areas. The algorithm exploits the learning ability of deep neural networks to learn high-level representations of skin tones. Experiments on test images show that the proposed algorithm achieves acceptable results on several publicly available data sets. To reduce the difficulty of detecting skin pixels in these data sets, the ground truths of these data sets are commonly focused on foreground skin area detection. Our skin detector is also able to detect background areas, as shown in our experiments.