Modal Sensor Research Articles

MSCS: Multi-stage feature learning with channel-spatial attention mechanism for infrared and visible image fusion

The intention of infrared and visible image fusion is to combine the images captured by different modal sensors in the same scene to enhance its understanding. Deep learning has been proven its powerful application in image fusion due to its fine generalization, robustness, and representability of deep features. However, the performance of these deep learning-based methods heavily depends on the illumination condition. Especially in dark or exposed scenes, the fused results are over-smoothness and low-contrast, resulting in inaccuracy of object detection. To address these issues, this paper develops a multi-stage feature learning approach with channel-spatial attention mechanism, namely MSCS, for infrared and visible image fusion. The MSCS is composed of the following four key procedures: Firstly, the infrared and visible images are decomposed into illumination and reflectance components by a proposed network called as Retinex_Net. Then, the components are transported to an encoder for features coding. Next, we propose an adaptive fusion module with attention mechanisms to fuse the features. Finally, the fused image is generated by the decoder for decoding the fused features. Meanwhile, a novel fusion loss function and a multi-stage training strategy are proposed to train the above modules. The subjective and objective results of experiments on TNO, LLVIP and MSRS datasets illustrate that the proposed method is effective and performs better than the state-of-the-art fusion methods on achieving enjoyable results in dark or over-exposure scenes. And the results of further experiments on the fused images for object detection demonstrate that the fusion outputs produced by our MSCS are more beneficial for detection tasks.

Simplification of Electrode Profiles for Piezoelectric Modal Sensors by Controlling Gap-Phase Length

This paper presents a method to optimize and simplify the electrode profile of a piezoelectric modal sensor. At the same time that the electrode profile is optimized to maximize the charge, a null-polarity phase is introduced. This gap-phase is modeled using the normalized norm of the spatial gradient of the polarity profile along with a two-step filtering and projection technique. High-order vibration modes generate a complex polarization profile that makes the manufacturing process difficult. The novelty of the proposed method is the addition of a constraint on the length of the interface in the topology optimization problem. This constraint simplifies the optimal designs and facilitates the fabrication process. Several examples show the simplified electrode profile that maximizes the electrical charge produced by a vibration mode, while reducing the number of different polarization regions by means of the gap-length constraint.

Trustworthiness of IoT Images Leveraging With Other Modal Sensor’s Data

Trustworthiness of IoT Images Leveraging With Other Modal Sensor’s Data

Learning-Based Multimodal Information Fusion and Behavior Recognition of Vascular Interventionists' Operating Skills.

The operating skills of vascular interventionists have an important impact on the effect of surgery. However, current research on behavior recognition and skills learning of interventionists' operating skills is limited. In this study, an innovative deep learning-based multimodal information fusion architecture is proposed for recognizing and analyzing eight common operating behaviors of interventionists. An experimental platform integrating four modal sensors is used to collect multimodal data from interventionists. The ANOVA and Manner-Whitney tests is used for relevance analysis of the data. The analysis results demonstrate that there is almost no significant difference ( p <0.001) between the actions related to the unimodal data, which cannot be used for accurate behavior recognition. Therefore, a study of the fusion architecture based on the existing machine learning classifier and the proposed deep learning fusion architecture is carried out. The research findings indicate that the proposed deep learning-based fusion architecture achieves an impressive overall accuracy of 98.5%, surpassing both the machine learning classifier (93.51%) and the unimodal data (90.05%). The deep learning-based multimodal information fusion architecture proves the feasibility of behavior recognition and skills learning of interventionist's operating skills. Furthermore, the application of deep learning-based multimodal fusion technology of surgeon's operating skills will help to improve the autonomy and intelligence of surgical robotic systems.

Capacitive/Resistive Strain Sensors Based on Organogel/Hydrogel Hybrids via a Low-Swelling Solvent Replacement Strategy

As potential flexible sensors to mimic skin functions or achieve human–computer interactions, organogel/hydrogel hybrids, prepared mostly through “adhesion strategies” and “interfacial polymerization strategies”, have gained increasing attention in recent years. Herein, the organogel/hydrogel hybrids with different structures and functions were prepared through a low-swelling solvent replacement strategy. Organogel/hydrogel hybrids were prepared by in situ solvent replacement with aqueous NaCl solution (10 wt %) on native organogels, which were fabricated by photoinitiated copolymerization of methacrylic acid (MAAc) and N-(pyridin-2-yl)acrylamide (NPAM) in glycerol. Capacitive or resistive modal sensors based on organogel/hydrogel hybrids can be obtained easily. In particular, because the patterns were completely dependent on the mold containing the displacing solvent, flexible electronic sensors based on organogel/hydrogel hybrids with complex topological structures and functions could be designed and prepared conveniently. Since the interface between the hydrogel and the organogel also formed in situ and was very strong, the organogel/hydrogel hybrids exhibit excellent puncture-resistant performance. Our experimental results indicated that organogel/hydrogel hybrids with complex patterns and topologies could be prepared through a low-swelling solvent replacement strategy. The design and practical application of flexible electronic sensors based on organogel/hydrogel hybrids will be enlarged and advanced substantially.

A multimodal human-robot sign language interaction framework applied in social robots.

Deaf-mutes face many difficulties in daily interactions with hearing people through spoken language. Sign language is an important way of expression and communication for deaf-mutes. Therefore, breaking the communication barrier between the deaf-mute and hearing communities is significant for facilitating their integration into society. To help them integrate into social life better, we propose a multimodal Chinese sign language (CSL) gesture interaction framework based on social robots. The CSL gesture information including both static and dynamic gestures is captured from two different modal sensors. A wearable Myo armband and a Leap Motion sensor are used to collect human arm surface electromyography (sEMG) signals and hand 3D vectors, respectively. Two modalities of gesture datasets are preprocessed and fused to improve the recognition accuracy and to reduce the processing time cost of the network before sending it to the classifier. Since the input datasets of the proposed framework are temporal sequence gestures, the long-short term memory recurrent neural network is used to classify these input sequences. Comparative experiments are performed on an NAO robot to test our method. Moreover, our method can effectively improve CSL gesture recognition accuracy, which has potential applications in a variety of gesture interaction scenarios not only in social robots.

Multi-modal pedestrian detection with misalignment based on modal-wise regression and multi-modal IoU

Multi-modal pedestrian detection, which integrates visible and thermal sensors, has been developed to overcome many limitations of visible-modal pedestrian detection, such as poor illumination, cluttered background, and occlusion. By adopting the combination of multiple modalities, we can efficiently detect pedestrians even with poor visibility. Nevertheless, the critical assumption of multi-modal pedestrian detection is that multi-modal images are perfectly aligned. In general, however, this assumption often becomes invalid in real-world situations. Viewpoints of the different modal sensors are usually different. Then, the positions of pedestrians on the different modal images have disparities. We proposed a multi-modal faster-RCNN specifically designed to handle misalignment between two modalities. The faster-RCNN consists of a region proposal network (RPN) and a detector. We introduce position regressors for both modalities in the RPN and the detector. Intersection over union (IoU) is one of the useful metrics for object detection but is defined only for a single-modal image. We extend it into multi-modal IoU to evaluate the preciseness of both modalities. Our experimental results with the proposed evaluation metrics demonstrate that the proposed method has comparable performance with state-of-the-art methods and outperforms them for data with significant misalignment.

SPV-SSD: An Anchor-Free 3D Single-Stage Detector with Supervised-PointRendering and Visibility Representation

Recently, 3D object detection based on multi-modal sensor fusion has been increasingly adopted in automated driving and robotics. For example, the semantic information provided by cameras and the geometric information provided by light detection and ranging (LiDAR) are fused to perceive 3D objects, as single modal sensors are unable to capture enough information from the environment. Many state-of-the-art methods fuse the signals sequentially for simplicity. By sequentially, we mean using the image semantic signals as auxiliary input for LiDAR-based object detectors would make the overall performance heavily rely on the semantic signals. Moreover, the error introduced by these signals may lead to detection errors. To remedy this dilemma, we propose an approach coined supervised-PointRendering to correct the potential errors in the image semantic segmentation results by training auxiliary tasks with fused features of the laser point geometry feature, the image semantic feature and a novel laser visibility feature. The laser visibility feature is obtained through the raycasting algorithm and is adopted to constrain the spatial distribution of fore- and background objects. Furthermore, we build an efficient anchor-free Single Stage Detector (SSD) powered by an advanced global-optimal label assignment to achieve a better time–accuracy balance. The new detection framework is evaluated on the extensively used KITTI and nuScenes datasets, manifesting the highest inference speed and at the same time outperforming most of the existing single-stage detectors with respect to the average precision.

Hybrid Ensemble Based Machine Learning for Smart Building Fire Detection Using Multi Modal Sensor Data

Hybrid Ensemble Based Machine Learning for Smart Building Fire Detection Using Multi Modal Sensor Data

Crack identification of functionally graded beam using distributed piezoelectric sensor

The present paper addresses development of a procedure for crack detection in functionally graded Timoshenko beam using a distributed piezoelectric sensor. Crack is represented by a pair of translational and rotational springs of stiffness calculated from its depth. A piezoelectric layer bonded to the beam is employed as a distributed sensor. Adopting the double beam model for the beam with the sensor, governing equations are conducted and utilized for establishing a database to propose a procedure for detecting a single crack in a functionally graded beam using the sensor output charge. The modal sensor charge provides a novel indicator more efficient for crack identification in functionally graded beams than natural frequencies. The effect of measurement noise, sensor thickness, and material parameters on crack identification has been investigated for beams with different boundary conditions.

CZU-MHAD: A Multimodal Dataset for Human Action Recognition Utilizing a Depth Camera and 10 Wearable Inertial Sensors

Human action recognition has been widely used in many fields of life, and many human action datasets have been published at the same time. However, most of the multi-modal databases have some shortcomings in the layout and number of sensors, which cannot fully represent the action features. Regarding the problems, this paper proposes a freely available dataset, named CZU-MHAD (Changzhou University: a comprehensive multi-modal human action dataset). It consists of 22 actions and three modals temporal synchronized data. These modals include depth videos and skeleton positions from a kinect v2 camera, and inertial signals from 10 wearable sensors. Compared with single modal sensors, multi-modal sensors can collect different modal data, so the use of multi-modal sensors can describe actions more accurately. Moreover, CZU-MHAD obtains the 3-axis acceleration and 3-axis angular velocity of 10 main motion joints by binding inertial sensors to them, and these data were captured at the same time. Experimental results are provided to show that this dataset can be used to study structural relationships between different parts of the human body when performing actions and fusion approaches that involve multi-modal sensor data.

Comparative analysis of HAR datasets using classification algorithms

In the current research and development era, Human Activity Recognition (HAR) plays a vital role in analyzing the movements and activities of a human being. The main objective of HAR is to infer the current behaviour by extracting previous information. Now-a-days, the continuous improvement of living condition of human beings changes human society dramatically. To detect the activities of human beings, various devices, such as smartphones and smart watches, use different types of sensors, such as multi modal sensors, non-video based and video-based sensors, and so on. Among the entire machine learning approaches, tasks in different applications adopt extensively classification techniques, in terms of smart homes by active and assisted living, healthcare, security and surveillance, making decisions, tele-immersion, forecasting weather, official tasks, and prediction of risk analysis in society. In this paper, we perform three classification algorithms, Sequential Minimal Optimization (SMO), Random Forest (RF), and Simple Logistic (SL) with the two HAR datasets, UCI HAR and WISDM, downloaded from the UCI repository. The experiment described in this paper uses the WEKA tool to evaluate performance with the matrices, Kappa statistics, relative absolute error, mean absolute error, ROC Area, and PRC Area by 10-fold cross validation technique. We also provide a comparative analysis of the classification algorithms with the two determined datasets by calculating the accuracy with precision, recall, and F-measure metrics. In the experimental results, all the three algorithms with the UCI HAR datasets achieve nearly the same accuracy of 98%.The RF algorithm with the WISDM dataset has the accuracy of 90.69%,better than the others.

Modal analysis of cracked FGM beam with piezoelectric layer

The present paper addresses modal analysis of cracked Timoshenko beam bonded with piezoelectric layer. The host beam is made of functionally graded material with the power law and crack appeared in the host beam is modeled by a pair of axial and rotational springs with stiffness calculated from its depth. The piezoelectric layer is represented as a homogeneous Timoshenko beam. Governing equations and general solution for free vibration of the cracked double beam are conducted in the frequency domain. The obtained solution has been used for examining modal parameters of the coupled beam that include natural frequencies, mode shapes and so-called modal sensor charge in dependence on crack parameter, material gradient index and thickness of piezoelectric layer. Analysis of the modal sensor charge in dependence upon crack parameters provides a novel indicator that is expected to be more efficient than the traditional ones for crack detection in functionally graded beam using distributed smart sensors. The theoretical development is validated and illustrated by numerical analysis accomplished for simply supported beam.

Multi-functional lanthanide coordination polymers for multi-modal detection of nitroaromatics and trace water in organic solvents

Multifunctional materials used for multimodal sensors are of great significance both for enriching variety of materials and overcoming the shortcomings of traditional single signal sensors. This work reports the ultrafast preparation of multifunctional coordination polymer (CP) spheres with metalloligand Ru (II)[4,4′-(COOH)2-bpy]3▪Cl2 (L) and Gd3+ ions. Gd-L CPs were used for multi-modal detection of nitroaromatic explosives and trace water in organic solvents with their red fluorescence and magnetic resonance response. The red fluorescence was derived from the ligand, which is electron-rich because of the bipyridine rings. There were different emission behaviors when Gd-L CPs reacted with nitroaromatic explosives that have various electron-deficient properties through electron transfer donor–acceptor mechanism. The changes in the emission depended closely on the nitroaromatic explosives ability to withdraw electrons. Gd-L CPs also showed magnetic resonance response with r1 of 12.1 mM−1s−1 due to the integration of the paramagnetic property of Gd3+ ions. Smart sensing of trace water in organic solvents was achieved by magnetic signals. The fluorescence of Gd-L CPs was enhanced and there was a tiny blue-shift in protonic solvents with increased polarity. The fluorescence sensor used in detecting trace water had a consistent result like the MR response. Thus, Gd-L CPs enabled the use of fluorescence and magnetic resonance sensors simultaneously for the first time to overcome the shortcoming of single modal sensor and expand the application of magnetic resonance relaxation.

Design of distributed piezoelectric modal sensor for a rotating beam with elastic boundary restraints

As an advanced sensing technique, modal sensors have been attracting a lot of research interest in modal filtering and active control fields. Most of the existing investigations are mainly focused on the static structure. In contrast, there is little effort made for its rotating counterpart, which is frequently encountered in various power machineries. Motivated by such limitation, a unified framework for the distributed piezoelectric modal sensor design of rotating beams with elastic boundary restraints is proposed using polyvinylidene fluoride piezoelectric integral equation and the second-order structural modal functions. A boundary smoothed Fourier series is used to obtain the modal information of rotating beams by solving the differential governing equation and elastic boundary conditions, simultaneously. Modal sensor shape of rotating beams can be determined for any boundary condition by simply setting the elastic restraining coefficients accordingly, instead of reformulating the equation or rewriting the codes like other approach usually does. Numerical examples are presented to demonstrate the correctness and effectiveness of the proposed framework. Modal sensitivity coefficient and charge output frequency response under external excitation are calculated to demonstrate the performance of the designed piezoelectric modal sensors. Influence of rotation speed and boundary restraining stiffness on the modal sensing accuracy of the shaped polyvinylidene fluoride sensor is analyzed and addressed. To our best knowledge, this work represents the first time that an analytical solution for the distributed piezoelectric modal sensor design of a rotating beam with general boundary conditions is derived, which can shed some new lights on further design and implementation of polyvinylidene fluoride modal sensing technique for rotating structures.

Time-Scale Analysis Based Modal Identification Using Mobile Sensors

Time-Scale Analysis Based Modal Identification Using Mobile Sensors

Implementation of Pressure Sensor of Optical Fiber Using Optical Interferometer

In present work, two types of Interferometric Fiber Optic Sensor (Fabry – Perot &amp; Modal Sensor) have been demonstrate and investigated. The main parameter studied of this contribute is the sensitivity, the strain could be induced by make a stress on the optical fiber. The strain effect at the fiber due to variation of the intensity in the output of the optical fiber. Then, the modes of electromagnetic waves that propagate in the fiber could be analyzed to determine the sensitivity depend on fringe rates. I conclude from this study the Extrinsic Fabry – Perot Interferometry structure is more sensitive than Modal Sensor.

Machine learning-based multi-modal information perception for soft robotic hands

This paper focuses on multi-modal Information Perception (IP) for Soft Robotic Hands (SRHs) using Machine Learning (ML) algorithms. A flexible Optical Fiber-based Curvature Sensor (OFCS) is fabricated, consisting of a Light-Emitting Diode (LED), photosensitive detector, and optical fiber. Bending the roughened optical fiber generates lower light intensity, which reflecting the curvature of the soft finger. Together with the curvature and pressure information, multi-modal IP is performed to improve the recognition accuracy. Recognitions of gesture, object shape, size, and weight are implemented with multiple ML approaches, including the Supervised Learning Algorithms (SLAs) of K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Logistic Regression (LR), and the unSupervised Learning Algorithm (un-SLA) of K-Means Clustering (KMC). Moreover, Optical Sensor Information (OSI), Pressure Sensor Information (PSI), and Double-Sensor Information (DSI) are adopted to compare the recognition accuracies. The experiment results demonstrate that the proposed sensors and recognition approaches are feasible and effective. The recognition accuracies obtained using the above ML algorithms and three modes of sensor information are higher than 85 percent for almost all combinations. Moreover, DSI is more accurate when compared to single modal sensor information and the KNN algorithm with a DSI outperforms the other combinations in recognition accuracy.

Polyvinylidene fluoride modal sensor design for elastically restrained beams resting on general non-uniform foundation

In this article, a unified solution of polyvinylidene fluoride modal sensor design for elastically restrained beams resting on general non-uniform foundation is proposed using an improved Fourier series method. Modal sensor shape is determined from the charge output equation of polyvinylidene fluoride film and the superposition of the second-order derivative of beam mode shape. With the aim to make the spatial differentials of various order sufficiently continuous in the interval [0, L], Fourier series with auxiliary boundary-smoothed terms are employed to construct the displacement expression. Energy formulation is utilized for system dynamic description, with all the unknown expansion coefficients solved in conjunction with Rayleigh–Ritz procedure. The proposed model is validated through the comparison of modal parameters of simply supported beam resting on elastic foundation with those available in literature. Correctness and effectiveness of the designed polyvinylidene fluoride modal sensor are verified by calculating the modal sensitivity coefficients and the charge output frequency response under external excitation. Based on the model established, the perturbation of practical boundary restraining stiffness on sensing accuracy of such polyvinylidene fluoride sensor is studied, and the influences of boundary condition on polyvinylidene fluoride modal sensor design are investigated for the beam structure resting on general non-uniform foundation.

A Simple Colorimetric and Fluorescent Sensor to Detect Organophosphate Pesticides Based on Adenosine Triphosphate-Modified Gold Nanoparticles

A simple and dual modal (colorimetric and fluorescent) sensor for organophosphate pesticides with high sensitivity and selectivity using adenosine triphosphate (ATP)- and rhodamine B-modified gold nanoparticles (RB-AuNPs), was successfully fabricated. This detection for ethoprophos afforded colorimetric and fluorescence imaging changes visualization. The quantitative determination was linearly proportional to the amounts of ethoprophos in the range of a micromolar scale (4.0–15.0 µM). The limit of detection for ethoprophos was as low as 37.0 nM at 3σ/k. Moreover, the extent application of this simple assay was successfully demonstrated in tap water samples with high reliability and applicability, indicating remarkable application in real samples.