Fusion Of Image Features Research Articles

Attention based deep convolutional U-Net with CSA optimization for hyperspectral image denoising

Hyperspectral image (HSI) de-noising plays a significant role in HSI quality enhancement because it consists of rich image information. Although, large amounts of information commonly have a lot of noise which will greatly impact HSI processing and application. Deep convolution neural networks (CNNs) have been predominant in image de-noising because of recent advances in deep learning. However, CNNs may result in performance degradation due to plain network depth growth. It leads to face challenges in the shallow image feature fusion and limits the potential of the network to extract HSI features. To overcome these problems, an attention-based deep convolutional U-Net is proposed in this research work. It improves the down-sampling and up-sampling strategies to extract the image features and restore the HSI information, respectively. In order to provide better de-noising performance, the edge information of the image is extracted with the aid of a depthwise and polarized self-attention mechanism. Moreover, the clonal selection optimization algorithm (CSOA) minimizes the loss function obtained from the proposed system by selecting optimal parameters. The dataset used to evaluate the proposed model is the Real-World Hyperspectral Images database and is implemented in the working platform of Python. The experimental result shows that the proposed system performs better than existing state-of-the-art de-noising schemes. The achieved PSNR, SSIM, SAM and ERGAS performance metrics values are: 35.82, 0.94, 3.14 and 19.42, respectively.

Anti-Noise 3D Object Detection of Multimodal Feature Attention Fusion Based on PV-RCNN.

3D object detection methods based on camera and LiDAR fusion are susceptible to environmental noise. Due to the mismatch of physical characteristics of the two sensors, the feature vectors encoded by the feature layer are in different feature spaces. This leads to the problem of feature information deviation, which has an impact on detection performance. To address this problem, a point-guided feature abstract method is presented to fuse the camera and LiDAR at first. The extracted image features and point cloud features are aggregated to keypoints for enhancing information redundancy. Second, the proposed multimodal feature attention (MFA) mechanism is used to achieve adaptive fusion of point cloud features and image features with information from multiple feature spaces. Finally, a projection-based farthest point sampling (P-FPS) is proposed to downsample the raw point cloud, which can project more keypoints onto the close object and improve the sampling rate of the point-guided image features. The 3D bounding boxes of the object is obtained by the region of interest (ROI) pooling layer and the fully connected layer. The proposed 3D object detection algorithm is evaluated on three different datasets, and the proposed algorithm achieved better detection performance and robustness when the image and point cloud data contain rain noise. The test results on a physical test platform further validate the effectiveness of the algorithm.

Loop Closure Detection for Mobile Robot based on Multidimensional Image Feature Fusion

Loop closure detection is a crucial part of VSLAM. However, the traditional loop closure detection algorithms are difficult to adapt to complex and changeable scenes. In this paper, we fuse Gist features, semantic features and appearance features of the image to detect the loop closures quickly and accurately. Firstly, we take advantage of the fast extraction speed of the Gist feature by using it to screen the loop closure candidate frames. Then, the current frame and the candidate frame are semantically segmented to obtain the mask blocks of various types of objects, and the semantic nodes are constructed to calculate the semantic similarity between them. Next, the appearance similarity between the images is calculated according to the shape of the mask blocks. Finally, based on Gist similarity, semantic similarity and appearance similarity, the image similarity calculation model can be built as the basis for loop closure detection. Experiments are carried out on both public and self-filmed datasets. The results show that our proposed algorithm can detect the loop closure in the scene quickly and accurately when the illumination, viewpoint and object change.

Road Pavement Damage Detection Based on Local Minimum of Grayscale and Feature Fusion

In this work, we propose a road pavement damage detection deep learning model based on feature points from a local minimum of grayscale. First, image blocks, consisting of the neighborhood of feature points, are cut from the image window to form an image block dataset. The image blocks are then input into a convolutional neural network (CNN) to train the model, extracting the image block features. In the testing process, the feature points as well as the image blocks are selected from a test image, and the trained CNN model can output the feature vectors for these feature image blocks. All the feature vectors will be combined to a composite feature vector as the feature descriptor of the test image. At last, the classifier of the model, constructed by a support vector machine (SVM), gives the classification as to whether the image window contains damaged areas or not. The experimental results suggest that the proposed pavement damage detection method based on feature-point image blocks and feature fusion is of high accuracy and efficiency. We believe that it has application potential in general road damage detection, and further investigation is desired in the future.

An efficient interactive segmentation framework for medical images without pre-training.

Accurate and efficient medical image segmentation plays an important role in subsequent clinical applications such as diagnosis and surgical planning. This paper proposes an efficient interactive framework based on a graph convolutional network (GCN) for medical imagesegmentation. The initial segmentation results showed that a set of boundary control points can be generated for further interactive segmentation. We presented an adaptive interactive manner that allows the user to click on the boundary for fast interaction or drag the erroneous predicted control points for accurate correction. Furthermore, we proposed an interactive segmentation network (referred to as IVIF-GCN) to learn user experience in the interactive process by transforming interactive cues into annotations. In IVIF-GCN, a module of information fusion of image features and vertex position features (IVIF) is proposed to learn the location relationship between the current vertex and the neighboring vertices. Finally, the locations of control points around the interaction point is predicted and updatedautomatically. The proposed method achieves mean Dice of 96.6% and 91.3% on PROMISE12 and our in-house nasopharyngeal carcinoma (NPC) test sets, respectively. The experimental results showed that the proposed method outperforms the state-of-the-art segmentationmethods. The proposed interactive medical image segmentation method can efficiently improve segmentation results for clinical applications in the absence of training data. The GUI tool based on our method is available at https://github.com/Tian-lab/IGMedSeg.

A Lightweight Multi-Level Information Network for Multispectral and Hyperspectral Image Fusion

The process of fusing the rich spectral information of a low spatial resolution hyperspectral image (LR-HSI) with the spatial information of a high spatial resolution multispectral image (HR-MSI) to obtain an HSI with the spatial resolution of an MSI image is called hyperspectral image fusion (HIF). To reconstruct hyperspectral images at video frame rate, we propose a lightweight multi-level information network (MINet) for multispectral and hyperspectral image fusion. Specifically, we develop a novel lightweight feature fusion model, namely residual constraint block based on global variance fine-tuning (GVF-RCB), to complete the feature extraction and fusion of hyperspectral images. Further, we define a residual activity factor to judge the learning ability of the residual module, thereby verifying the effectiveness of GVF-RCB. In addition, we use cascade cross-level fusion to embed the different spectral bands of the upsampled LR-HSI in a progressive manner to compensate for lost spectral information at different levels and to maintain spatial high frequency information at all times. Experiments on different datasets show that our MINet outperforms the state-of-the-art methods in terms of objective metrics, in particular by requiring only 30% of the running time and 20% of the number of parameters.

Attention-Based Multimodal Image Feature Fusion Module for Transmission Line Detection

Transmission line (TL) inspection is important for ensuring a stable supply of electricity to rural areas. Currently, there are several TL detection approaches based on computer vision; however, they have limitations owing to background clutter in visible light images. This article presents a novel multimodal image feature fusion module that utilizes both visible light and infrared images to enhance the TL-detection performance. The proposed module consists of a multibranch feature extraction (MFE) block followed by a channelwise attention (CA) block. The first block extracts the representative features of each modal input using multiple branches. The outputs of the MFE block are jointly aggregated into an attention vector in the CA block. Finally, the attention vector recalibrates each input feature of the proposed module. To reduce the number of additional parameters due to the insertion of the module, we introduced a channel-shrink factor in the MFE block and utilized a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1\times {1}$</tex-math></inline-formula> convolution in the CA block. Comparison experiments with various augmented conditions of day, night, fog, and snow were conducted on a real-world dataset, which we constructed by visible light and infrared images. The results showed that the proposed module outperformed not only the case of single modal input but also the state-of-the-art fusion methods, regardless of the baseline networks. Additionally, the proposed module showed effectiveness in terms of capacity when the baseline network has a large number of weight parameters.

Detection of wheat Fusarium head blight using UAV-based spectral and image feature fusion.

Infection caused by Fusarium head blight (FHB) has severely damaged the quality and yield of wheat in China and threatened the health of humans and livestock. Inaccurate disease detection increases the use cost of pesticide and pollutes farmland, highlighting the need for FHB detection in wheat fields. The combination of spectral and spatial information provided by image analysis facilitates the detection of infection-related damage in crops. In this study, an effective detection method for wheat FHB based on unmanned aerial vehicle (UAV) hyperspectral images was explored by fusing spectral features and image features. Spectral features mainly refer to band features, and image features mainly include texture and color features. Our aim was to explain all aspects of wheat infection through multi-class feature fusion and to find the best FHB detection method for field wheat combining current advanced algorithms. We first evaluated the quality of the two acquired UAV images and eliminated the excessively noisy bands in the images. Then, the spectral features, texture features, and color features in the images were extracted. The random forest (RF) algorithm was used to optimize features, and the importance value of the features determined whether the features were retained. Feature combinations included spectral features, spectral and texture features fusion, and the fusion of spectral, texture, and color features to combine support vector machine, RF, and back propagation neural network in constructing wheat FHB detection models. The results showed that the model based on the fusion of spectral, texture, and color features using the RF algorithm achieved the best performance, with a prediction accuracy of 85%. The method proposed in this study may provide an effective way of FHB detection in field wheat.

Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language

Machine translation makes it easy for people to communicate across languages. Multimodal machine translation is also one of the important directions of research in machine translation, which uses feature information such as images and audio to assist translation models in obtaining higher quality target languages. However, in the vast majority of current research work has been conducted on the basis of commonly used corpora such as English, French, German, less research has been done on low-resource languages, and this has left the translation of low-resource languages relatively behind. This paper selects the English-Hindi and English-Hausa corpus, researched on low-resource language translation. The different models we use for image feature information extraction are fusion of image features with text information in the text encoding process of translation, using image features to provide additional information, and assisting the translation model for translation. Compared with text-only machine translation, the experimental results show that our method improves 3 BLEU in the English-Hindi dataset and improves 0.47 BLEU in the English-Hausa dataset. In addition, we also analyze the effect of image feature information extracted by different feature extraction models on the translation results. Different models pay different attention to each region of the image, and ResNet model is able to extract more feature information compared to VGG model, which is more effective for translation.

FS-IDS: A framework for intrusion detection based on few-shot learning

Due to the high dependency of traditional intrusion detection method on a fully-labeled large dataset, existing works can hardly be applied in real-world scenarios, especially facing zero-day attacks. In this paper we present a novel intrusion detection framework called “FS-IDS”, including flow data encoding method, feature fusion mechanism and architecture of intrusion detection system based on few-shot learning. We utilize task generator to split the dataset into separate tasks and train model in an episodic way, hoping model to learn general knowledge rather than those specific to a single class. The extraction module and distance metric module are responsible for learning and determining whether the traffic data are benign or not. We conduct three sets of experiments on “FS-IDS”, i.e., comparison study, ablation study and multiclass study. Comparison study firstly determines that the best measure metric for discrimination is Euclidean distance. Based on the optimal implementation, “FS-IDS” achieves comparable performance with existing works by using much fewer malicious samples. Ablation study sets two base models to explore how proposed encoding method and feature fusion mechanism improve detection capacity. Both the image representation and feature fusion achieve more than 2% improvement in accuracy and recall. Finally, to test whether “FS-IDS” can perform well under real-world scenario or not, we design network traffic containing various attacks to simulate complex malicious network environment. Experimental results show that “FS-IDS” maintains more than 90% detection accuracy and recall under the worst circumstances, which composes of various seen or unseen attacks with only a few malicious samples available.

Classification for Fruit Fly Images Using Convolutional Neural Networks

The discriminative information of fruit fly images often exists in a very fine region, resulting in hard extracting the desired features from the entire image level. Hence, it is a tough task to obtain the discriminative features in the fine-grained image for the classification of fruit fly images. To address this, here proposed a convolutional neural network of bilinear pooling based on feature fusion for the classification of fruit fly images. Firstly, Gaussian blur is performed on fruit fly images to reduce the detailed level of fruit fly images, which conveniently extracts high-level features. In the fruit fly images processed by Gaussian blur, the convolutional layers in the proposed model (i.e., consisting of CNN A and CNN B) are used for the features extraction from the processed images. Then, these extracted features are fused by using the matrix dot multiplication in the Bilinear layers. According to these fused features, the softmax layer performs the classification of fruit fly images. Experimental results on the 4000 image set containing different fruit fly morphology show that the propose method not only outperforms the state-of-the-art methods in the classification accuracy of fruit fly images, but also the precision, recall and score are above 94.87% on testing set and validation set. We find that image feature fusion has positive effects on promoting the accuracy of image classification. We also demonstrate that using feature connection operation after performing the matrix point multiplication operation is beneficial to feature fusion, instead of using feature connection operation directly. In addition, our findings indicates that convolutional neural networks easily obtain the desired features from the images performed by Gaussian blur.

Extraction of soybean planting area based on feature fusion technology of multi-source low altitude unmanned aerial vehicle images

Soybean is an important food and oil crop in the world. It is of great significance to statics the planting scale accurately for optimizing the crop planting structure and world food security. The technology of accurately extracting the area of soybean planting areas at the field scale using UAV images combined with deep learning algorithms is important for the application. In this study, firstly, RGB images and multispectral images (RGN) were acquired simultaneously by the quad-rotor UAV DJ-Phantom4 Pro at a flying height of 200 m. And the features were extracted from the RGB and RGN images. Further, the fusion image of RGB + VIs and the fusion image of RGN + VIs were obtained by concatenating the band reflectivity of the original image with the calculated Vegetation Index (VI). Then, the soybean planting area was segmented from the feature fusion images by U-Net. And the accuracy of the two sensors was compared. The results showed that the Kappa coefficients obtained based on RGB image, RGN image, CME(the combination of CIVE, MExG, and ExGR), ODR(the combination of OSAVI, DVI, and RDVI), RGB + CME(the combination of RGB and CME), and RGN + ODR(the combination of RGN and ODR) were 0.8806, 0.9327, 0.8437, 0.9330, 0.9420, and 0.9238, respectively. The Kappa coefficient of the combination of the original image and the vegetation index was higher than the original image, indicating that the vegetation index calculation was beneficial to improving the soybean recognition accuracy of the U-Net model. Among them, the precision of the soybean planting area extracted from RGB + CME was the highest, and the Kappa coefficient was 0.9420. Finally, the soybean recognition accuracy of U-Net was compared with the results of DeepLabv3+, Random Forest, and Support Vector Machine. The accuracy of U-Net was the best. It can be concluded that this research proposed the method that was using U-Net trained the fusion image of the original image and vegetation index feature fusion image obtained by the UAV platform, which can effectively segment soybean planting areas. The conclusion of this work provided important technical support for farm level, family cooperatives, and other business entities to manage finely soybean planting and production at low cost.

Educational Psychology Analysis Method for Extracting Students' Facial Information Based on Image Big Data.

At present, most of the research on academic emotions focuses on the concept, current situation, and relevance. There are not many researches on the application of artificial intelligence-based neural network facial expression recognition technology in practical teaching. With reference to image-based big data, this research integrates the application of artificial intelligence facial expression recognition technology with the research on educational theory and applies information technology to the actual teaching process, in order to promote the optimization of the teaching process and improve the learning effect. Method. A Hadoop cluster consisting of 3 nodes is built on the Linux system, and the environment required for Opencv execution is compiled for each node, which provides support for subsequent parallel optimization, feature extraction, feature fusion, and recognition of student facial images. The image data type and input and output format based on MapReduce framework are designed, and the image data is optimized by means of serialized files. The color features, texture features, and Sift features of students' facial images and common distractors were analyzed. A parallel extraction framework of student facial image features is designed, and based on this, the student facial image feature extraction under Hadoop platform is implemented. This paper proposes a dynamic sequential facial expression recognition method that combines shallow and deep features with an attention mechanism. The relative position of facial landmarks and local area texture features based on FACS represent shallow-level features. At the same time, the structure of ALexNet is improved to extract the deep features of sequence images to express high-level semantic features. The effectiveness of the facial expression recognition system is improved by introducing three attention mechanisms: self-attention, weight-attention, and convolutional attention. Results/Discussion. Through the analysis of the teaching effect, we found that when teachers can obtain the correct student's academic mood, they can intervene on the students' positive academic mood. The purpose of the intervention is to improve the positive academic emotions of students. After the students receive the intervention, their academic emotions are also improved and are positively correlated with their academic performance. Through the analysis of teaching effect, the research can achieve the predetermined goal. From the specific teaching effect of this study, it is concluded that in classroom teaching, teachers should devote energy to intervene in students' positive academic emotions, in order to improve students' positive academic emotions, which will improve students' academic performance and teaching.

Proposal and Experimental Study on a Diagnosis Method for Hermetic Refrigeration Compressor Using Dual Time-Frequency Image Fusion

The hermetic refrigeration compressor is the core component of the refrigeration system, failure of which will cause energy waste and reduce service life. Fault diagnosis based on vibration signal is a research hotspot. However, it is challenging to extract features of nonlinear and nonstationary vibration signals, which severely restricts the development of this method. This paper proposes a dual time-frequency images fusion method to obtain more effective features for diagnosing compressor manufacturing defects. Firstly, two time-frequency images are obtained by implementing continuous wavelet transform and Hilbert-Huang transform of the same vibration signal sample. Then, a convolutional neural network is used for image feature extraction and fusion, where the features extracted from two time-frequency images have complementarity. A data set containing six categories of typical manufacturing defects is used to verify the proposed method. The results show that the average diagnostic accuracy of the proposed method reaches 95.9%, and the proposed method has a better performance than other methods.

Outdoor Image Weather Recognition Based on Image Blocks and Feature Fusion

Outdoor Image Weather Recognition Based on Image Blocks and Feature Fusion

A Vaginitis Classification Method Based on Multi-Spectral Image Feature Fusion.

Vaginitis is one of the commonly encountered diseases of female reproductive tract infections. The clinical diagnosis mainly relies on manual observation under a microscope. There has been some investigation on the classification of vaginitis diseases based on computer-aided diagnosis to reduce the workload of clinical laboratory staff. However, the studies only using RGB images limit the development of vaginitis diagnosis. Through multi-spectral technology, we propose a vaginitis classification algorithm based on multi-spectral image feature layer fusion. Compared with the traditional RGB image, our approach improves the classification accuracy by 11.39%, precision by 15.82%, and recall by 27.25%. Meanwhile, we prove that the level of influence of each spectrum on the disease is distinctive, and the subdivided spectral image is more conducive to the image analysis of vaginitis disease.

A Multi-View Image Feature Fusion Network Applied in Analysis of Aeration Velocity for WWTP

The instability of the aeration system brings a significant challenge to the management of wastewater treatment plants (WWTP). Using image recognition methods to monitor aeration conditions accurately and enhance management efficiency is a promising way to solve this problem. To improve the efficiency of aeration condition identification and provide support for troubleshooting, we propose a method for aeration velocity condition identification based on a multi-view image feature fusion network (MVNN). Firstly, an experimental platform for simulating aeration tanks is established, and two cameras are used to acquire aeration images from different perspectives. Secondly, an image data set with 10 aeration velocity gradients is constructed and applied to the network’s training. Finally, the MVNN is used to extract and fuse the features of aeration images, and the model’s performance is evaluated on the dataset. Experiments show that the average accuracy of the method is over 98.3%, and the AUC of aeration identification is above 0.98, which indicates that the model has the potential for practical application in WWTP.

DeepCyto: a hybrid framework for cervical cancer classification by using deep feature fusion of cytology images

&lt;abstract&gt; &lt;p&gt;Cervical cancer is the second most commonly seen cancer in women. It affects the cervix portion of the vagina. The most preferred diagnostic test required for screening cervical cancer is the pap smear test. Pap smear is a time-consuming test as it requires detailed analysis by expert cytologists. Cytologists can screen around 100 to 1000 slides depending upon the availability of advanced equipment. Due to this reason Artificial intelligence (AI) based computer-aided diagnosis system for the classification of pap smear images is needed. There are some AI-based solutions proposed in the literature, still an effective and accurate system is under research. In this paper, the deep learning-based hybrid methodology namely DeepCyto is proposed for the classification of pap smear cytology images. The DeepCyto extracts the feature fusion vectors from pre-trained models and passes these to two workflows. Workflow-1 applies principal component analysis and machine learning ensemble to classify the pap smear images. Workflow-2 takes feature fusion vectors as an input and applies an artificial neural network for classification. The experiments are performed on three benchmark datasets namely Herlev, SipakMed, and LBCs. The performance measures of accuracy, precision, recall and F1-score are used to evaluate the effectiveness of the DeepCyto. The experimental results depict that Workflow-2 has given the best performance on all three datasets even with a smaller number of epochs. Also, the performance of the DeepCyto Workflow 2 on multi-cell images of LBCs is better compared to single cell images of other datasets. Thus, DeepCyto is an efficient method for accurate feature extraction as well as pap smear image classification.&lt;/p&gt; &lt;/abstract&gt;

Multiple Attention Siamese Network for High-Resolution Image Change Detection

Change detection (CD) remains an important issue in remote sensing applications, especially for high-resolution images, but it has yet to be fully resolved. In this study, we propose a novel Siamese network model, i.e., the multiple attention Siamese network (MASNet), for high-resolution image change detection (HRCD). The selective kernel convolution (SKConv) is first embedded into the encoders of the Siamese network to improve its feature extraction ability. The attention feature fusion module (AFFM) is then proposed and utilized to reform the network’s decoder, which realizes the fusion and selection of image features extracted from different scales and times. Experiments on three benchmark CD datasets, namely, the season-varying change detection dataset (SCDDS), Wuhan University (WHU) building change detection dataset (WBDS), and Google dataset (GDS), are conducted to validate the proposed model. The results show that the proposed attention module (the AFFM) is superior to multiple classic attention modules, including the squeeze-and-excitation block (SE), spatial SE block (sSE), spatial and channel SE block (scSE), convolutional block attention module (CBAM), and efficient channel attention module (ECA), in terms of the fusing of CD features. Furthermore, MASNet achieves a higher overall accuracy (OA), mean intersection over union (mIoU), F1-score (F1), and kappa coefficient (kappa) than ten state-of-the-art HRCD methods, such as the deeply supervised image fusion network (DSIFN), spatial–temporal attention neural network (STANet), and dual attentive fully convolutional Siamese network (DASNet) while maintaining low missed alarms and false alarms and having a fair method efficiency.

Pneumothorax Recognition Neural Network Based on Feature Fusion of Frontal and Lateral Chest X-Ray Images

Pneumothorax Recognition Neural Network Based on Feature Fusion of Frontal and Lateral Chest X-Ray Images