Convolutional Neural Network Feature Extractor Research Articles

A Visible and Synthetic Aperture Radar Image Fusion Algorithm Based on a Transformer and a Convolutional Neural Network

For visible and Synthetic Aperture Radar (SAR) image fusion, this paper proposes a visible and SAR image fusion algorithm (TCFuse) based on a Transformer and a Convolutional Neural Network (CNN). Firstly, in this paper, the Restormer Block is used to extract cross-modal shallow features. Then, we introduce an improved Transformer–CNN Feature Extractor (TCFE) with a two-branch residual structure. This includes a Transformer branch that introduces the Lite Transformer (LT) and DropKey for extracting global features and a CNN branch that introduces the Convolutional Block Attention Module (CBAM) for extracting local features. Finally, the fused image is output based on global features extracted by the Transformer branch and local features extracted by the CNN branch. The experiments show that the algorithm proposed in this paper can effectively achieve the extraction and fusion of global and local features of visible and SAR images, so that high-quality visible and SAR fusion images can be obtained.

Video quality adaptation using CNN and RNN models for cost-effective and scalable video streaming Services

AbstractVideo streaming services require adaptive bit rate strategies that optimize video quality based on network conditions and user preferences to provide a cost-effective and scalable solution. In this manuscript, we present a novel architecture that utilizes a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to extract features from the video stream and predict optimal bit rates for each frame. The CNN feature extractor extracts relevant features from the input video stream, which are then passed on to the RNN predictor, where the optimal bit rate is predicted to ensure a personalized viewing experience tailored to the current network conditions and specific user preferences. Our experimental results demonstrate that the proposed architecture achieves significant improvements in video quality while minimizing bandwidth usage and providing a better user experience. Specifically, our results show a 37.1% improvement in average bit rate, indicating that the proposed architecture optimizes the video quality and reduces bandwidth usage by 37.1% on average. We also observed a 16.6% improvement in Quality of Experience (QoE), meaning that users will perceive the video quality as better. Additionally, rebuffering was reduced by 87.5%, indicating that users enjoy smoother video playback without interruptions. Importantly, our architecture is optimized for Dynamic Adaptive Streaming over HTTP (DASH), addressing the need for efficient streaming over the most widely used protocol in the industry.

Multimodal Deep Learning for Rice Yield Prediction Using UAV-Based Multispectral Imagery and Weather Data

Precise yield predictions are useful for implementing precision agriculture technologies and making better decisions in crop management. Convolutional neural networks (CNNs) have recently been used to predict crop yields in unmanned aerial vehicle (UAV)-based remote sensing studies, but weather data have not been considered in modeling. The aim of this study was to explore the potential of multimodal deep learning on rice yield prediction accuracy using UAV multispectral images at the heading stage, along with weather data. The effects of the CNN architectures, layer depths, and weather data integration methods on the prediction accuracy were evaluated. Overall, the multimodal deep learning model integrating UAV-based multispectral imagery and weather data had the potential to develop more precise rice yield predictions. The best models were those trained with weekly weather data. A simple CNN feature extractor for UAV-based multispectral image input data might be sufficient to predict crop yields accurately. However, the spatial patterns of the predicted yield maps differed from model to model, although the prediction accuracy was almost the same. The results indicated that not only the prediction accuracies, but also the robustness of within-field yield predictions, should be assessed in further studies.

Semi-supervised visual anomaly detection based on convolutional autoencoder and transfer learning

Recent advances in deep neural networks have shown that reconstruction-based methods using autoencoders have potential for anomaly detection in visual inspection tasks. However, there are challenges when applying these methods to high-resolution images, such as the need for large network training and computation of anomaly scores. Autoencoder-based methods detect anomalies by comparing an input image to its reconstruction in pixel space, which can result in poor performance due to imperfect reconstruction. In this paper, we propose a method to address these challenges by using a conditional patch-based convolutional autoencoder and one-class deep feature classification. We train an autoencoder using only normal images and compute anomaly maps as the difference between the input and output of the autoencoder. We then embed these anomaly maps using a pretrained convolutional neural network feature extractor. Using the deep feature embeddings from the anomaly maps of training samples, we train a one-class classifier to compute an anomaly score for an unseen sample. A simple threshold-based criterion is used to determine if the unseen sample is anomalous or not. We compare our proposed algorithm to state-of-the-art methods on multiple challenging datasets, including a dataset of zipper cursors and eight datasets from the MVTec dataset collection. We find that our approach outperforms alternatives in all cases, achieving an average precision score of 94.77% for zipper cursors and 96.51% for MVTec datasets.

Infant’s MRI Brain Tissue Segmentation using Integrated CNN Feature Extractor and Random Forest

Infant MRI brain soft tissue segmentation become more difficult task compare with adult MRI brain tissue segmentation, due to Infant’s brain have a very low Signal to noise ratio among the white matter_WM and the gray matter _GM. Due the fast improvement of the overall brain at this time , the overall shape and appearance of the brain differs significantly. Manual segmentation of anomalous tissues is time-consuming and unpleasant. Essential Feature extraction in traditional machine algorithm is based on experts, required prior knowledge and also system sensitivity has change. Recently, bio-medical image segmentation based on deep learning has presented significant potential in becoming an important element of the clinical assessment process. Inspired by the mentioned objective, we introduce a methodology for analysing infant image in order to appropriately segment tissue of infant MRI images. In this paper, we integrated random forest classifier along with deep convolutional neural networks (CNN) for segmentation of infants MRI of Iseg 2017 dataset. We segmented infants MRI brain images into such as WM- white matter, GM-gray matter and CSF-cerebrospinal fluid tissues, the obtained result show that the recommended integrated CNN-RF method outperforms and archives a superior DSC-Dice similarity coefficient, MHD-Modified Hausdorff distance and ASD-Average surface distance for respective segmented tissue of infants brain MRI.

End-to-End Convolutional Neural Network Feature Extraction for Remote Sensed Images Classification

ABSTRACT Recently, land cover and land use (LCLU) classification in remote sensing imagery has attracted research interest. The LCLU contains dynamic remote sensed images due to sensor technology ability, seasonal changes, and distance for resolution. Therefore, the deep learning-based LCLU classification system needs more investigation using deep learning techniques. Deep learning approaches have gotten more attention for their powerful performance improvements. Most recent studies have been performed on deep convolutional neural networks (CNNs) that have been trained on pre-trained networks in remote sensing classification. However, designing CNNs from scratch has not yet been widely investigated in remote sensed images as they need ample training time and a powerful processor. Therefore, we used hyperparameters and early stopping techniques to apply an end-to-end CNN feature extractor (CNN-FE) model for LCLU classification in the UC-Merced dataset. We approved the model's applicability in the domain area by retraining it on another dataset called SIRI-WHU and building the VGG19 pre-trained feature extractor model built on the same hyperparameters. The CNN-FE has outperformed the state-of-the-art baseline studies' accuracy and the VGG19 pre-trained model. Moreover, a better CNN-FE performance was achieved when trained in the UC-Merced dataset than the model performance when trained in the SIRI-WHU dataset.

Spatiality Sensitive Learning for Cancer Metastasis Detection in Whole-Slide Images

Metastasis detection in lymph nodes via microscopic examination of histopathological images is one of the most crucial diagnostic procedures for breast cancer staging. The manual analysis is extremely labor-intensive and time-consuming because of complexities and diversities of histopathology images. Deep learning has been utilized in automatic cancer metastasis detection in recent years. Due to the huge size of whole-slide images, most existing approaches split each image into smaller patches and simply treat these patches independently, which ignores the spatial correlations among them. To solve this problem, this paper proposes an effective spatially sensitive learning framework for cancer metastasis detection in whole-slide images. Moreover, a novel spatial loss function is designed to ensure the consistency of prediction over neighboring patches. Specifically, through incorporating long short-term memory and spatial loss constraint on top of a convolutional neural network feature extractor, the proposed method can effectively learn both the appearance of each patch and spatial relationships between adjacent image patches. With the standard back-propagation algorithm, the whole framework can be trained in an end-to-end way. Finally, the regions with high tumor probability in the resulting probability map are the metastasis locations. Extensive experiments on the benchmark Camelyon 2016 Grand Challenge dataset show the effectiveness of the proposed approach with respect to state-of-the-art competitors. The obtained precision, recall, and balanced accuracy are 0.9565, 0.9167, and 0.9458, respectively. It is also demonstrated that the proposed approach can provide more accurate detection results and is helpful for early diagnosis of cancer metastasis.

Few-shot class-incremental learning based on representation enhancement

Few-shot class-incremental learning (FSCIL) is crucial and practical for artificial intelligence in the real world, which learns novel classes incrementally from few samples without forgetting the previously learned classes. However, FSCIL confronts two significant challenges: “catastrophic forgetting” and “overfitting new.” We focus on convolutional neural network (CNN)-based FSCIL and propose a human cognition-inspired FSCIL method, in which the knowledge of novel classes is learned under the guidance of the previously learned knowledge. Specifically, we learn a discriminative and generalized CNN feature extractor from the base classes in the first task. We generate the representations of base and novel classes in unified feature space without training on novel classes, thus avoiding “forgetting old.” For the novel classes in long sequential tasks, beyond the representation generation, we enhance the representation by exploring the correlations with the previously learned classes to alleviate overfitting new and ensure that the novel classes adapt to the feature space. Experimental results show that our proposed method achieves very competitive results on MiniImageNet and CIFAR-100 datasets.

Pre-trained CNNs as Feature-Extraction Modules for Image Captioning

In this work, we present a thorough experimental study about feature extraction using Convolutional NeuralNetworks (CNNs) for the task of image captioning in the context of deep learning. We perform a set of 72experiments on 12 image classification CNNs pre-trained on the ImageNet [29] dataset. The features areextracted from the last layer after removing the fully connected layer and fed into the captioning model. We usea unified captioning model with a fixed vocabulary size across all the experiments to study the effect of changingthe CNN feature extractor on image captioning quality. The scores are calculated using the standard metrics inimage captioning. We find a strong relationship between the model structure and the image captioning datasetand prove that VGG models give the least quality for image captioning feature extraction among the testedCNNs. In the end, we recommend a set of pre-trained CNNs for each of the image captioning evaluation metricswe want to optimise, and show the connection between our results and previous works. To our knowledge, thiswork is the most comprehensive comparison between feature extractors for image captioning.

Facial Makeup Detection Using Multi-Scale Local Binary Patterns and Convolutional Neural Network Fusion

This study presents a novel facial makeup detection scheme using fusion of multi-scale Local Binary Patterns (ms-LBP) texture methods and convolutional neural network (CNN) deep learning extractor. Facial makeups affect the accuracy of face recognition systems due to appearance alteration of individuals. In order to fuse the facial features, the proposed scheme first considers concatenation of extracted global histogram of a whole image using three different LBP operators. The LBP scales are used to extract and then concatenate the local histogram of overlapping and nonoverlapping blocks of images. This way utilizes the advantages of microtexton information of local primitives besides the extracted global textures. Finally, the extracted features using CNN feature extractor are combined with global and local multi-scale textures. In general, CNN deep learning extractor learns high-level discriminative characteristics of images and therefore the proposed system improves the facial makeup detection rate by involving both microtexton and discriminative information of facial images. In addition, the proposed method attempts to select the optimized subset of facial features to increase the detection performance of system by applying Particle Swarm Optimization (PSO) technique after feature level fusion. The paper uses Support Vector Machine (SVM) classifier for classifying the facial vectors into makeup or no-makeup classes. The proposed scheme is then evaluated using YMU, VMU and MIW facial makeup databases with consideration of light, medium and heavy makeups on several datasets. Experimental results analysis clarifies the effectiveness of proposed facial makeup detection framework of this study.

Hybrid FPGA-CPU-Based Architecture for Object Recognition in Visual Servoing of Arm Prosthesis.

The present paper proposes an implementation of a hybrid hardware–software system for the visual servoing of prosthetic arms. We focus on the most critical vision analysis part of the system. The prosthetic system comprises a glass-worn eye tracker and a video camera, and the task is to recognize the object to grasp. The lightweight architecture for gaze-driven object recognition has to be implemented as a wearable device with low power consumption (less than 5.6 W). The algorithmic chain comprises gaze fixations estimation and filtering, generation of candidates, and recognition, with two backbone convolutional neural networks (CNN). The time-consuming parts of the system, such as SIFT (Scale Invariant Feature Transform) detector and the backbone CNN feature extractor, are implemented in FPGA, and a new reduction layer is introduced in the object-recognition CNN to reduce the computational burden. The proposed implementation is compatible with the real-time control of the prosthetic arm.

Deep Learning Models Performance Evaluations for Remote Sensed Image Classification

Deep learning-based land cover and land use (LCLU) classification systems are a significant aspiration for remote sensing communities. In nature, remote sensing images have various properties that need to be analyzed. Analyzing and interpreting image properties is difficult due to the nature of the image, the sensor technology’s capability, and other determinant variables such as seasons and weather conditions. The problem is essential for environmental monitoring, agricultural decision-making, and urban planning if it can be supported by deep learning systems. Therefore, deep learning approaches are proposed to quickly analyze and interpret the remote sensing image to classify the LCLU. The deep learning methods could be designed starting from scratch or using pre-trained networks. However, there are few comparisons of deep learning methods developed from scratch and trained on pre-trained networks. Thus, we proposed evaluating and comparing the deep learning models convolutional neural network feature extractor (CNN-FE) by developing it from scratch, transfer learning, and fine-tuning it for the LCLU classification system using remote sensed images. Using CNN-FE, TL, and fine-tuning deep learning models as examples, this paper compares and analyzes deep learning algorithms for remote sensed image classification. After developing and training each deep learning model on the UCM dataset, we evaluated and compared their performances using the performance measurement metrics accuracy, precision, recall, f1-score, and confusion matrix. The proposed deep learning algorithms can adapt and learn the features of the remote sensing images, and the TL and fine-tuning classification performances are significantly improved. As a result of the efficient time used for training the models, this paper discovered that the fine-tuned deep learning model achieved profound accuracy performance results in the UCM dataset.

Constructing a PPI Network Based on Deep Transfer Learning for Protein Complex Detection

In the detection of protein complexes, the completeness of a protein–protein interaction (PPI) network is crucial. Complete PPI networks, however, are not available for most species because experimentally identified PPIs are usually very limited. This paper proposes a deep learning based PPI predictor to construct a complete PPI network, from which protein complexes are detected using a spectral clustering method. For this purpose, the unknown PPIs are estimated by using a deep PPI predictor consisting of a semi‐supervised SVM classifier and a deep feature extractor of the convolutional neural network (CNN). Meanwhile, the similarities of gene ontology (GO) annotations contribute to protein interactions, and the differences of subcellular localizations contribute to negative interactions. Considering that, we pretrain the deep CNN feature extractor in a class of deep GO annotation and subcellular localization predictors using datasets from the type species, then transfer it to the PPI prediction model for fine‐tuning. In this way, we have a deep PPI detector enhanced with transfer learning of GO annotation and subcellular localization prediction. Experimental results on benchmark datasets show that the proposed method outperforms the state‐of‐the‐art methods. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

A Semi‐supervised Classification Method of Parasites Using Contrastive Learning

An expected shortfall of supervised learning for medical imaging classification is the insatiable need for human annotation, which is very expensive. Further, for microscopic parasite images, the salient structures are generally fuzzy, and the insignificant textures are noising. This paper proposes a semi‐supervised classification method of three parasites and Erythrocytes microscopic images using a large amount of unlabeled data and a small amount of labeled data for training. It contains a feature extractor trained by contrastive learning and a classifier optimized by Laplacian Support Vector Machine (LapSVM). First, for the deep Convolutional Neural Network feature extractor, we introduce real‐world images with similar and clear semantic information to enhance the structure at the representation level. In addition, we introduce variant appearance transformations to eliminate the texture at the representation level. Second, the gated linear network is adopted as the classifier to realize a piecewise linear separation boundary. The parameters are optimized implicitly by LapSVM using a kernel function composed of the representation and the gate control signals, which are generated from the learned feature extractor. The proposed method shows excellent performance when only 1% of the microscopic images are labeled. As a result, we achieve affordable and accurate diagnostic testing in parasite classification. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

SSDAN: Multi-Source Semi-Supervised Domain Adaptation Network for Remote Sensing Scene Classification

We present a new method for multi-source semi-supervised domain adaptation in remote sensing scene classification. The method consists of a pre-trained convolutional neural network (CNN) model, namely EfficientNet-B3, for the extraction of highly discriminative features, followed by a classification module that learns feature prototypes for each class. Then, the classification module computes a cosine distance between feature vectors of target data samples and the feature prototypes. Finally, the proposed method ends with a Softmax activation function that converts the distances into class probabilities. The feature prototypes are also divided by a temperature parameter to normalize and control the classification module. The whole model is trained on both the unlabeled and labeled target samples. It is trained to predict the correct classes utilizing the standard cross-entropy loss computed over the labeled source and target samples. At the same time, the model is trained to learn domain invariant features using another loss function based on entropy computed over the unlabeled target samples. Unlike the standard cross-entropy loss, the new entropy loss function is computed on the model’s predicted probabilities and does not need the true labels. This entropy loss, called minimax loss, needs to be maximized with respect to the classification module to learn features that are domain-invariant (hence removing the data shift), and at the same time, it should be minimized with respect to the CNN feature extractor to learn discriminative features that are clustered around the class prototypes (in other words reducing intra-class variance). To accomplish these maximization and minimization processes at the same time, we use an adversarial training approach, where we alternate between the two processes. The model combines the standard cross-entropy loss and the new minimax entropy loss and optimizes them jointly. The proposed method is tested on four RS scene datasets, namely UC Merced, AID, RESISC45, and PatternNet, using two-source and three-source domain adaptation scenarios. The experimental results demonstrate the strong capability of the proposed method to achieve impressive performance despite using only a few (six in our case) labeled target samples per class. Its performance is already better than several state-of-the-art methods, including RevGrad, ADDA, Siamese-GAN, and MSCN.

Attention-aware invertible hashing network with skip connections

Abstract In recent years, Convolutional Neural Networks (CNNs) have shown promising performance on image hashing retrieval. However, due to the information-discarded nature of CNN, some meaningful information can not be further extracted into a deep level and embedded into hash codes. To solve the problem, this study attempts to design an invertible CNN feature extractor to fully maintain input information meanwhile having well generalization ability. Specifically, we propose a novel Attention-Aware Invertible Hashing Network with Skip Connection (AIHN-SC) for image retrieval. Represented by an invertible feature, the hash code can be learned and generated from image characteristics preserving all input information. For achieving favourable generalization ability in our invertible architecture, we present a novel spatial attention mechanism to highlight regions involving semantic information. In addition, we introduce two kinds of skip connection, i.e. hierarchical and residual connections, which aim to provide richer knowledges for hash code learning and ease our training process. Extensive experiments on benchmark datasets demonstrate the effectiveness of our proposed AIHN-SC and show the significant performance in image retrieval against the state-of-the-arts.

Learning CNN Filters From User-Drawn Image Markers for Coconut-Tree Image Classification

Identifying species of trees in aerial images is essential for land-use classification, plantation monitoring, and impact assessment of natural disasters. The manual identification of trees in aerial images is tedious, costly, and error-prone, so automatic classification methods are necessary. Convolutional Neural Network (CNN) models have well succeeded in image classification applications from different domains. However, CNN models usually require intensive manual annotation to create large training sets. One may conceptually divide a CNN into convolutional layers for feature extraction and fully connected layers for feature space reduction and classification. We present a method that needs a minimal set of user-selected images to train the CNN's feature extractor, reducing the number of required images to train the fully connected layers. The method learns the filters of each convolutional layer from user-drawn markers in image regions that discriminate classes, allowing better user control and understanding of the training process. It does not rely on optimization based on backpropagation, and we demonstrate its advantages on the binary classification of coconut-tree aerial images against one of the most popular CNN models.

Operational state detection in hydrocyclones with convolutional neural networks and transfer learning

In the mineral processing industries, the performance of hydrocyclones used in solids classification tasks influences the efficiency of downstream processing. Typical operation involves a fan shaped underflow profile, while undesirable roping and underflow blockage conditions result in excessive coarse particles in the overflow and have visually different underflow characteristics. Different approaches were investigated previously as potential sensing options for undesirable state detection, and while some have been commercialised, they currently have not found widespread implementation in the industry. Analysis of hydrocyclone underflow by use of image analysis has been one method investigated, in which it has been shown that meaningful features can be extracted for operational state detection.Deep convolutional neural networks have recently led to significant advances in computer vision tasks. In addition, the use of transfer learning making it possible to leverage this improved performance on smaller datasets. Video frames of the underflow of a laboratory hydrocyclone were used to produce a three-state (fan/rope/blocked) classifier, via transfer learning with a pretrained convolutional neural network feature extractor, which was able to produce meaningful state detection with the ability to handle considerable noise in the images.A dual state (fan/rope) classifier was also developed using largely industrial hydrocyclone underflow video frames, again via transfer learning with a pretrained convolutional neural network with both support vector machine and neural network based classification investigated. This study demonstrates the robust ability of deep convolutional neural networks to extract meaningful features from hydrocyclone underflow images without requiring significant image preprocessing. This is a major advantage over previous methods that rely on engineered features, and which may be more susceptible to changing environmental conditions during operation.

Few-shot cotton pest recognition and terminal realization

Xinjiang is the major cotton-producing area in China, also well known in the world for its high-quality cotton. The growth and quality of cotton are closely related to the pest attack, but it is difficult for farmers to manually recognize all the types of pests because of the similar appearance. To solve this problem, we propose a few-shot cotton pest recognition method, which only needs few raw training data, quite different from the typical deep learning methods. We use two datasets to verify the effectiveness and feasibility of the few-shot model, one is the National Bureau of Agricultural Insect Resources (NBAIR), the other is a dataset with the natural scenes. The convolutional neural network (CNN) is adopted to extract feature vectors of images. The CNN feature extractor is trained by the triplet loss to distinguish different pest species to ensure the system robustness. Furthermore, the few-shot recognition model is finally running in an embedded terminal, based on the compiled convolutional and max-pooling circuit in the FPGA and the control program in the ARM. The running speed reaches 2 frames per second and can be faster by the further parallelism in hardware. The testing accuracy for the two datasets is 95.4% and 96.2% respectively, shown the generalization ability of the proposed few-shot model. Moreover, this work can also be regarded as a positive attempt to combine the software and hardware for the landing of intelligent algorithms in the agricultural applications.

Feature learning for Human Activity Recognition using Convolutional Neural Networks

The use of Convolutional Neural Networks (CNNs) as a feature learning method for Human Activity Recognition (HAR) is becoming more and more common. Unlike conventional machine learning methods, which require domain-specific expertise, CNNs can extract features automatically. On the other hand, CNNs require a training phase, making them prone to the cold-start problem. In this work, a case study is presented where the use of a pre-trained CNN feature extractor is evaluated under realistic conditions. The case study consists of two main steps: (1) different topologies and parameters are assessed to identify the best candidate models for HAR, thus obtaining a pre-trained CNN model. The pre-trained model (2) is then employed as feature extractor evaluating its use with a large scale real-world dataset. Two CNN applications were considered: Inertial Measurement Unit (IMU) and audio based HAR. For the IMU data, balanced accuracy was 91.98% on the UCI-HAR dataset, and 67.51% on the real-world Extrasensory dataset. For the audio data, the balanced accuracy was 92.30% on the DCASE 2017 dataset, and 35.24% on the Extrasensory dataset.