Middle-level Features Research Articles

Scene-aware classifier and re-detector for thermal infrared tracking

Compared with common visible light scenes, the target of infrared scenes lacks information such as the color, texture. Infrared images have low contrast, which not only lead to interference between targets, but also interference between the target and the background. In addition, most infrared tracking algorithms lack a redetection mechanism after lost target, resulting in poor tracking effect after occlusion or blurring. To solve these problems, we propose a scene-aware classifier to dynamically adjust low, middle, and high level features, improving the ability to utilize features in different infrared scenes. Besides, we designed an infrared target re-detector based on multi-domain convolutional network to learn from the tracked target samples and background samples, improving the ability to identify the differences between the target and the background. The experimental results on VOT-TIR2015, VOT-TIR2017 and LSOTB-TIR show that the proposed algorithm achieves the most advanced results in the three infrared object tracking benchmark.

Lightweight cross-modal transformer for RGB-D salient object detection

Recently, Transformer-based RGB-D salient object detection (SOD) models have pushed the performance to a new level. However, they come at the cost of consuming abundant resources, including memory and power, thus hindering their real-life applications. To remedy this situation, a novel lightweight cross-modal Transformer (LCT) for RGB-D SOD will be presented in this paper. Specifically, LCT will first reduce its parameters and computational costs by employing a middle-level feature fusion structure and taking a lightweight Transformer as the backbone. Then, with the aid of Transformers, it will compensate for performance degradation by effectively capturing the cross-modal and cross-level complementary information from the multi-modal input images. To this end, a cross-modal enhancement and fusion module (CEFM) with a lightweight channel-wise cross attention block (LCCAB) will be designed to capture the cross-modal complementary information effectively but with fewer costs. A bi-directional multi-level feature interaction module (Bi-MFIM) with a lightweight spatial-wise cross attention block (LSCAB) will be designed to capture the cross-level complementary context information. By virtue of CEFM and Bi-MFIM, the performance degradation caused by parameter reduction can be well compensated, thus boosting the performances. By doing so, our proposed model has only 2.8M parameters with 7.6G FLOPs and runs at 66 FPS. Furthermore, experimental results on several benchmark datasets show that our proposed model can achieve competitive or even better results than other models. Our code will be released on https://github.com/nexiakele/lightweight-cross-modal-Transformer-LCT-for-RGB-D-SOD.

Deep correlation network for synthetic speech detection

Synthetic speech is becoming increasingly rampant, and automatic speaker verification (ASV) systems are vulnerable to its attacks. However, most current synthetic speech detection methods focus on the influence of a single feature in the detection. Since different features can represent the difference between real speech and synthetic speech to a certain extent, there must be common information between different types of features. Effectively finding and fully utilizing this information will facilitate the extraction of better discriminative features and achieve improved performance. Based on the above analysis, we propose a deep correlation network (DCN) to learn the latent common information between different embeddings. It consists of two parts, the bi-parallel network and the correlation learning network. Bi-parallel networks consist of different neural models to learn the middle-level representations from front-end acoustical features. The correlation learning network is the core part of the DCN and is proposed to explore the common information between the above middle-level features. The common information obtained after DCN processing have better discriminative ability for synthetic speech detection. Experimental results show that the proposed DCN can significantly improve the performance of synthetic speech detection system on ASVspoof 2019 and ASVspoof 2021 logical access sub-challenge.

LayerNet: A One-step Layered Network for Semantic Segmentation at Night.

We have collected a novel, nighttime scene dataset, called Rebecca, including 600 real images captured at night with pixel-level semantic annotations, which is currently scarce and can be invoked as a new benchmark. In addition, we proposed a one-step layered network, named LayerNet, to combine local features rich in appearance information in the shallow layer, global features abundant in semantic information in the deep layer, and middle-level features in between by explicitly model multi-stage features of objects in the nighttime. And a multi-head decoder and a well-designed hierarchical module are utilized to extract and fuse features of different depths. Numerous experiments show that our dataset can significantly improve the segmentation ability of the existing models for nighttime images. Meanwhile, our LayerNet achieves the state-of-the-art accuracy on Rebecca (65.3% mIOU). The dataset is available: https://github.com/Lihao482/REebecca.

Lightweight multi-scale attention-guided network for real-time semantic segmentation

The wide application of small mobile devices makes the demand for lightweight real-time semantic segmentation algorithm become more and more intense, which makes it become one of the most popular research topics in the field of computer vision. However, some current methods blindly pursue low parameter numbers and high inference speeds, resulting in excessively low model accuracy and a loss of practical value. Therefore, a lightweight multi-scale attention-guided network for real-time semantic segmentation(LMANet) based on asymmetric encoder-decoder is proposed in this paper to solve the above dilemmas. In the encoder, we propose multi-scale asymmetric residual(MAR) modules to extract local spatial information and context information to enhance feature expression. In the decoder, we design an attention feature fusion(AFF) module and an attention pyramid refining (APR) module. AFF module guides the fusion of low-level and middle-level feature information through high-level semantic information, and finally refines the fusion result through APR module. In addition, we improve the segmentation performance of the model with the help of the attention modules in the network. Our network is tested on two complex urban road datasets. The experimental results show that LMANet achieves 70.6% mIoU and 66.5% mIoU on Cityscapes and Camvid datasets at 112FPS and 333FPS respectively, only 0.95 M parameters without any pre-training or pre-processing. Compared with most of existing state-of-the-art models, our network not only guarantees reasonable inference speed and parameter quantity, but also improves the accuracy as much as possible, which makes it more practical.

A small sample nonstandard gear surface defect detection method

To address the issues of low detection precision, slow detection speed, and imbalanced defect samples in the surface defect detection of nonstandard parts, we propose a method based on the defect detection model YOLOv8-2d and the balancing generative adversarial network (BAGAN). First, we use an attention mechanism without parameters to help the detection model focus on the defect area in the middle-level features based on the feature scale and distribution characteristics of surface defects on nonstandard parts. Second, we introduce deformable convolution and propose a path aggregation feature pyramid network (PADSFPN) with depth-separable convolution, effectively integrating multiscale feature information. Then, we use a Mish network with WIoU as the network head and extract the position and category information of the defect separately, learn through different network branches, and merge the information. Finally, we use BAGAN to expand the small sample defect dataset on the surface of nonstandard parts to achieve dataset balance. The proposed improvements are evaluated through comparative experiments in published twenty test groups, and the effectiveness of the proposed method is demonstrated. Compared with the original model, our method achieves a defect detection precision of 99.83%, a 7.7% improvement; an inference speed of 455 FPS, a 374% increase; and a mAP50 of 97.3%, a gain of 5.03%. Our proposed method maintains high precision while reducing the model's computational complexity and parameter volume, improving its efficiency and robustness in surface defect detection on nonstandard parts.

Image emotion analysis based on semantic concepts

With the increasing number of users express their emotions via images on social media, image emotion analysis attracts much attention of researchers. For the ambiguity and subjectivity of emotion, image emotion analysis is more challenging than other computer vision tasks. Previous methods merely learn a direct mapping between image feature and emotion. However, in emotion perception theory of psychology, it is demonstrated that human beings perceive emotion in a stepwise way. Therefore, we propose a novel image emotion analysis framework that makes use of emotional concepts as middle-level feature to bridge image and emotion. Firstly, the relationship between the concept and the emotion is organized in the form of knowledge graph. The relation between the image and the emotion in the semantic embedding space is explored where the knowledge is encoded into. On the other hand, a multi-level deep metric learning method to optimize the model from both label level and instance level is proposed. Extensive experimental results on two image emotion datasets, demonstrate that the present approach performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks.

Dual Learning for Joint Facial Landmark Detection and Action Unit Recognition

Facial landmark detection and action unit~(AU) recognition are two essential tasks in facial analysis. Previous works rarely consider the relationship between these complementary tasks. In this paper, we introduce a novel multi-task dual learning framework to exploit the relationship between facial landmark detection and AU recognition while simultaneously addressing both tasks. When both tasks share middle-level features, common patterns can be exploited and middle- and high-level features can be used to perform facial landmark detection and AU recognition, respectively. In addition, a dual learning mechanism is designed to convert the predicted landmarks and AUs of the label space to the corresponding facial image of the image space, further exploring the strong correlations between the tasks. By jointly training the proposed method at both the feature and label levels, each task improves the other. Experiments on two benchmark databases demonstrate that the proposed method can leverage dependencies to boost the generalization of both tasks.

RGB-T Semantic Segmentation With Location, Activation, and Sharpening

Semantic segmentation is important for scene understanding. To address the scenes of adverse illumination conditions of natural images, thermal infrared (TIR) images are introduced. Most existing RGB-T semantic segmentation methods follow three cross-modal fusion paradigms, i.e. encoder fusion, decoder fusion, and feature fusion. Some methods, unfortunately, ignore the properties of RGB and TIR features or the properties of features at different levels. In this paper, we propose a novel feature fusion-based network for RGB-T semantic segmentation, named \emph{LASNet}, which follows three steps of location, activation, and sharpening. The highlight of LASNet is that we fully consider the characteristics of cross-modal features at different levels, and accordingly propose three specific modules for better segmentation. Concretely, we propose a Collaborative Location Module (CLM) for high-level semantic features, aiming to locate all potential objects. We propose a Complementary Activation Module for middle-level features, aiming to activate exact regions of different objects. We propose an Edge Sharpening Module (ESM) for low-level texture features, aiming to sharpen the edges of objects. Furthermore, in the training phase, we attach a location supervision and an edge supervision after CLM and ESM, respectively, and impose two semantic supervisions in the decoder part to facilitate network convergence. Experimental results on two public datasets demonstrate that the superiority of our LASNet over relevant state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/LASNet.

CCTseg: A cascade composite transformer semantic segmentation network for UAV visual perception

Semantic segmentation could obtain the pixel level classification of surrounding environments which is an essential task for autonomous vehicles and mobile robots visual perception. Most existing semantic segmentation networks were focused on the visual perception of autonomous vehicles. Little attention is paid to the semantic segmentation for UAV (Unmanned Aerial Vehicle) visual perception, which is crucial to UAV autonomous flight and landing spot searching. Compared with views from autonomous vehicles, the UAV-based views were more challenging for the semantic segmentation task due to images captured by UAV containing large-scale variation of objects size caused by different altitude and angle. The existing semantic segmentation networks for the visual perception of autonomous vehicles are generally inadequate to effectively extract the representative features of UAV images which required contain context information and local information simultaneously. A cascade composite transformer-based semantic segmentation network is proposed in this study for UAV visual perception. A cascade composite encoder is designed which consists of three transformer-based feature extraction backbones and cascade fused low-level features, middle-level features and high-level features to achieve better feature representation capacity. The spatial enhanced transformer block is implemented as the basic feature extraction block of each backbone to make the extracted features contain context information of environments and local information of objects. A symmetric rhombus decoder is proposed to integrate multi-stage features and make fully utilise of middle stage features which contained abundance of useful information, thus accurately pixel level prediction could be obtained in this way. Ablation studies and comparison experiments for the proposed CCTseg have been conducted on two public UAV imagery datasets suitable for UAV autonomous flight and landing spot observing. Experimental results have demonstrated the effectiveness of the proposed network structure and the superiority of proposed network over other state-of-the-art methods for the semantic segmentation of UAV visual perception.

Feature relocation network for fine-grained image classification

In fine-grained image classification, there are only very subtle differences between classes. It is challenging to learn local discriminative features and remove distractive features in fine-grained image classification. Existing fine-grained image classification methods learn discriminative feature mainly via manual part annotation or attention mechanisms. However, due to the large intraclass variance and interclass similarity, the discriminative information and distractive information still are not distinguished effectively. To address this problem, we propose a feature relocation network (FRe-Net) which takes advantage of the different natures of features learned from different stages of the network. Our network consists of a distractive feature learning module and a relocated high-level feature learning module. In the distractive feature learning module, we propose to exploit the difference between low-level features and high-level features to design a distractive loss Ldistractive, which guides the attention to locate distractive regions more accurately. In the relocated high-level feature learning module, we enhance the representing capacity of the middle-level feature via the attention module and subtract the distractive feature learned from the distractive feature learning module in order to learn more local discriminative features. In end-to-end model training, the distractive feature learning module and the relocated high-level feature learning module are beneficial to each other via joint optimization. We conducted comprehensive experiments on three benchmark datasets widely used in fine-grained image classification. The experimental results show that FRe-Net achieves state-of-the-art performance, which validates the effectiveness of FRe-Net.

A lightweight network for smoke semantic segmentation

To obtain real-time performance on computation limited devices, we propose a lightweight network for smoke segmentation. To enhance the ability of feature encoding, we first propose an Attention Encoding Module (AEM) by designing a Channel Split and Shuffle Attention Module (CSSAM), which can extract powerful features and reduce computations simultaneously. CSSAM adopts Channel split and shuffle to greatly reduce learnable parameters for improving computation speed, and uses attention mechanism to focus on salient objects to enhance the effectiveness of features. In addition, AEM repeatedly stacks CSSAM in different encoding stages to achieve scale invariance. For the middle-level features of encoding stages, we propose a Spatial Enhancement Module (SEM) to boost the representation ability of spatial details. SEM concatenates feature maps produced by average and maximum pooling to achieve dominant and global responses, which are then weighted by the activated output of global average pooling to generate attention features. In the highest level of encoding stages, we present a Channel Attention Module (CAM) to explicitly model interdependency between channels. By reshaping 2D features into 1D features, we use element-wise matrix multiplications to reduce computation complexity for extracting channel-related information. Finally, we design a Feature Fusion Module (FFM) and a Global Coefficient Path (GCP) to fuse the outputs of SEM and CAM in an attention way for further improving robustness of final features. Experiments show that our method is significantly superior to existing state-of-the-art algorithms in smoke datasets, and also obtains excellent results in both synthetic and real smoke datasets. However, our method has less than 1 M network parameters.

Progressive Local Filter Pruning for Image Retrieval Acceleration

This paper focuses on network pruning for image retrieval acceleration. Prevailing image retrieval works target at the discriminative feature learning, while little attention is paid to how to accelerate the model inference, which should be taken into consideration in real-world practice. The challenge of pruning image retrieval models is that the middle-level feature should be preserved as much as possible. Such different requirements of the retrieval and classification model make the traditional pruning methods not that suitable for our task. To solve the problem, we propose a new Progressive Local Filter Pruning (PLFP) method for image retrieval acceleration. Specifically, layer by layer, we analyze the local geometric properties of each filter and select the one that can be replaced by the neighbors. Then we progressively prune the filter by gradually changing the filter weights. In this way, the representation ability of the model is preserved. To verify this, we evaluate our method on two widely-used image retrieval datasets,i.e., Oxford5k and Paris6K, and one person re-identification dataset,i.e., Market-1501. The proposed method arrives with superior performance to the conventional pruning methods, suggesting the effectiveness of the proposed method for image retrieval.

Salient Object Detection in Optical Remote Sensing Images Driven by Transformer.

Existing methods for Salient Object Detection in Optical Remote Sensing Images (ORSI-SOD) mainly adopt Convolutional Neural Networks (CNNs) as the backbone, such as VGG and ResNet. Since CNNs can only extract features within certain receptive fields, most ORSI-SOD methods generally follow the local-to-contextual paradigm. In this paper, we propose a novel Global Extraction Local Exploration Network (GeleNet) for ORSI-SOD following the global-to-local paradigm. Specifically, GeleNet first adopts a transformer backbone to generate four-level feature embeddings with global long-range dependencies. Then, GeleNet employs a Direction-aware Shuffle Weighted Spatial Attention Module (D-SWSAM) and its simplified version (SWSAM) to enhance local interactions, and a Knowledge Transfer Module (KTM) to further enhance cross-level contextual interactions. D-SWSAM comprehensively perceives the orientation information in the lowest-level features through directional convolutions to adapt to various orientations of salient objects in ORSIs, and effectively enhances the details of salient objects with an improved attention mechanism. SWSAM discards the direction-aware part of D-SWSAM to focus on localizing salient objects in the highest-level features. KTM models the contextual correlation knowledge of two middle-level features of different scales based on the self-attention mechanism, and transfers the knowledge to the raw features to generate more discriminative features. Finally, a saliency predictor is used to generate the saliency map based on the outputs of the above three modules. Extensive experiments on three public datasets demonstrate that the proposed GeleNet outperforms relevant state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/GeleNet.

Enhanced Feature Model Based Hybrid Neural Network for Text Detection on Signboard, Billboard and News Tickers

Recognizing text from the nature scene images and videos has been the challenging task of computer vision and machine learning research community in recent years. These texts are difficult to recognize because of their shapes, complex backgrounds, color, shape and size variations. However, text recognition is very much useful in indexing, keyword-based image and video search, and information retrieval. In this research paper, a model is proposed to detect the isolated text characters in the photographic images of natural scenes. The proposed model uses the combination of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) for recognizing the text in natural images. The model uses two networks, where the first network combines the low-level and middle-level features to increase the feature size and passes the enriched information to the second network. Here, features are again widened by combining with high-level features, resulting in powerful and robust features. To evaluate the proposed model, ICDAR2003 (IC03), ICDAR2013 (IC13), SVT (Street View Text) datasets have been used. And an extensive Tamil news tickers image dataset has been developed to evaluate the model. The experimental results show that the combined feature fusion technique outperforms the other methods on the ICDAR2003, ICDAR2013, SVT and Tamil news tickers datasets.

CCFNet: Cross-Complementary fusion network for RGB-D scene parsing of clothing images

Schemes to complement context relationships by cross-scale feature fusion have appeared in many RGB-D scene parsing algorithms; however, most of these works conduct multi-scale information interaction after multi-modal feature fusion, which ignores the information loss of the two modes in the original coding. Therefore, a cross-complementary fusion network (CCFNet) is designed in this paper to calibrate the multi-modal information before feature fusion, so as to improve the feature quality of each mode and the information complementarity ability of RGB and the depth map. First, we divided the features into low, middle, and high levels, among which the low-level features contain the global details of the image and the main learning features include texture, edge, and other features. The middle layer features contain not only some global detail features but also some local semantic features. Additionally, the high-level features contain rich local semantic features. Then, the feature information lost in the coding process of low and middle level features is supplemented and extracted through the designed cross feature enhancement module, and the high-level features are extracted through the feature enhancement module. In addition, the cross-modal fusion module is designed to integrate multi-modal features of different levels. The experimental results verify that the proposed CCFNet achieves excellent performance on the RGB-D scene parsing dataset containing clothing images, and the generalization ability of the model is verified by the dataset NYU Depth V2.

Two-Stage Selective Ensemble of CNN via Deep Tree Training for Medical Image Classification.

Medical image classification is an important task in computer-aided diagnosis systems. Its performance is critically determined by the descriptiveness and discriminative power of features extracted from images. With rapid development of deep learning, deep convolutional neural networks (CNNs) have been widely used to learn the optimal high-level features from the raw pixels of images for a given classification task. However, due to the limited amount of labeled medical images with certain quality distortions, such techniques crucially suffer from the training difficulties, including overfitting, local optimums, and vanishing gradients. To solve these problems, in this article, we propose a two-stage selective ensemble of CNN branches via a novel training strategy called deep tree training (DTT). In our approach, DTT is adopted to jointly train a series of networks constructed from the hidden layers of CNN in a hierarchical manner, leading to the advantage that vanishing gradients can be mitigated by supplementing gradients for hidden layers of CNN, and intrinsically obtain the base classifiers on the middle-level features with minimum computation burden for an ensemble solution. Moreover, the CNN branches as base learners are combined into the optimal classifier via the proposed two-stage selective ensemble approach based on both accuracy and diversity criteria. Extensive experiments on CIFAR-10 benchmark and two specific medical image datasets illustrate that our approach achieves better performance in terms of accuracy, sensitivity, specificity, and F1 score measurement.

COVIDNet: An Automatic Architecture for COVID-19 Detection With Deep Learning From Chest X-Ray Images

Up to now, the coronavirus disease 2019 (COVID-19) has been sweeping across all over the world, which has affected individual’s lives in an overwhelming way. To fight efficiently against the COVID-19, radiography and radiology images are used by clinicians in hospitals. This article presents an integrated framework, named COVIDNet, for classifying COVID-19 patients and healthy controls. Specifically, ResNet (i.e., ResNet-18 and ResNet-50) is adopted as a backbone network to extract the discriminative features first. Second, the spatial pyramid pooling (SPP) layer is adopted to capture the middle-level features from the features of ResNet. To learn the high-level features, the NetVLAD layer is used to aggregate the features representation from middle-level features. The context gating (CG) mechanism is adopted to further learn the high-level features for predicting the COVID-19 patients or not. Finally, extensive experiments are conducted on the collected database, showing the excellent performance of the proposed integrated architecture, with the sensitivity up to 97% and specificity of 99.5% of the ResNet-18, and with the sensitivity up to 99% and specificity of 99.4% of the ResNet-50.

A New Spatial–Temporal Depthwise Separable Convolutional Fusion Network for Generating Landsat 8-Day Surface Reflectance Time Series over Forest Regions

Landsat has provided the longest fine resolution data archive of Earth’s environment since 1972; however, one of the challenges in using Landsat data for various applications is its frequent large data gaps and heavy cloud contaminations. One pressing research topic is to generate the regular time series by integrating coarse-resolution satellite data through data fusion techniques. This study presents a novel spatiotemporal fusion (STF) method based on a depthwise separable convolutional neural network (DSC), namely, STFDSC, to generate Landsat-surface reflectance time series at 8-day intervals by fusing Landsat 30 m with high-quality Moderate Resolution Imaging Spectroradiometer (MODIS) 500 m surface reflectance data. The STFDSC method consists of three main stages: feature extraction, feature fusion and prediction. Features were first extracted from Landsat and MODIS surface reflectance changes, and the extracted multilevel features were then stacked and fused. Both low-level and middle-level features that were generally ignored in convolutional neural network (CNN)-based fusion models were included in STFDSC to avoid key information loss and thus ensure high prediction accuracy. The prediction stage generated a Landsat residual image and is combined with original Landsat data to obtain predictions of Landsat imagery at the target date. The performance of STFDSC was evaluated in the Greater Khingan Mountains (GKM) in Northeast China and the Ziwuling (ZWL) forest region in Northwest China. A comparison of STFDSC with four published fusion methods, including two classic fusion methods (FSDAF, ESTARFM) and two machine learning methods (EDCSTFN and STFNET), was also carried out. The results showed that STFDSC made stable and more accurate predictions of Landsat surface reflectance than other methods in both the GKM and ZWL regions. The root-mean-square-errors (RMSEs) of TM bands 2, 3, 4, and 7 were 0.0046, 0.0038, 0.0143, and 0.0055 in GKM, respectively, and 0.0246, 0.0176, 0.0280, and 0.0141 in ZWL, respectively; it can be potentially used for generating the global surface reflectance and other high-level land products.

A Combination of Lie Group Machine Learning and Deep Learning for Remote Sensing Scene Classification Using Multi-Layer Heterogeneous Feature Extraction and Fusion

Discriminative feature learning is the key to remote sensing scene classification. Previous research has found that most of the existing convolutional neural networks (CNN) focus on the global semantic features and ignore shallower features (low-level and middle-level features). This study proposes a novel Lie Group deep learning model for remote sensing scene classification to solve the above-mentioned challenges. Firstly, we extract shallower and higher-level features from images based on Lie Group machine learning (LGML) and deep learning to improve the feature representation ability of the model. In addition, a parallel dilated convolution, a kernel decomposition, and a Lie Group kernel function are adopted to reduce the model’s parameters to prevent model degradation and over-fitting caused by the deepening of the model. Then, the spatial attention mechanism can enhance local semantic features and suppress irrelevant feature information. Finally, feature-level fusion is adopted to reduce redundant features and improve computational performance, and cross-entropy loss function based on label smoothing is used to improve the classification accuracy of the model. Comparative experiments on three public and challenging large-scale remote-sensing datasets show that our model improves the discriminative ability of features and achieves competitive accuracy against other state-of-the-art methods.