Low-level Feature Representation Research Articles

DATFNets-dynamic adaptive assigned transformer network for fire detection

Fires cause severe damage to the ecological environment and threaten human life and property. Although the traditional convolutional neural network method effectively detects large-area fires, it cannot capture small fires in complex areas through a limited receptive field. At the same time, fires can change at any time due to the influence of wind direction, which challenges fire prevention and control personnel. To solve these problems, a novel dynamic adaptive distribution transformer detection framework is proposed to help firefighters and researchers develop optimal fire management strategies. On the one hand, this framework embeds a context aggregation layer with a masking strategy in the feature extractor to improve the representation of low-level and salient features. The masking strategy can reduce irrelevant information and improve network generalization. On the other hand, designed a dynamic adaptive direction conversion function and sample allocation strategy to fully use adaptive point representation while achieving accurate positioning and classification of fires and screening out representative fire samples in complex backgrounds. In addition, to prevent the network from being limited to the local optimum and discrete points in the sample from causing severe interference to the overall performance, designed a weighted loss function with spatial constraints to optimize the network and penalize the discrete points in the sample. The mAP in the three baseline data sets of FireDets, WildFurgFires, and FireAndSmokes are 0.871, 0.909, and 0.955, respectively. The experimental results are significantly better than other detection methods, which proves that the proposed method has good robustness and detection performance.

3D seismic mask auto encoder: Seismic inversion using transformer-based reconstruction representation learning

Estimating acoustic impedance from seismic data is a crucial step in reservoir characterization. While data-driven impedance inversion based on deep learning has shown promising results, it relies heavily on extensive well logs for labeling, which is often impractical in many exploration scenarios. Recently, the zero-shot or few-shot learning performance of Pretrained Foundation Models like Generative Pre-trained Transformer (GPT) and Mask Auto Encoder (MAE) has highlighted that knowledge learned from vast amounts of unlabeled data can be transferred to downstream tasks with minimal labeled data. However, applying Transformer-based representation learning models to 3D seismic data inversion poses three challenges: (1) Computational and memory constraints due to the high-dimensional nature of the data; (2) Difficulty in extracting fine-grained image features using Transformers, hampering high-frequency impedance inversion; (3) Fixed input size in Transformers, leading to inversion artifacts. In this work, we introduce the Seismic Mask Auto Encoder (SeisMAE), a Transformer-based representation model tailored for the inversion of 3D seismic data. It incorporates three key components: (1) Aggregated dimensionality reduction encoding to handle redundancy in seismic data, significantly improving computational efficiency; (2) Multi-scale self-attention feature fusion to enhance the model’s capacity for low-level feature representation; and (3) A stitching decoding strategy to eliminate inversion stitching artifacts. Experimental validations highlight the efficacy of our approach. On the synthetic SEAM I dataset, we demonstrate the effectiveness of each component and SeisMAE’s superior performance. For real-world data on The Netherlands F3, SeisMAE delivers reliable inversion outcomes with only four labeled examples. We compared SeisMAE against various inversion techniques, including 1D Convolutional Neural Network (1D-CNN), UNet-based, HRNet-based, and TransInver, where SeisMAE exhibited significant advantages.

Motion direction is represented as a bimodal probability distribution in the human visual cortex

Humans infer motion direction from noisy sensory signals. We hypothesize that to make these inferences more precise, the visual system computes motion direction not only from velocity but also spatial orientation signals – a ‘streak’ created by moving objects. We implement this hypothesis in a Bayesian model, which quantifies knowledge with probability distributions, and test its predictions using psychophysics and fMRI. Using a probabilistic pattern-based analysis, we decode probability distributions of motion direction from trial-by-trial activity in the human visual cortex. Corroborating the predictions, the decoded distributions have a bimodal shape, with peaks that predict the direction and magnitude of behavioral errors. Interestingly, we observe similar bimodality in the distribution of the observers’ behavioral responses across trials. Together, these results suggest that observers use spatial orientation signals when estimating motion direction. More broadly, our findings indicate that the cortical representation of low-level visual features, such as motion direction, can reflect a combination of several qualitatively distinct signals.

Hierarchical Multi-Attention Transfer for Knowledge Distillation

Knowledge distillation (KD) is a powerful and widely applicable technique for the compression of deep learning models. The main idea of knowledge distillation is to transfer knowledge from a large teacher model to a small student model, where the attention mechanism has been intensively explored in regard to its great flexibility for managing different teacher-student architectures. However, existing attention-based methods usually transfer similar attention knowledge from the intermediate layers of deep neural networks, leaving the hierarchical structure of deep representation learning poorly investigated for knowledge distillation. In this paper, we propose a hierarchical multi-attention transfer framework (HMAT) , where different types of attention are utilized to transfer the knowledge at different levels of deep representation learning for knowledge distillation. Specifically, position-based and channel-based attention knowledge characterize the knowledge from low-level and high-level feature representations, respectively, and activation-based attention knowledge characterize the knowledge from both mid-level and high-level feature representations. Extensive experiments on three popular visual recognition tasks, image classification, image retrieval, and object detection, demonstrate that the proposed hierarchical multi-attention transfer or HMAT significantly outperforms recent state-of-the-art KD methods.

MCSF-Net: a multi-scale channel spatial fusion network for real-time polyp segmentation.

Colorectal cancer is a globally prevalent cancer type that necessitates prompt screening. Colonoscopy is the established diagnostic technique for identifying colorectal polyps. However, missed polyp rates remain a concern. Early detection of polyps, while still precancerous, is vital for minimizing cancer-related mortality and economic impact. In the clinical setting, precise segmentation of polyps from colonoscopy images can provide valuable diagnostic and surgical information. Recent advances in computer-aided diagnostic systems, specifically those based on deep learning techniques, have shown promise in improving the detection rates of missed polyps, and thereby assisting gastroenterologists in improving polyp identification. In the present investigation, we introduce MCSF-Net, a real-time automatic segmentation framework that utilizes a multi-scale channel space fusion network. The proposed architecture leverages a multi-scale fusion module in conjunction with spatial and channel attention mechanisms to effectively amalgamate high-dimensional multi-scale features. Additionally, a feature complementation module is employed to extract boundary cues from low-dimensional features, facilitating enhanced representation of low-level features while keeping computational complexity to a minimum. Furthermore, we incorporate shape blocks to facilitate better model supervision for precise identification of boundary features of polyps. Our extensive evaluation of the proposed MCSF-Net on five publicly available benchmark datasets reveals that it outperforms several existing state-of-the-art approaches with respect to different evaluation metrics. The proposed approach runs at an impressive ∼45 FPS, demonstrating notable advantages in terms of scalability and real-time segmentation.

Enhancing Dynamic Hand Gesture Recognition using Feature Concatenation via Multi-Input Hybrid Model

Radar-based hand gesture recognition is an important research area that provides suitable support for various applications, such as human-computer interaction and healthcare monitoring. Several deep learning algorithms for gesture recognition using Impulse Radio Ultra-Wide Band (IR-UWB) have been proposed. Most of them focus on achieving high performance, which requires a huge amount of data. The procedure of acquiring and annotating data remains a complex, costly, and time-consuming task. Moreover, processing a large volume of data usually requires a complex model with very large training parameters, high computation, and memory consumption. To overcome these shortcomings, we propose a simple data processing approach along with a lightweight multi-input hybrid model structure to enhance performance. We aim to improve the existing state-of-the-art results obtained using an available IR-UWB gesture dataset consisting of range-time images of dynamic hand gestures. First, these images are extended using the Sobel filter, which generates low-level feature representations for each sample. These represent the gradient images in the x-direction, the y-direction, and both the x- and y-directions. Next, we apply these representations as inputs to a three-input Convolutional Neural Network- Long Short-Term Memory- Support Vector Machine (CNN-LSTM-SVM) model. Each one is provided to a separate CNN branch and then concatenated for further processing by the LSTM. This combination allows for the automatic extraction of richer spatiotemporal features of the target with no manual engineering approach or prior domain knowledge. To select the optimal classifier for our model and achieve a high recognition rate, the SVM hyperparameters are tuned using the Optuna framework. Our proposed multi-input hybrid model achieved high performance on several parameters, including 98.27% accuracy, 98.30% precision, 98.29% recall, and 98.27% F1-score while ensuring low complexity. Experimental results indicate that the proposed approach improves accuracy and prevents the model from overfitting.

Self-attention enhanced deep residual network for spatial image steganalysis

As a specially designed tool and technique for the detection of image steganography, image steganalysis conceals information under the carriers for covert communications. Being developed on the BOSSbase dataset and released a decade ago, most of the Convolutional Neural Network (CNN) architectures for spatial image steganalysis fail to achieve satisfactory performance on new challenging datasets, i.e. ALASKA#2, which was released recently and is more complex yet consistent with the real scenarios. In this paper, we propose an enhanced residual network (ERANet) with self-attention ability, which utilizes a more complex residual method and a global self-attention technique, to alleviate the problem. Compared to the residual network that was widely used in the state-of-the-art, the enhanced residual network mathematically employed a more sophisticated way to extract more effective features in the images and hence it is suitable for more complex situations in the new dataset. Our proposed Enhanced Low-Level Feature Representation Module can be easily mounted on other CNNs in selecting the most representative features. Although it comes with a slightly extra computational cost, comprehensive experiments on the BOSSbase and ALASKA#2 datasets at various sizes have demonstrated the effectiveness of the proposed methodology. In short, ERANet provides an improvement of about 3.77% on average, compared to a few state-of-the-art CNNs.

Feature Fusion Pyramid Network for End-to-end Scene Text Detection

How to properly involve text characteristics like multi-scale, arbitrary direction, length aspect ratio, into detection network design has become a hot topic in computer vision. Feature Pyramid Network (FPN) is a typical method to achieve robust text detection, where its low-level and high-level feature map retains spatial structure and global semantic information, respectively. However, its strict hierarchical structure fails to fuse low-level and high-level information to improve distinguish ability of feature map. To address this problem, we propose a novel feature fusion pyramid network for end-to-end scene text detection by fusing multi-modal information. By diving pyramid structure into high-level and low-level layers, channel and spatial attention modules are adopted to enhance high-level and low-level feature representation by encoding channel and spatial -wise context information, respectively. In order to reduce information loss by layer transmission, a special residual network is designed to achieve short-cut between high-level and low-level features, so as to realize multi-modal feature fusion. Experiments show the precision and recall of the propose method on ICDAR2015, ICDAR2017-MLT and MSRA-TD500 datasets reach 88.7%/82.1%, 77.0%/60.3% and 85.3%/74.8%, respectively.

Remote Sensing Image Classification: A Comprehensive Review and Applications

Remote sensing is mainly used to investigate sites of dams, bridges, and pipelines to locate construction materials and provide detailed geographic information. In remote sensing image analysis, the images captured through satellite and drones are used to observe surface of the Earth. The main aim of any image classification-based system is to assign semantic labels to captured images, and consequently, using these labels, images can be arranged in a semantic order. The semantic arrangement of images is used in various domains of digital image processing and computer vision such as remote sensing, image retrieval, object recognition, image annotation, scene analysis, content-based image analysis, and video analysis. The earlier approaches for remote sensing image analysis are based on low-level and mid-level feature extraction and representation. These techniques have shown good performance by using different feature combinations and machine learning approaches. These earlier approaches have used small-scale image dataset. The recent trends for remote sensing image analysis are shifted to the use of deep learning model. Various hybrid approaches of deep learning have shown much better results than the use of a single deep learning model. In this review article, a detailed overview of the past trends is presented, based on low-level and mid-level feature representation using traditional machine learning concepts. A summary of publicly available image benchmarks for remote sensing image analysis is also presented. A detailed summary is presented at the end of each section. An overview regarding the current trends of deep learning models is presented along with a detailed comparison of various hybrid approaches based on recent trends. The performance evaluation metrics are also discussed. This review article provides a detailed knowledge related to the existing trends in remote sensing image classification and possible future research directions.

Adaptive time-frequency feature resolution network for acoustic scene classification

Convolutional neural networks with time-frequency feature representation for acoustic scene classification have been attracting increasing attentions. However, most of the existing methods are restricted to design a plain discriminative model with low-level feature representations and hard to leverage a priori knowledge of sound events in a particular scene. This provides some leeway to boost the performance. In this paper, Adaptive time-frequency Feature Resolution Network (AdaFRN) is proposed to embrace the progress in attention mechanism, feature fusion, and very deep architecture. Specifically, a multi-resolution attention distributor is applied to automatically select a proper feature resolution to capture remarkable events and a bi-direction feature pyramid network is utilized to transmit information across multiple scales. Extensive experiments demonstrate that the proposed model yields better results than state-of-the-art methods. With the normalized mixup strategy, our model outperforms the state-of-the-art method by 1.6%.

CutResize: Improved data augmentation method for RGB-D Object Recognition

RGB-D object recognition is an essential but challenging task in computer vision. Most existing methods often exploit popular neural network architectures and various multi-modal fusion manners, which fail to explore the relation between low-level and high-level feature representations. To address this issue, we propose a convolutional recurrent fusion method for RGB-D object recognition. Firstly, a novel regularization strategy, called CutResize, is utilized to generate more reliable multi-scale RGB-D fusion input. Secondly, a convolutional attention gate recurrent unit (ConvAGRU) is proposed to fuse multi-modal features, which can capture the relation between low-level and high-level feature representations. Finally, the network recognizes RGB-D objects in an end-to-end manner. Especially, the CutResize uses the scaled image from another patch to mix with the input image, which avoids the mixed image only contains background and improves the localization ability of the model for small-scale features. The proposed ConvAGRU enhances the context correlation of the network and produces more compact and robust multi-modal fusion features. The proposed method is evaluated on several public datasets (including CIFAR-10, CIFAR-100, JHUIT-50, and OCID datasets). The experiment results show that the proposed method obtains better recognition performance than existing methods and achieves state-of-the-art results.

Dual-Attention-Driven Multiscale Fusion Object Searching Network for Remote Sensing Imagery

Object search is a challenging yet important task. Many efforts have been made to address this issue and achieve great progress in natural image, yet searching all specified types of objects from remote sensing image is barely studied. In this work, we are interested in searching objects from remote sensing images. Compared to person search in natural scenes, this task is challenging in two factors: One is that remote image usually contains a large number of objects, which poses a great challenge to characterize the object features. Another is that the objects in remote sensing images are dense, which easily yields erroneous localization. To address these issues, we propose a new end-toend deep learning framework for object search in remote sensing images. First, we propose a multi-scale feature aggregation (MSFA) module, which strengthens the representation of lowlevel features by fusing multi-layer features. The fused features with richer details significantly improve the accuracy of object search. Second, we propose a dual-attention object enhancement (DAOE) module to enhance features from channel and spatial dimensions. The enhanced features significantly improve the localization accuracy for dense objects. Finally, we built two challenging datasets based on the remote sensing images, which contains complex changes in space and time. The experiments and comparisons demonstrate the state-of-the-art performance of our method on the challenging datasets.

COVID-MTL: Multitask learning with Shift3D and random-weighted loss for COVID-19 diagnosis and severity assessment

There is an urgent need for automated methods to assist accurate and effective assessment of COVID-19. Radiology and nucleic acid test (NAT) are complementary COVID-19 diagnosis methods. In this paper, we present an end-to-end multitask learning (MTL) framework (COVID-MTL) that is capable of automated and simultaneous detection (against both radiology and NAT) and severity assessment of COVID-19. COVID-MTL learns different COVID-19 tasks in parallel through our novel random-weighted loss function, which assigns learning weights under Dirichlet distribution to prevent task dominance; our new 3D real-time augmentation algorithm (Shift3D) introduces space variances for 3D CNN components by shifting low-level feature representations of volumetric inputs in three dimensions; thereby, the MTL framework is able to accelerate convergence and improve joint learning performance compared to single-task models. By only using chest CT scans, COVID-MTL was trained on 930 CT scans and tested on separate 399 cases. COVID-MTL achieved AUCs of 0.939 and 0.846, and accuracies of 90.23% and 79.20% for detection of COVID-19 against radiology and NAT, respectively, which outperformed the state-of-the-art models. Meanwhile, COVID-MTL yielded AUC of 0.800 ± 0.020 and 0.813 ± 0.021 (with transfer learning) for classifying control/suspected, mild/regular, and severe/critically-ill cases. To decipher the recognition mechanism, we also identified high-throughput lung features that were significantly related (P < 0.001) to the positivity and severity of COVID-19.

Detection of COVID-19 from voice, cough and breathing patterns: Dataset and preliminary results

COVID-19 heavily affects breathing and voice and causes symptoms that make patients’ voices distinctive, creating recognizable audio signatures. Initial studies have already suggested the potential of using voice as a screening solution. In this article we present a dataset of voice, cough and breathing audio recordings collected from individuals infected by SARS-CoV-2 virus, as well as non-infected subjects via large scale crowdsourced campaign. We describe preliminary results for detection of COVID-19 from cough patterns using standard acoustic features sets, wavelet scattering features and deep audio embeddings extracted from low-level feature representations (VGGish and OpenL3). Our models achieve accuracy of 88.52%, sensitivity of 88.75% and specificity of 90.87%, confirming the applicability of audio signatures to identify COVID-19 symptoms. We furthermore provide an in-depth analysis of the most informative acoustic features and try to elucidate the mechanisms that alter the acoustic characteristics of coughs of people with COVID-19.

Robust Continuous Hand Motion Recognition Using Wearable Array Myoelectric Sensor

With the advantages of comfortable wearing and outdoor usage, the myoelectric gesture recognition techniques have gained much attention in the field of human-machine interaction (HMI). The purpose of this study is to optimize model structure and transfer generalized features to improve the robustness of myoelectric hand motion decoding. We derived the hand motion recognition framework from the muscle synergy theory, which is formulated as a temporal convolutional (TC) model of array sEMG signals, then a hierarchical myoelectric decoding model was proposed to predict simultaneous and continuous hand motion. The model was trained by the methods of unsupervised low-level feature learning and automated data labeling to minimize training supervision. Extensive experiments on the public sEMG database (17 subjects in Biopatrec) show that the TC model can extract muscle synergy features with higher fidelity ( R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.85±0.23) than the traditional instantaneous mixture model, the results of online test demonstrate robust myoelectric decoding on multiple simultaneous and continuous hand motions. More importantly, the analysis of weights visualization shows that the low-level feature representation layer of TC model can be migrated across the individuals, which provides a transferrable feature extraction layer for generalized hand motion decoding.

Satellite and Scene Image Classification Based on Transfer Learning and Fine Tuning of ResNet50

Image classification has gained lot of attention due to its application in different computer vision tasks such as remote sensing, scene analysis, surveillance, object detection, and image retrieval. The primary goal of image classification is to assign the class labels to images according to the image contents. The applications of image classification and image analysis in remote sensing are important as they are used in various applied domains such as military and civil fields. Earlier approaches for remote sensing images and scene analysis are based on low-level feature representations such as color- and texture-based features. Vector of Locally Aggregated Descriptors (VLAD) and orderless Bag-of-Features (BoF) representations are the examples of mid-level approaches for remote sensing image classification. Recent trends for remote sensing and scene classification are focused on the use of Convolutional Neural Network (CNN). Keeping in view the success of CNN models, in this research, we aim to fine-tune ResNet50 by using network surgery and creation of network head along with the fine-tuning of hyperparameters. The learning of hyperparameters is tuned by using a linear decay learning rate scheduler known as piecewise scheduler. To tune the optimizer hyperparameter, Stochastic Gradient Descent with Momentum (SGDM) is used with the usage of weight learn and bias learn rate factor. Experiments and analysis are conducted on five different datasets, that is, UC Merced Land Use Dataset (UCM), RSSCN (the remote sensing scene classification image dataset), SIRI-WHU, Corel-1K, and Corel-1.5K. The analysis and competitive results exemplify that our proposed image classification-based model can classify the images in a more effective and efficient manner as compared to the state-of-the-art research.

Utterance Verification-Based Dysarthric Speech Intelligibility Assessment Using Phonetic Posterior Features

In the literature, the task of dysarthric speech intelligibility assessment has been approached through development of different low-level feature representations, subspace modeling, phone confidence estimation or measurement of automatic speech recognition system accuracy. This paper proposes a novel approach where the intelligibility is estimated as the percentage of correct words uttered by a speaker with dysarthria by matching and verifying utterances of the speaker with dysarthria against control speakers’ utterances in phone posterior feature space and broad phonetic posterior feature space. Experimental validation of the proposed approach on the UA-Speech database, with posterior feature estimators trained on the data from auxiliary domain and language, obtained a best Pearson's correlation coefficient ( $r$ ) of 0.950 and Spearman's correlation coefficient ( $\rho$ ) of 0.957. Furthermore, replacing control speakers’ speech with speech synthesized by a neural text-to-speech system obtained a best $r$ of 0.931 and $\rho$ of 0.961.

AP-CNN: Weakly Supervised Attention Pyramid Convolutional Neural Network for Fine-Grained Visual Classification.

Classifying the sub-categories of an object from the same super-category (e.g., bird species and cars) in fine-grained visual classification (FGVC) highly relies on discriminative feature representation and accurate region localization. Existing approaches mainly focus on distilling information from high-level features. In this article, by contrast, we show that by integrating low-level information (e.g., color, edge junctions, texture patterns), performance can be improved with enhanced feature representation and accurately located discriminative regions. Our solution, named Attention Pyramid Convolutional Neural Network (AP-CNN), consists of 1) a dual pathway hierarchy structure with a top-down feature pathway and a bottom-up attention pathway, hence learning both high-level semantic and low-level detailed feature representation, and 2) an ROI-guided refinement strategy with ROI-guided dropblock and ROI-guided zoom-in operation, which refines features with discriminative local regions enhanced and background noises eliminated. The proposed AP-CNN can be trained end-to-end, without the need of any additional bounding box/part annotation. Extensive experiments on three popularly tested FGVC datasets (CUB-200-2011, Stanford Cars, and FGVC-Aircraft) demonstrate that our approach achieves state-of-the-art performance. Models and code are available at https://github.com/PRIS-CV/AP-CNN_Pytorch-master.

A Multimodal Deep Learning-Based Fault Detection Model for a Plastic Injection Molding Process

The authors of this work propose a deep learning-based fault detection model that can be implemented in the field of plastic injection molding. Compared to conventional approaches to fault detection in this domain, recent deep learning approaches prove useful for on-site problems involving complex underlying dynamics with a large number of variables. In addition, the advent of advanced sensors that generate data types in multiple modalities prompts the need for multimodal learning with deep neural networks to detect faults. This process is able to facilitate information from various modalities in an end-to-end learning fashion. The proposed deep learning-based approach opts for an early fusion scheme, in which the low-level feature representations of modalities are combined. A case study involving real-world data, obtained from a car parts company and related to a car window side molding process, validates that the proposed model outperforms late fusion methods and conventional models in solving the problem.

Multi-level prediction Siamese network for real-time UAV visual tracking

Existing deployed Unmanned Aerial Vehicles (UAVs) visual trackers are usually based on the correlation filter framework. Although these methods have certain advantages of low computational complexity, the tracking performance of small targets and fast motion scenarios is not satisfactory. In this paper, we present a novel multi-level prediction Siamese network (MLPS) for object tracking in UAV videos, which consists of Siamese feature extraction module and multi-level prediction module. The multi-level prediction module can make full use of the characteristics of each layer features to achieve robust evaluation of targets with different scales. Meanwhile, for small-size target tracking, we design a residual feature fusion block, which is used to constrain the low-level feature representation by using high-level abstract semantics, and obtain the improvement of the tracker's ability to distinguish scene details. In addition, we propose a layer attention fusion block which is sensitive to the informative features of each layers to achieve adaptive fusion of different levels of correlation responses by dynamically balancing the multi-layer features. Sufficient experiments on several UAV tracking benchmarks demonstrate that MLPS achieves state-of-the-art performance and runs at a speed over 97 FPS.