Local Receptive Fields Research Articles

Vision Transformer and Deep Sequence Learning for Human Activity Recognition in Surveillance Videos

Human Activity Recognition is an active research area with several Convolutional Neural Network (CNN) based features extraction and classification methods employed for surveillance and other applications. However, accurate identification of HAR from a sequence of frames is a challenging task due to cluttered background, different viewpoints, low resolution, and partial occlusion. Current CNN-based techniques use large-scale computational classifiers along with convolutional operators having local receptive fields, limiting their performance to capture long-range temporal information. Therefore, in this work, we introduce a convolution-free approach for accurate HAR, which overcomes the above-mentioned problems and accurately encodes relative spatial information. In the proposed framework, the frame-level features are extracted via pretrained Vision Transformer; next, these features are passed to multilayer long short-term memory to capture the long-range dependencies of the actions in the surveillance videos. To validate the performance of the proposed framework, we carried out extensive experiments on UCF50 and HMDB51 benchmark HAR datasets and improved accuracy by 0.944% and 1.414%, respectively, when compared to state-of-the-art deep models.

RODFormer: High-Precision Design for Rotating Object Detection with Transformers.

Aiming at the problem of Transformers lack of local spatial receptive field and discontinuous boundary loss in rotating object detection, in this paper, we propose a Transformer-based high-precision rotating object detection model (RODFormer). Firstly, RODFormer uses a structured transformer architecture to collect feature information of different resolutions to improve the collection range of feature information. Secondly, a new feed-forward network (spatial-FFN) is constructed. Spatial-FFN fuses the local spatial features of 3 × 3 depthwise separable convolutions with the global channel features of multilayer perceptron (MLP) to solve the deficiencies of FFN in local spatial modeling. Finally, based on the space-FFN architecture, a detection head is built using the CIOU-smooth L1 loss function and only returns to the horizontal frame when the rotating frame is close to the horizontal, so as to alleviate the loss discontinuity of the rotating frame. Ablation experiments of RODFormer on the DOTA dataset show that the Transformer-structured module, the spatial-FFN module and the CIOU-smooth L1 loss function module are all effective in improving the detection accuracy of RODFormer. Compared with 12 rotating object detection models on the DOTA dataset, RODFormer has the highest average detection accuracy (up to 75.60%), that is, RODFormer is more competitive in rotating object detection accuracy.

Data Model of Key Indicators of Urban Architectural Design Based on Long- and Short-Term Convolutional Memory Network

The analysis and design of key indicators of urban architecture has always been a frontier subject in the field of urban architecture. Building key indicator network area flow is challenging due to its high degree of nonlinearity and randomness. Based on the extraction method of key indicators of urban architectural design based on long and short-term convolutional memory network, this paper designs a data model of key indicators of urban architectural design. The model can effectively simulate the urban architectural design system to obtain the key index information of urban architectural design of objects step by step. The architecture of cascaded full convolutional long and short-term convolutional memory network is designed and improved. The experiment adopts the block method to ensure the output result with the same resolution as the input urban building image, which solves the problems of the traditional fully convolutional network such as small local receptive field, low output resolution. The simulation results show that the data prediction accuracy of the key indicators proposed in this paper reaches 92.3%, which is 16% and 11.5% higher than the prediction accuracy of the other two depth algorithms (76.3% and 80.8%) respectively, which promotes the key indicators of information flow within the network.

Weakly supervised fine-grained image classification via two-level attention activation model

Fine-grained image classification is challenging due to the fact that categories can only be distinguished by subtle and local differences. Existing weakly supervised fine-grained image classification methods usually directly extract discriminative regions from the high-level feature maps. We observe that the operation of overlaying local receptive fields in the convolutional neural network makes the discriminative regions spread in the high-level feature maps, which can cause inaccurate region localization. In this paper, we propose an end-to-end Two-Level Attention Activation Model (TL-AAM), which can solve the problem of discriminative region spreading and obtain more effective fine-grained features. Specifically, the TL-AAM consists of: (1) an object attention activation module (OAAM), which links the correct classification score with the object region localization through gradient reflow to accurately localize the object region in a mutually reinforcing way, (2) a multi-scale pyramid attention localization module (MPALM), which locates local feature region by selecting the region with the largest response value in the feature channel, and this module can accurately obtain the detailed features in the local region, (3) a local cross-channel attention module (LCAM), which can filter irrelevant information in the high-level semantic feature maps by giving higher weights to the feature channels with high response values in the feature maps. Extensive experiments verify that TL-AAM yields the state-of-the-art performance under the same settings with the most competitive approaches, in CUB-200-2011, FGVC-Aircrafts, and Stanford-Cars datasets.

Channel and band attention embedded 3D CNN for model development of hyperspectral image in object-scale analysis

Recently there is a rising trend of employing convolutional neural network (CNN) for modeling the complex high-dimensional hyperspectral images in object-scale analysis. Compared with 1D CNN and 2D CNN for merely extracting spectral or spatial features, the 3D CNN naturally offers a more effective method for simultaneously extracting the integrated deep spectral-spatial features. Due to the convolution characteristics of operating within a local receptive field, computer vision studies had incorporated the attention mechanism into 2D CNN to exploit the relationship between features for adaptive feature refinement. No exploration has been reported on incorporating the attention mechanism into 3D CNN for model development of hyperspectral image in object-scale analysis. In this study, we investigated an improved 3D CNN architecture with attention modules embedded for adaptive feature refinement in object-scale hyperspectral image modeling. Besides the adapted channel attention, the band attention module was specially designed to learn the band-wise relationship. Based on the 3D ResNet architecture, various modifications on the arrangement and structure of channel and band attention modules were explored systematically for higher modeling performance. An exemplar hyperspectral image dataset of basil leaves for predicting their relative chlorophyll content (RCC), was applied to evaluate the proposed model. Comprehensive comparison experiments showed performance improvement after adding attention modules into the residual block of 3D ResNet, demonstrating the effectiveness of adaptive feature refinement along channel and band dimensions through the learned attention maps. The sequential channel-band attention module achieved the highest model performance, with testing determination coefficient (R2) of 0.8998. The results indicated the effectiveness of the channel and band attention embedded 3D CNN for model development of hyperspectral image in object-scale analysis.

Periodicity Pitch Perception Part III: Sensibility and Pachinko Volatility.

Neuromorphic computer models are used to explain sensory perceptions. Auditory models generate cochleagrams, which resemble the spike distributions in the auditory nerve. Neuron ensembles along the auditory pathway transform sensory inputs step by step and at the end pitch is represented in auditory categorical spaces. In two previous articles in the series on periodicity pitch perception an extended auditory model had been successfully used for explaining periodicity pitch proved for various musical instrument generated tones and sung vowels. In this third part in the series the focus is on octopus cells as they are central sensitivity elements in auditory cognition processes. A powerful numerical model had been devised, in which auditory nerve fibers (ANFs) spike events are the inputs, triggering the impulse responses of the octopus cells. Efficient algorithms are developed and demonstrated to explain the behavior of octopus cells with a focus on a simple event-based hardware implementation of a layer of octopus neurons. The main finding is, that an octopus' cell model in a local receptive field fine-tunes to a specific trajectory by a spike-timing-dependent plasticity (STDP) learning rule with synaptic pre-activation and the dendritic back-propagating signal as post condition. Successful learning explains away the teacher and there is thus no need for a temporally precise control of plasticity that distinguishes between learning and retrieval phases. Pitch learning is cascaded: At first octopus cells respond individually by self-adjustment to specific trajectories in their local receptive fields, then unions of octopus cells are collectively learned for pitch discrimination. Pitch estimation by inter-spike intervals is shown exemplary using two input scenarios: a simple sinus tone and a sung vowel. The model evaluation indicates an improvement in pitch estimation on a fixed time-scale.

Driver Fatigue Detection Using Improved Deep Learning and Personalized Framework

In transportation, drivers’ state directly affects traffic safety. Therefore, an accurate driver’s fatigue detection is crucial for ensuring driving safety. Real-time and accurate technology is needed for driver fatigue detection. To address this problem, this article proposes a fatigue detection method based on an improved deep learning and personalized framework. First, clustering is applied to face size, and cluster numbers are used to determine the number of detection layers. Then, the size of anchor boxes is set according to the face size. In the proposed framework, the number of convolutional networks is set according to the principle that the receptive field should match the face size in the predicted feature map. Finally, a variety of fatigue features are learned by minimizing the loss function. In addition, a personalized face fatigue detection method is put forward for building a fatigue detection framework to judge the driver’s fatigue status more reasonably. The experimental results show that the proposed method based on an improved clustering method and local receptive field can improve the detection speed of driver’s fatigue while maintaining high detection accuracy. The proposed method can reach 125 fps by using GPU GeForce GTX TITAN, which satisfies the real-time requirement. In addition, the personalized framework can achieve high detection accuracy while keeping acceptable speed. The proposed model can accurately and timely detect driver fatigue, which can help to avoid accidents.

A sophisticated offline network developed for recognizing handwritten Chinese character efficiently

In this paper, we designed an efficient network structure by combining multiple scale convolution shuffle (MSCS) module and attention features spatial aggregation (ASA) module. MSCS module can obtain local and global multiple receptive fields feature information, and it contains the structure idea of the lightweight network convolution block. Moreover, we have proposed ASA module to reducing the information loss of different features in spatial dimension reduction. Meanwhile, we introduce the joint loss function to strengthen the ability of the model to distinguish the difference of attention features between inter-class and intra-class. The experimental results tested on offline HCCR competition dataset ICDAR-2013 show that our network model trained only on handwritten dataset only takes 3.97 ms to recognize a character image and achieves 97.63% accuracy just requires 22.9 MB for storage. Therefore, our method reaches the state-of-the-art in efficient single networks particularly in view of the storage space and inference time.

HDAM: Heuristic Difference Attention Module for Convolutional Neural Networks

The attention mechanism is one of the most important priori knowledge to enhance convolutional neural networks. Most attention mechanisms are bound to the convolutional layer and use local or global contextual information to recalibrate the input. This is a popular attention strategy design method. Global contextual information helps the network to consider the overall distribution, while local contextual information is more general. The contextual information makes the network pay attention to the mean or maximum value of a particular receptive field. Different from the most attention mechanism, this article proposes a novel attention mechanism with the heuristic difference attention module (HDAM). HDAM's input recalibration is based on the difference between the local and global contextual information instead of the mean and maximum values. At the same time, to make different layers have a more suitable local receptive field sizes and increase the flexibility of the local receptive field design, we use genetic algorithm to heuristically produce local receptive fields. First, HDAM extracts the mean value of the global and local receptive fields as the corresponding contextual information. Then the difference between the global and local contextual information is calculated. Finally, HDAM uses this difference to recalibrate the input. In addition, we use the heuristic ability of genetic algorithm to search for the local receptive field size of each layer. Our experiments on CIFAR-10 and CIFAR-100 show that HDAM can use fewer parameters than other attention mechanisms to achieve higher accuracy. We implement HDAM with the Python library, Pytorch, and the code and models will be publicly available.

Attention Guided Global Enhancement and Local Refinement Network for Semantic Segmentation

The encoder-decoder architecture is widely used as a lightweight semantic segmentation network. However, it struggles with a limited performance compared to a well-designed Dilated-FCN model for two major problems. First, commonly used upsampling methods in the decoder such as interpolation and deconvolution suffer from a local receptive field, unable to encode global contexts. Second, low-level features may bring noises to the network decoder through skip connections for the inadequacy of semantic concepts in early encoder layers. To tackle these challenges, a Global Enhancement Method is proposed to aggregate global information from high-level feature maps and adaptively distribute them to different decoder layers, alleviating the shortage of global contexts in the upsampling process. Besides, aLocal Refinement Module is developed by utilizing the decoder features as the semantic guidance to refine the noisy encoder features before the fusion of these two (the decoder features and the encoder features). Then, the two methods are integrated into a Context Fusion Block, and based on that, a novel Attention guided Global enhancement and Local refinement Network (AGLN) is elaborately designed. Extensive experiments on PASCAL Context, ADE20K, and PASCAL VOC 2012 datasets have demonstrated the effectiveness of the proposed approach. In particular, with a vanilla ResNet-101 backbone, AGLN achieves the state-of-the-art result (56.23% mean IOU) on the PASCAL Context dataset. The code is available at https://github.com/zhasen1996/AGLN.

Yolosa: Object Detection Based on 2d Local Feature Superimposed Self-Attention

We analyzed the network structure of real-time object detection models and found that the features in the feature concatenation stage are very rich. Applying an attention module here can effectively improve the detection accuracy of the model. However, the commonly used attention module or self-attention module shows poor performance in detection accuracy and inference efficiency. Therefore, we propose a novel self-attention module, called 2D local feature superimposed self-attention, for the feature concatenation stage of the neck network. This self-attention module reflects global features through local features and local receptive fields. We also propose and optimize an efficient decoupled head and AB-OTA, and achieve SOTA results. Average precisions of 49.0\% (66.2 FPS), 46.1\% (80.6 FPS), and 39.1\% (100 FPS) were obtained for large, medium, and small-scale models built using our proposed improvements. Our models exceeded YOLOv5 by 0.8\% -- 3.1\% in average precision.

Single-Image Super-Resolution for Remote Sensing Images Using a Deep Generative Adversarial Network With Local and Global Attention Mechanisms

Super-resolution (SR) technology is an important way to improve spatial resolution under the condition of sensor hardware limitations. With the development of deep learning (DL), some DL-based SR models have achieved state-of-the-art performance, especially the convolutional neural network (CNN). However, considering that remote sensing images usually contain a variety of ground scenes and objects with different scales, orientations, and spectral characteristics, previous works usually treat important and unnecessary features equally or only apply different weights in the local receptive field, which ignores long-range dependencies; it is still a challenging task to exploit features on different levels and reconstruct images with realistic details. To address these problems, an attention-based generative adversarial network (SRAGAN) is proposed in this article, which applies both local and global attention mechanisms. Specifically, we apply local attention in the SR model to focus on structural components of the earth&#x2019;s surface that require more attention, and global attention is used to capture long-range interdependencies in the channel and spatial dimensions to further refine details. To optimize the adversarial learning process, we also use local and global attentions in the discriminator model to enhance the discriminative ability and apply the gradient penalty in the form of hinge loss and loss function that combines <inline-formula> <tex-math notation="LaTeX">$L1$ </tex-math></inline-formula> pixel loss, <inline-formula> <tex-math notation="LaTeX">$L1$ </tex-math></inline-formula> perceptual loss, and relativistic adversarial loss to promote rich details. The experiments show that SRAGAN can achieve performance improvements and reconstruct better details compared with current state-of-the-art SR methods. A series of ablation investigations and model analyses validate the efficiency and effectiveness of our method.

Res2Fusion: Infrared and Visible Image Fusion Based on Dense Res2net and Double Nonlocal Attention Models

Infrared and visible image fusion intends to generate a synthetic image with superior scene representation and better visual perception. The existing deep learning-based fusion methods merely make use of the convolution operation to extract features with a local receptive field, without fully considering their multiscale and long-range dependency characteristics, which may fail to preserve some essential global context from source images. To this end, we develop a novel and efficient fusion network based on dense Res2net and double nonlocal attention models, termed Res2Fusion. We introduce Res2net and dense connections into the encoder network with multiple available receptive fields, which is used to extract the multiscale features, and can retain as much meaningful information as possible for fusion tasks. In addition, we develop double nonlocal attention models as a fusion layer to model long-range dependencies on the local features. Specifically, these attention models can refine feature maps obtained by the encoder network to more focus on prominent infrared targets and distinct visible details. Finally, the comprehensive attention maps are used to generate a fused result through the simple decoder network. Extensive experiments demonstrate that the proposed method can simultaneously retain highlighted infrared targets and rich visible details and transcends other state-of-the-art fusion methods in terms of subjective and objective evaluation. The corresponding code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Zhishe-Wang/Res2Fusion</uri> .

Semantic Segmentation of Remote Sensing Image Based on Regional Self-Attention Mechanism

In remote sensing images (RSIs), accurate semantic segmentation faces more challenges because of small targets, unbalanced categories, and complex scenes. Restricted by local receptive field of convolution layers, the traditional semantic segmentation models cannot use global information of RSIs. According to the characteristics of RSIs, we propose an RSANet based on regional self-attention mechanism. Our model is no longer limited by the locality of convolution, but transfers the information flow in the whole image. It can mine out the relationship between pixels in the surrounding areas, which is more logical for understanding images content. Moreover, compared with the traditional self-attention mechanism, RSANet can effectively reduce the noise of feature maps and the interference of redundant features. Our model can get better semantic segmentation results than other current models on the DroneDeploy data set and the Chreos semantic segmentation data set. The experiments show that our RSANet achieves 2% higher mean intersection over union (mIoU) than the baseline model, especially in terms of fineness, edge integrity, and classification accuracy.

TwinsReID: Person re-identification based on twins transformer's multi-level features.

In the traditional person re-identification model, the CNN network is usually used for feature extraction. When converting the feature map into a feature vector, a large number of convolution operations are used to reduce the size of the feature map. In CNN, since the receptive field of the latter layer is obtained by convolution operation on the feature map of the previous layer, the size of this local receptive field is limited, and the computational cost is large. For these problems, combined with the self-attention characteristics of Transformer, an end-to-end person re-identification model (twinsReID) is designed that integrates feature information between levels in this article. For Transformer, the output of each layer is the correlation between its previous layer and other elements. This operation is equivalent to the global receptive field because each element needs to calculate the correlation with other elements, and the calculation is simple, so its cost is small. From these perspectives, Transformer has certain advantages over CNN's convolution operation. This paper uses Twins-SVT Transformer to replace the CNN network, combines the features extracted from the two different stages and divides them into two branches. First, convolve the feature map to obtain a fine-grained feature map, perform global adaptive average pooling on the second branch to obtain the feature vector. Then divide the feature map level into two sections, perform global adaptive average pooling on each. These three feature vectors are obtained and sent to the Triplet Loss respectively. After sending the feature vectors to the fully connected layer, the output is input to the Cross-Entropy Loss and Center-Loss. The model is verified On the Market-1501 dataset in the experiments. The mAP/rank1 index reaches 85.4%/93.7%, and reaches 93.6%/94.9% after reranking. The statistics of the parameters show that the parameters of the model are less than those of the traditional CNN model.

Vehicle Detection Based on Semantic-Context Enhancement for High-Resolution SAR Images in Complex Background

Small-scale target detection (such as vehicles) in complex synthetic aperture radar (SAR) image scenes has always been a pain point for the advanced convolutional neural network (CNN)-based target detectors because of the downsampling operations and the local receptive field characteristics of CNNs. To tackle these limitations, a vehicle detector named SCEDet for the small-scale vehicles in SAR images is proposed to improve the detection performance in this letter. SCEDet mainly consists of two parts: subaperture semantic feature extraction and subaperture semantic-context enhancement (SCE) with SCE module. First, ResNet34 with subaperture decomposition is used to efficiently exploit the latent subaperture semantic features. Then, the SCE module is proposed to balance the multiscale semantic information as well as aggregate the global context information for vehicle detection with a small number of parameters and computation costs. The experimental results on the FARAD dataset (0.1 m <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times0.1$ </tex-math></inline-formula> m, Ka-band) demonstrate that both the detection performance and the speed are much better than other detection methods under the same hardware conditions.

Deep Feature Interactive Aggregation Network for Single Image Deraining

Single image deraining task aims at removing rain streaks from a degraded input and reconstructing the high-quality image. In recent years, image processing tasks mostly apply the U-shaped architecture to capture rich contextual information. However, it is difficult to achieve long-range pixel dependencies due to the local receptive field of the convolution operation. In this paper, we propose a deep feature interactive aggregation network for single image deraining to enhance the long-range dependencies among features and realize the interaction of information. To fully utilize the high-level semantic features, we design a long-range dependency feature aggregation module to greatly improve the representational ability of the original U-shaped architecture. It aggregates multi-scale features and calculates the interactive attention of non-overlapping patches among feature maps. In addition, we adopt group normalization to retain the independence of each given image. It interacts with the information among features in an individual image and normalizes the channels of each group to weaken the correlation between batch data processing. Experimental results on widely acknowledged datasets also demonstrate the superiority of our proposed network over previous state-of-the-art methods.

Class-Guided Swin Transformer for Semantic Segmentation of Remote Sensing Imagery

Semantic segmentation of remote sensing images plays a crucial role in a wide variety of practical applications, including land cover mapping, environmental protection, and economic assessment. In the last decade, convolutional neural network (CNN) is the mainstream deep learning-based method of semantic segmentation. Compared with conventional methods, CNN-based methods learn semantic features automatically, thereby achieving strong representation capability. However, the local receptive field of the convolution operation limits CNN-based methods from capturing long-range dependencies. In contrast, Vision Transformer (ViT) demonstrates its great potential in modeling long-range dependencies and obtains superior results in semantic segmentation. Inspired by this, in this letter, we propose a class-guided Swin Transformer (CG-Swin) for semantic segmentation of remote sensing images. Specifically, we adopt a Transformer-based encoder–decoder structure, which introduces the Swin Transformer backbone as the encoder and designs a class-guided Transformer block to construct the decoder. The experimental results on ISPRS Vaihingen and Potsdam datasets demonstrate the significant breakthrough of the proposed method over ten benchmarks, outperforming both advanced CNN-based and recent Transformer-based approaches.

Convolutional Sparse Coding Fast Approximation With Application to Seismic Reflectivity Estimation

In sparse coding, we attempt to extract features of input vectors, assuming that the data is inherently structured as a sparse superposition of basic building blocks. Similarly, neural networks perform a given task by learning features of the training data set. Recently both data-driven and model-driven feature extracting methods have become extremely popular and have achieved remarkable results. Nevertheless, practical implementations are often too slow to be employed in real-life scenarios, especially for real-time applications. We propose a speed-up upgraded version of the classic iterative thresholding algorithm, that produces a good approximation of the convolutional sparse code within 2-5 iterations. The speed advantage is gained mostly from the observation that most solvers are slowed down by inefficient global thresholding. The main idea is to normalize each data point by the local receptive field energy, before applying a threshold. This way, the natural inclination towards strong feature expressions is suppressed, so that one can rely on a global threshold that can be easily approximated, or learned during training. The proposed algorithm can be employed with a known predetermined dictionary, or with a trained dictionary. The trained version is implemented as a neural net designed as the unfolding of the proposed solver. The performance of the proposed solution is demonstrated via the seismic inversion problem in both synthetic and real data scenarios. We also provide theoretical guarantees for a stable support recovery. Namely, we prove that under certain conditions the true support is perfectly recovered within the first iteration.

ECA-MobileNetV3(Large)+SegNet Model for Binary Sugarcane Classification of Remotely Sensed Images

In recent years, several models based on fully convolutional neural networks have been proposed. These models mainly focused on improving accuracy but ignored computational efficiency. For this reason, this research proposes an innovative deep learning model, entitled “ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ECA-MobileNetV3(large)+Seg-Net model</i> ” for simultaneously concerning both. In the encoder, the structure-reduced MobileNetV3(large) is selected as the backbone network, which uses 11 network layers to replace 20 network layers of MobileNetV3(large). The hybrid dilated convolution with dilation rates of 1, 2, and 4 is introduced in the depthwise separable convolution to expand the local receptive field and to enhance the connection of contextual information. Finally, the ECA module is fused with the bneck structure. In the decoder, the 18 network layers of SegNet’s decoder are replaced with 9 layers to achieve a lightweight network with small parameters. When compared with the original SegNet, the overall accuracy (OA) of the model proposed in this research is averagely improved by 5.97%, 5.56%, and 4.13%, and the sugarcane identification accuracy is averagely improved by 7.16%, 4.01%, and 9.13%, respectively in the three tested areas. Additionally, the memory size, the number of parameters, and FLOPs are all reduced by 6/7.