Extract Contextual Information Research Articles

AIRHF-Net: an adaptive interaction representation hierarchical fusion network for occluded person re-identification

To tackle the high resource consumption in occluded person re-identification, sparse attention mechanisms based on Vision Transformers (ViTs) have become popular. However, they often suffer from performance degradation with long sequences, omission of crucial information, and token representation convergence. To address these issues, we introduce AIRHF-Net: an Adaptive Interaction Representation Hierarchical Fusion Network, named AIRHF-Net, designed to enhance pedestrian identity recognition in occluded scenarios. Our approach begins with the development of an Adaptive Local-Window Interaction Encoder (AL-WIE), which aims to overcome the inherent subjective limitations of traditional sparse attention mechanisms. This innovative encoder merges window attention, adaptive local attention, and interaction attention, facilitating automatic localization and focusing on visible pedestrian regions within images. It effectively extracts contextual information from window-level features while minimizing the impact of occlusion noise. Additionally, recognizing that ViTs may lose spatial information in deeper structural layers, we implement a Local Hierarchical Encoder (LHE). This component segments the input sequence in the spatial dimension, integrating features from various spatial positions to construct hierarchical local representations that substantially enhance feature discriminability. To further augment the quality and breadth of datasets, we adopt an Occlusion Data Augmentation Strategy (ODAS), which bolsters the model’s capacity to extract critical information under occluded conditions. Extensive experiments demonstrate that our method achieves improved performance on the Occluded-DukeMTMC dataset, with a rank-1 accuracy of 69.6% and an mAP of 61.6%.

CoMaSa:Context Multi-aware Self-attention for emotional response generation

Dialogue generation is an important research direction in natural language generation, and generating utterances with appropriate contextual emotion is a challenging task. The previous work incorporates commonsense knowledge as an auxiliary information source by simply concatenating or adding it to the embeddings of the dialogue context. They did not consider updating the commonsense knowledge from the perspective of context semantic information or selecting context from the perspective of commonsense knowledge semantic information. At the same time, they did not extract emotional information from dialogue context and commonsense knowledge to enhance the embeddings of the dialogue text. These defects may lead to the dialogue model’s inability to judge the semantic consistency between context and generated utterance, and result in the generation of utterance unassociated with the dialogue context and the corresponding context emotion being untrue. Our goal is to build an emotional dialogue generation model that can deeply integrate commonsense knowledge with dialogue context and achieve bidirectional semantic interaction and enhancement. Therefore, in this paper, we propose a context multi-aware self-attention emotion dialogue generation model that models from multiple aspects such as the semantic information, emotional information of context and external knowledge, it effectively alleviates the problems of generating utterance unrelated to the context and disordered emotional expressions in utterance. First, we adopt the transformer encoder as the basic framework and carefully construct a context-aware encoder capable of deeply extracting semantic information. Specifically, we ingeniously integrate different levels of positional information to simulate the natural flow of conversation between speakers. Then, we effectively integrate these positional information with the basic word embedding vectors, and enrich the dialogue’s embedding representation. Lastly, by introducing the attention mechanism, the model can precisely capture and extract context information closely related to the current situation and achieve a deep understanding and accurate grasp of the dialogue content. Second, we consider that the relationship between context and commonsense knowledge is not just a unidirectional selection but a more intimate bidirectional interaction. Therefore, we designed a bidirectional interaction attention component for context and external knowledge. Specifically, we use the method of iterative similarity matrices to calculate the bidirectional similarity relationship between context and external knowledge. Then, using the attention algorithm, we calculate the connectivity from context to commonsense knowledge and from commonsense knowledge to context. Finally, by combining the semantic embeddings of both aspects, we achieve bidirectional updating of context and external knowledge. Third, we establish an emotion-aware decoder based on multi-task learning, which includes an emotion classifier and an utterance generator. In the emotion classifier, we establish an auxiliary task for emotion recognition to help the model understand the process of emotional transition. We use the output of the emotion recognition model as an additional supervisory signal to better control the generation of emotional responses. In the utterance generator, we integrate the output of the emotion classifier with the interaction vector from the bidirectional interaction attention component. This fusion allows us to generate utterance that aligns with the emotional context. Our experiments on DailyDialog and ESConv datasets show that CoMaSa outperforms the baselines in terms of Perplexity, Distinct-1, Distinct-2, and human evaluation. The experiment demonstrates the effectiveness of our Context Multi-aware Self-attention model in the task of emotion generation.

YOLO-SegNet: A Method for Individual Street Tree Segmentation Based on the Improved YOLOv8 and the SegFormer Network

In urban forest management, individual street tree segmentation is a fundamental method to obtain tree phenotypes, which is especially critical. Most existing tree image segmentation models have been evaluated on smaller datasets and lack experimental verification on larger, publicly available datasets. Therefore, this paper, based on a large, publicly available urban street tree dataset, proposes YOLO-SegNet for individual street tree segmentation. In the first stage of the street tree object detection task, the BiFormer attention mechanism was introduced into the YOLOv8 network to increase the contextual information extraction and improve the ability of the network to detect multiscale and multishaped targets. In the second-stage street tree segmentation task, the SegFormer network was proposed to obtain street tree edge information more efficiently. The experimental results indicate that our proposed YOLO-SegNet method, which combines YOLOv8+BiFormer and SegFormer, achieved a 92.0% mean intersection over union (mIoU), 95.9% mean pixel accuracy (mPA), and 97.4% accuracy on a large, publicly available urban street tree dataset. Compared with those of the fully convolutional neural network (FCN), lite-reduced atrous spatial pyramid pooling (LR-ASPP), pyramid scene parsing network (PSPNet), UNet, DeepLabv3+, and HRNet, the mIoUs of our YOLO-SegNet increased by 10.5, 9.7, 5.0, 6.8, 4.5, and 2.7 percentage points, respectively. The proposed method can effectively support smart agroforestry development.

DFEDC: Dual fusion with enhanced deformable convolution for medical image segmentation

Considering the complexity of lesion regions in medical images, current researches relying on CNNs typically employ large-kernel convolutions to expand the receptive field and enhance segmentation quality. However, these convolution methods are hindered by substantial computational requirements and limited capacity to extract contextual and multi-scale information, making it challenging to efficiently segment complex regions. To address this issue, we propose a dual fusion with enhanced deformable convolution network, namely DFEDC, which dynamically adjusts the receptive field and simultaneously integrates multi-scale feature information to effectively segment complex lesion areas and process boundaries. Firstly, we combine global channel and spatial fusion in a serial way, which integrates and reuses global channel attention and fully connected layers to achieve lightweight extraction of channel and spatial information. Additionally, we design a structured deformable convolution (SDC) that structures deformable convolution with inceptions and large kernel attention, and enhances the learning of offsets through parallel fusion to efficiently extract multi-scale feature information. To compensate for the loss of spatial information of SDC, we introduce a hybrid 2D and 3D feature extraction module to transform feature extraction from a single dimension to a fusion of 2D and 3D. Extensive experimental results on the Synapse, ACDC, and ISIC-2018 datasets demonstrate that our proposed DFEDC achieves superior results.

A PV cell defect detector combined with transformer and attention mechanism

Automated defect detection in electroluminescence (EL) images of photovoltaic (PV) modules on production lines remains a significant challenge, crucial for replacing labor-intensive and costly manual inspections and enhancing production capacity. This paper presents a novel PV defect detection algorithm that leverages the YOLO architecture, integrating an attention mechanism and the Transformer module. We introduce a polarized self-attention mechanism in the feature extraction stage, enabling separate extraction of spatial and semantic features of PV modules, combined with the original input features, to enhance the network’s feature representation capabilities. Subsequently, we integrate the proposed CNN Combined Transformer (CCT) module into the model. The CCT module employs the transformer to extract contextual semantic information more effectively, improving detection accuracy. The experimental results demonstrate that the proposed method achieves a 77.9% mAP50 on the PVEL-AD dataset while preserving real-time detection capabilities. This method enhances the mAP50 by 17.2% compared to the baseline, and the mAP50:95 metric exhibits an 8.4% increase over the baseline.

Infrared Weak and Small Target Detection Algorithm Based on Deep Learning

Abstract In the infrared imaging scene where the target is at a long distance and the background is cluttered, due to the interference of noise and background texture information, the infrared image is prone to problems such as low contrast between the target and the background, and feature confusion, which makes it difficult to accurately extract and detect the target. To solve this problem, firstly, the infrared image is enhanced by combining DDE and MSR algorithm to improve the contrast and detail visibility of the image. For the RT-DETR network structure, the EMA attention mechanism is introduced into the backbone to enhance the feature extraction ability of the model by extracting context information. The CAMixing convolutional attention module is introduced into CCFM, and the multi-scale convolutional self-attention mechanism is introduced to focus on local information and enhance the detection ability of small targets. The filtering rules of the prediction box are improved, combined with Shape-IoU, and the convergence speed of the loss function in the detection and the detection accuracy of small targets are improved by paying attention to the influence of the intrinsic properties of the bounding box itself on the regression. In the experiment, the infrared weak target image dataset of the National University of Defense Technology was selected, labeled and trained. Experimental results show that compared with the original DETR algorithm, the average precision of the improved algorithm (mAP) is increased by 3.2%, and it can effectively detect infrared weak and small targets in different complex backgrounds, which reflects good robustness and adaptability, and can be effectively applied to infrared weak and small target detection in complex backgrounds.

MMHCA: Multi-feature representations based on multi-scale hierarchical contextual aggregation for UAV-view geo-localization

In global navigation satellite system denial environment, cross-view geo-localization based on image retrieval presents an exceedingly critical visual localization solution for Unmanned Aerial Vehicle (UAV) systems. The essence of cross-view geo-localization resides in matching images containing the same geographical targets from disparate platforms, such as UAV-view and satellite-view images. However, images of the same geographical targets may suffer from occlusions and geometric distortions due to variations in the capturing platform, view, and timing. The existing methods predominantly extract features by segmenting feature maps, which overlook the holistic semantic distribution and structural information of objects, resulting in loss of image information. To address these challenges, dilated neighborhood attention Transformer is employed as the feature extraction backbone, and Multi-feature representations based on Multi-scale Hierarchical Contextual Aggregation (MMHCA) is proposed. In the proposed MMHCA method, the multi-scale hierarchical contextual aggregation method is utilized to extract contextual information from local to global across various granularity levels, establishing feature associations of contextual information with global and local information in the image. Subsequently, the multi-feature representations method is utilized to obtain rich discriminative feature information, bolstering the robustness of model in scenarios characterized by positional shifts, varying distances, and scale ambiguities. Comprehensive experiments conducted on the extensively utilized University-1652 and SUES-200 benchmarks indicate that the MMHCA method surpasses the existing techniques, showing outstanding results in UAV localization and navigation.

SPNet: Dual-Branch Network with Spatial Supplementary Information for Building and Water Segmentation of Remote Sensing Images

Semantic segmentation is primarily employed to generate accurate prediction labels for each pixel of the input image, and then classify the images according to the generated labels. Semantic segmentation of building and water in remote sensing images helps us to conduct reasonable land planning for a city. However, many current mature networks face challenges in simultaneously attending to both contextual and spatial information when performing semantic segmentation on remote sensing imagery. This often leads to misclassifications and omissions. Therefore, this paper proposes a Dual-Branch Network with Spatial Supplementary Information (SPNet) to address the aforementioned issues. We introduce a Context-aware Spatial Feature-Extractor Unit (CSF) to extract contextual and spatial information, followed by the Feature-Interaction Module (FIM) to supplement contextual semantic information with spatial details. Additionally, incorporating the Goal-Oriented Attention Mechanism helps in handling noise. Finally, to obtain more detailed branches, a Multichannel Deep Feature-Extraction Module (MFM) is introduced to extract features from shallow-level network layers. This branch guides the fusion of low-level semantic information with high-level semantic information. Experiments were conducted on building and water datasets, respectively. The results indicate that the segmentation accuracy of the model proposed in this paper surpasses that of other existing mature models. On the building dataset, the mIoU reaches 87.57, while on the water dataset, the mIoU achieves 96.8, which means that the model introduced in this paper demonstrates strong generalization capabilities.

DAEiS-Net: Deep Aggregation Network with Edge Information Supplement for Tunnel Water Stain Segmentation.

Tunnel disease detection and maintenance are critical tasks in urban engineering, and are essential for the safety and stability of urban transportation systems. Water stain detection presents unique challenges due to its variable morphology and scale, which leads to insufficient multiscale contextual information extraction and boundary information loss in complex environments. To address these challenges, this paper proposes a method called Deep Aggregation Network with Edge Information Supplement (DAEiS-Net) for detecting tunnel water stains. The proposed method employs a classic encoder-decoder architecture. Specifically, in the encoder part, a Deep Aggregation Module (DAM) is introduced to enhance feature representation capabilities. Additionally, a Multiscale Cross-Attention Module (MCAM) is proposed to suppress noise in the shallow features and enhance the texture information of the high-level features. Moreover, an Edge Information Supplement Module (EISM) is designed to mitigate semantic gaps across different stages of feature extraction, improving the extraction of water stain edge information. Furthermore, a Sub-Pixel Module (SPM) is proposed to fuse features at various scales, enhancing edge feature representation. Finally, we introduce the Tunnel Water Stain Dataset (TWS), specifically designed for tunnel water stain segmentation. Experimental results on the TWS dataset demonstrate that DAEiS-Net achieves state-of-the-art performance in tunnel water stain segmentation.

Deep video compression based on Long-range Temporal Context Learning

Video compression allows for efficient storage and transmission of data, benefiting imaging and vision applications, e.g. computational imaging, photography, and displays by delivering high-quality videos. To exploit more informative contexts of video, we propose DVCL, a novel Deep Video Compression based on Long-range Temporal Context Learning. Aiming at high coding performance, this new compression paradigm makes full use of long-range temporal correlations derived from multiple reference frames to learn richer contexts. Motion vectors (MVs) are estimated to represent the motion relations of videos. By employing MVs, a long-range temporal context learning (LTCL) module is presented to extract context information from multiple reference frames, such that a more accurate and informative temporal contexts can be learned and constructed. The long-range temporal contexts serve as conditions and generate the predicted frames by contextual encoder and decoder. To address the challenge of imbalanced training, we develop a multi-stage training strategy to ensure the whole DVCL framework is trained progressively and stably. Extensive experiments demonstrate the proposed DVCL achieves the highest objective and subjective quality, while maintaining relatively low complexity. Specifically, 25.30% and 45.75% bitrate savings on average can be obtained than x265 codec at the same PSNR and MS-SSIM, respectively.

An Effective and Lightweight Full-Scale Target Detection Network for UAV Images Based on Deformable Convolutions and Multi-Scale Contextual Feature Optimization

Currently, target detection on unmanned aerial vehicle (UAV) images is a research hotspot. Due to the significant scale variability of targets and the interference of complex backgrounds, current target detection models face challenges when applied to UAV images. To address these issues, we designed an effective and lightweight full-scale target detection network, FSTD-Net. The design of FSTD-Net is based on three principal aspects. Firstly, to optimize the extracted target features at different scales while minimizing background noise and sparse feature representations, a multi-scale contextual information extraction module (MSCIEM) is developed. The multi-scale information extraction module (MSIEM) in MSCIEM can better capture multi-scale features, and the contextual information extraction module (CIEM) in MSCIEM is designed to capture long-range contextual information. Secondly, to better adapt to various target shapes at different scales in UAV images, we propose the feature extraction module fitting different shapes (FEMFDS), based on deformable convolutions. Finally, considering low-level features contain rich details, a low-level feature enhancement branch (LLFEB) is designed. The experiments demonstrate that, compared to the second-best model, the proposed FSTD-Net achieves improvements of 3.8%, 2.4%, and 2.0% in AP50, AP, and AP75 on the VisDrone2019, respectively. Additionally, FSTD-Net achieves enhancements of 3.4%, 1.7%, and 1% on the UAVDT dataset. Our proposed FSTD-Net has better detection performance compared to state-of-the-art detection models. The experimental results indicate the effectiveness of the FSTD-Net for target detection in UAV images.

An aspect sentiment analysis model with Aspect Gated Convolution and Dual-Feature Filtering layers

Aspect level sentiment analysis is a basic task to determine the sentiment bias based on the contextual information near the aspect words. Some sentences contain many confusing feature words due to incomplete structure or high complexity in sentiment prediction, which can easily lead to the problem that the model pays too much attention to unimportant features. Furthermore, the role of aspect-related sentiment features is not significant in the feature extraction process. The capsule network, due to its complex network structure, leads to capsule detachment when too much information is found in the dataset. Too much computational resources are easily consumed during dynamic routing, which in turn affects the model's judgement of the polarity of aspect-related sentiment. To address these issues, we propose a capsule network (AGCDFF-Caps) with aspect-gated convolution and Dual-Feature Filtering layers for aspect-level sentiment analysis. A pre-trained BERT model is used to generate a serialised representation of the text, and the semantic representation in the contextual text is enhanced by the self-attention mechanism and bi-GRU. An aspect-gating based convolutional neural network is constructed to selectively extract contextual sentiment information about aspects and discard irrelevant information. A capsule network is then used to learn the multispatial semantic features of the aspect and the text. In particular, we incorporate a Dual-Feature Filtering network structure into the capsule network structure to strengthen the interaction between the particular aspect and the context from global and local perspectives, filtering the redundant information in local semantics and global semantics. The opinion feature representations that can more accurately express the emotional tendency of the aspect are obtained. Experimental results on SemEval2014 and Twitter datasets show that the proposed hybrid network structure has superior classification performance compared to 12 advanced baseline methods.

Optimization of Remote-Sensing Image-Segmentation Decoder Based on Multi-Dilation and Large-Kernel Convolution

Land-cover segmentation, a fundamental task within the domain of remote sensing, boasts a broad spectrum of application potential. We address the challenges in land-cover segmentation of remote-sensing imagery and complete the following work. Firstly, to tackle the issues of foreground–background imbalance and scale variation, a module based on multi-dilated rate convolution fusion was integrated into a decoder. This module extended the receptive field through multi-dilated convolution, enhancing the model’s capability to capture global features. Secondly, to address the diversity of scenes and background interference, a hybrid attention module based on large-kernel convolution was employed to improve the performance of the decoder. This module, based on a combination of spatial and channel attention mechanisms, enhanced the extraction of contextual information through large-kernel convolution. A convolution kernel selection mechanism was also introduced to dynamically select the convolution kernel of the appropriate receptive field, suppress irrelevant background information, and improve segmentation accuracy. Ablation studies on the Vaihingen and Potsdam datasets demonstrate that our decoder significantly outperforms the baseline in terms of mean intersection over union and mean F1 score, achieving an increase of up to 1.73% and 1.17%, respectively, compared with the baseline. In quantitative comparisons, the accuracy of our improved decoder also surpasses other algorithms in the majority of categories. The results of this paper indicate that our improved decoder achieves significant performance improvement compared with the old decoder in remote-sensing image-segmentation tasks, which verifies its application potential in the field of land-cover segmentation.

Attention-Based Neural Network for Cardiac MRI Segmentation: Application to Strain and Volume Computation

ContextDeep learning algorithms have been widely used for cardiac image segmentation. However, most of these architectures rely on convolutions that hardly model long-range dependencies, limiting their ability to extract contextual information. Moreover, the traditional U-net architecture suffers from the difference of semantic information between feature maps of the encoder and decoder (also known as the semantic gap). Material and methodTo address this issue, a new network architecture relying on attention mechanism was introduced. Swin Filtering Blocks (SFB), that use Swin Transformer blocks in a cross-attention manner, were added between the encoder and the decoder to filter information coming from the encoder based on the feature map from the decoder. Attention was also employed at the lowest resolution in the form of a transformer layer to increase the receptive field of the network.We conducted experiments to assess both generalization capability and to evaluate how training on all frames of the cardiac cycle rather than only the end-diastole and end-systole impacts strain and segmentation performances. Results and conclusionVisual inspection of feature maps suggested that Swin Filtering Blocks contribute to the reduction of the semantic gap. Performing attention between all patches using a transformer layer brought higher performance than convolutions. Training the model with all phases of the cardiac cycle resulted in slightly more accurate segmentations while leading to a more noticeable improvement for strain estimation. A limited decrease in performance was observed when testing on out-of-distribution data, but the gap widens for the most apical slices.

Identifying rice field weeds from unmanned aerial vehicle remote sensing imagery using deep learning

BackgroundRice field weed object detection can provide key information on weed species and locations for precise spraying, which is of great significance in actual agricultural production. However, facing the complex and changing real farm environments, traditional object detection methods still have difficulties in identifying small-sized, occluded and densely distributed weed instances. To address these problems, this paper proposes a multi-scale feature enhanced DETR network, named RMS-DETR. By adding multi-scale feature extraction branches on top of DETR, this model fully utilizes the information from different semantic feature layers to improve recognition capability for rice field weeds in real-world scenarios.MethodsIntroducing multi-scale feature layers on the basis of the DETR model, we conduct a differentiated design for different semantic feature layers. The high-level semantic feature layer adopts Transformer structure to extract contextual information between barnyard grass and rice plants. The low-level semantic feature layer uses CNN structure to extract local detail features of barnyard grass. Introducing multi-scale feature layers inevitably leads to increased model computation, thus lowering model inference speed. Therefore, we employ a new type of Pconv (Partial convolution) to replace traditional standard convolutions in the model.ResultsCompared to the original DETR model, our proposed RMS-DETR model achieved an average recognition accuracy improvement of 3.6% and 4.4% on our constructed rice field weeds dataset and the DOTA public dataset, respectively. The average recognition accuracies reached 0.792 and 0.851, respectively. The RMS-DETR model size is 40.8 M with inference time of 0.0081 s. Compared with three classical DETR models (Deformable DETR, Anchor DETR and DAB-DETR), the RMS-DETR model respectively improved average precision by 2.1%, 4.9% and 2.4%.DiscussionThis model is capable of accurately identifying rice field weeds in complex real-world scenarios, thus providing key technical support for precision spraying and management of variable-rate spraying systems.

An Improved U-Net Infrared Small Target Detection Algorithm Based on Multi-Scale Feature Decomposition and Fusion and Attention Mechanism.

Infrared small target detection technology plays a crucial role in various fields such as military reconnaissance, power patrol, medical diagnosis, and security. The advancement of deep learning has led to the success of convolutional neural networks in target segmentation. However, due to challenges like small target scales, weak signals, and strong background interference in infrared images, convolutional neural networks often face issues like leakage and misdetection in small target segmentation tasks. To address this, an enhanced U-Net method called MST-UNet is proposed, the method combines multi-scale feature decomposition and fusion and attention mechanisms. The method involves using Haar wavelet transform instead of maximum pooling for downsampling in the encoder to minimize feature loss and enhance feature utilization. Additionally, a multi-scale residual unit is introduced to extract contextual information at different scales, improving sensory field and feature expression. The inclusion of a triple attention mechanism in the encoder structure further enhances multidimensional information utilization and feature recovery by the decoder. Experimental analysis on the NUDT-SIRST dataset demonstrates that the proposed method significantly improves target contour accuracy and segmentation precision, achieving IoU and nIoU values of 80.09% and 80.19%, respectively.

Gated graph spiking neural P network for session-based recommendation

Session-based recommendation (SBR) is a recommendation system application scenario that focuses on user interests and behaviors in sessions formed within a short period. However, existing SBR methods do not fully extract user contextual information, and gated graph neural networks(GNNs) still face challenges in handling long-term dependencies. The nonlinear spiking neural P system is a novel spiking neural-like computing model in which the nonlinear spiking mechanism is a typical feature. Based on this nonlinear spiking mechanism, we propose a new recurrent-like model called the extended gated spiking neural P model or the EGSNP model, which can effectively capture temporal relationships in their sequence and obtain the user’s contextual information to alleviate information extraction problems in SBR. Based on this EGSNP model, we propose a new type of GNN called the extended gated graph spiking neural P network or termed the EGGSNP network. Finally, we use the EGSNP model and EGGSNP network to construct a new SBR model for the session recommendation task, termed the SR-EGGSNP model. This SR-EGGSNP model can effectively extract contextual information and flexibly capture the long-term evolution of interest while alleviating the long-distance dependency problem in GNNs. We conducted extensive comparative experiments on two public datasets, and the results demonstrated the effectiveness of the proposed model and accuracy of the recommendations.

Landslide extraction from aerial imagery considering context association characteristics

Accurate extraction of landslide information is crucial for timely disaster emergency response, yet this process faces significant challenges due to the interference of bare land and vegetation cover as well as the variability in landslide scales. Furthermore, the scarcity of comprehensive landslide datasets has hindered the in-depth exploration of deep learning techniques in this domain. Addressing these issues, this study introduces a novel framework for landslide extraction that leverages the characteristics of context association. In the encoding phase, a two-branch multiscale context feature extraction module (TMCFM) is established, which captures the contextual relationships across different scales through an attention mechanism while concurrently extracting context information within the same scale. To refine the interplay between features at different levels, self-attention is utilized, allowing for the effective fusion of context information at both ends of the same scale through feature fusion exploiting static and dynamic context feature modules (FSDC). Building upon this, this study develops a deeply supervised classifier (DSC) that enhances the network’s discriminative capabilities in the prediction phase via six auxiliary branches. Additionally, this study contributes a new aerial image dataset for landslide extraction created through meticulous visual interpretation. The proposed method is thoroughly compared with 17 contemporary deep learning methods, demonstrating an increase in the intersection-over-union (IoU) metric of 0.92%-16.94% over these methods. The robustness and superiority of the proposed approach are further validated through various discussions and analyses.

Hyperspectral Image Classification Based on Two-Branch Multiscale Spatial Spectral Feature Fusion with Self-Attention Mechanisms

In recent years, the use of deep neural network in effective network feature extraction and the design of efficient and high-precision hyperspectral image classification algorithms has gradually become a research hotspot for scholars. However, due to the difficulty of obtaining hyperspectral images and the high cost of annotation, the training samples are very limited. In order to cope with the small sample problem, researchers often deepen the network model and use the attention mechanism to extract features; however, as the network model continues to deepen, the gradient disappears, the feature extraction ability is insufficient, and the computational cost is high. Therefore, how to make full use of the spectral and spatial information in limited samples has gradually become a difficult problem. In order to cope with such problems, this paper proposes two-branch multiscale spatial–spectral feature aggregation with a self-attention mechanism for a hyperspectral image classification model (FHDANet); the model constructs a dense two-branch pyramid structure, which can achieve the high efficiency extraction of joint spatial–spectral feature information and spectral feature information, reduce feature loss to a large extent, and strengthen the model’s ability to extract contextual information. A channel–space attention module, ECBAM, is proposed, which greatly improves the extraction ability of the model for salient features, and a spatial information extraction module based on the deep feature fusion strategy HLDFF is proposed, which fully strengthens feature reusability and mitigates the feature loss problem brought about by the deepening of the model. Compared with five hyperspectral image classification algorithms, SVM, SSRN, A2S2K-ResNet, HyBridSN, SSDGL, RSSGL and LANet, this method significantly improves the classification performance on four representative datasets. Experiments have demonstrated that FHDANet can better extract and utilise the spatial and spectral information in hyperspectral images with excellent classification performance under small sample conditions.

Complete contextual information extraction for self-supervised monocular depth estimation

Self-supervised learning methods are increasingly important for monocular depth estimation since they do not require ground-truth data during training. Although existing methods have achieved great success for better monocular depth estimation based on Convolutional Neural Networks (CNNs), the limited receptive field of CNNs usually is insufficient to effectively model the global information, e.g., relationship between foreground and background or relationship among objects, which are crucial for accurately capturing scene structure. Recently, some studies based on Transformers have attracted significant interest in computer vision. However, duo to the lack of spatial locality bias, they may fail to model the local information, e.g., fine-grained details with an image. To tackle these issues, we propose a novel self-supervised learning framework by incorporating the advantages of both the CNNs and Transformers so as to model the complete contextual information for high-quality monocular depth estimation. Specifically, the proposed method mainly includes two branches, where the Transformer branch is considered to capture the global information while the Convolution branch is exploited to preserve the local information. We also design a rectangle convolution module with pyramid structure to perceive the semi-global information, e.g. thin objects, along the horizontal and vertical directions within an image. Moreover, we propose a shape refinement module by learning the affinity matrix between pixel and its neighborhood to obtain accurate geometrical structure of scenes. Extensive experiments evaluated on KITTI, Cityscapes and Make3D dataset demonstrate that the proposed method achieves the competitive result compared with the state-of-the-art self-supervised monocular depth estimation methods and shows good cross-dataset generalization ability.