Multi-level Feature Representation Research Articles

Blind image quality index with high-level Semantic Guidance and low-level fine-grained Representation

Image Quality Assessment (IQA) has received unprecedented attention due to the extensive applications in benchmarking image processing algorithms and systems. Despite great progress in IQA, most previous frameworks either utilized only high-level semantic features or simply stacked multi-level features to account for the distortions, resulting in limited performance. The initial visual perception is formed after that the information is processed by the primary visual cortex and the advanced visual cortex. However, the information transmission of the cerebral cortex is not unidirectional, that is, the higher brain area can affect the primary brain area in turn, regulate its sensitivity and preference, so as to help the brain to understand the external world more finely. Inspired by this, we propose a novel blind image quality index with high-level Semantic Guidance and low-level fine-grained Representation (SGRNet). First, the versatile backbone, called pyramid vision Transformer, is used to simulate the multilevel information processing in the brain, generating multilevel feature representation. Second, we propose a novel high-level semantic guidance module to simulate the feedback mechanism between levels of the visual cortex. Third, a simple but effective fine-grained feature extraction module is proposed for high-level information compensation and lower-level content perception. After this, an attention mechanism-based enhancement module is proposed to further learn the above two features respectively. Finally, a bilinear pooling-based regression module is proposed to integrate the enhanced features of the two parts and map them to the quality score. Extensive experiments on six challenging public datasets show that the proposed SGRNet can deal well with both the simulated and authentic distortions and achieves state-of-the-art performance.

CRMEFNet: A coupled refinement, multiscale exploration and fusion network for medical image segmentation

Accurate segmentation of target areas in medical images, such as lesions, is essential for disease diagnosis and clinical analysis. In recent years, deep learning methods have been intensively researched and have generated significant progress in medical image segmentation tasks. However, most of the existing methods have limitations in modeling multilevel feature representations and identification of complex textured pixels at contrasting boundaries. This paper proposes a novel coupled refinement and multiscale exploration and fusion network (CRMEFNet) for medical image segmentation, which explores in the optimization and fusion of multiscale features to address the abovementioned limitations. The CRMEFNet consists of three main innovations: a coupled refinement module (CRM), a multiscale exploration and fusion module (MEFM), and a cascaded progressive decoder (CPD). The CRM decouples features into low-frequency body features and high-frequency edge features, and performs targeted optimization of both to enhance intraclass uniformity and interclass differentiation of features. The MEFM performs a two-stage exploration and fusion of multiscale features using our proposed multiscale aggregation attention mechanism, which explores the differentiated information within the cross-level features, and enhances the contextual connections between the features, to achieves adaptive feature fusion. Compared to existing complex decoders, the CPD decoder (consisting of the CRM and MEFM) can perform fine-grained pixel recognition while retaining complete semantic location information. It also has a simple design and excellent performance. The experimental results from five medical image segmentation tasks, ten datasets and twelve comparison models demonstrate the state-of-the-art performance, interpretability, flexibility and versatility of our CRMEFNet.

LFSimCC: Spatial fusion lightweight network for human pose estimation

To address the limitations of existing 2D human pose estimation methods in terms of speed and lightweight, we propose a method called Lightweight Fusion SimCC (LFSimCC). LFSimCC incorporates two modules: LiteFNet, which enhances multi-scale spatial information fusion, and LKC-GAU, which improves the modeling capability of spatial information. Specifically, LiteFNet utilizes a combination of self-attention mechanism and novel spatial convolution to enable feature maps to capture richer multi-level global feature representations within the network. On the other hand, LKC-GAU enhances SimCC’s ability to capture spatial relationships between joints by incorporating a large kernel of convolution and a self-attention mechanism. Furthermore, we design a keypoint information fusion loss (IFL) that enhances the model’s sensitivity to information between keypoints in the human body. Experimental results demonstrate that our method is capable of extracting more decisive information and suppressing redundant feature representations, leading to high recognition accuracy and low inference latency.

Ancient mural segmentation based on multiscale feature fusion and dual attention enhancement

To address the fuzzy segmentation boundaries, missing details, small target losses and low efficiency of traditional segmentation methods in ancient mural image segmentation scenarios, this paper proposes a mural segmentation model based on multiscale feature fusion and a dual attention-augmented segmentation model (MFAM). The model uses the MobileViT network, which integrates a coordinate attention mechanism, as the feature extraction backbone network. It attains global and local expression capabilities through self-attention, class convolution, and coordinate attention and focuses on location information to expand the receptive field and achieve improved feature extraction efficiency. An A_R_ASPP feature enhancement module is proposed for the attention-optimized residual atrous spatial pyramid pooling module. The module uses residual connections to solve the small target loss problem in murals caused by the excessive sampling rate of atrous convolution and uses a feature attention mechanism to adaptively adjust the feature map weight according to the channel importance levels. A dual attention-enhanced feature fusion module is proposed for multiscale decoder feature fusion to improve the mural segmentation effect. This module uses a cross-level aggregation strategy and an attention mechanism to weight the importance of different feature levels to obtain multilevel semantic feature representations. The model improves the mean intersection over union (MIoU) by 3.06% and the MPA by 1.81% on a mural dataset compared with other models. The model is proven to be effective at improving the segmentation details, efficiency and small target segmentation results produced for mural images, and a new method is proposed for segmenting ancient mural images.

Multi-Level Feature Representation Framework With Adaptive Margin Loss for Few-Shot Sonar Images Classification of AUVs.

Multi-Level Feature Representation Framework With Adaptive Margin Loss for Few-Shot Sonar Images Classification of AUVs.

NA

NA

Dual-correlate optimized coarse-fine strategy for monocular laparoscopic videos feature matching via multilevel sequential coupling feature descriptor

Feature matching of monocular laparoscopic videos is crucial for visualization enhancement in computer-assisted surgery, and the keys to conducting high-quality matches are accurate homography estimation, relative pose estimation, as well as sufficient matches and fast calculation. However, limited by various monocular laparoscopic imaging characteristics such as highlight noises, motion blur, texture interference and illumination variation, most exiting feature matching methods face the challenges of producing high-quality matches efficiently and sufficiently. To overcome these limitations, this paper presents a novel sequential coupling feature descriptor to extract and express multilevel feature maps efficiently, and a dual-correlate optimized coarse-fine strategy to establish dense matches in coarse level and adjust pixel-wise matches in fine level. Firstly, a novel sequential coupling swin transformer layer is designed in feature descriptor to learn and extract multilevel feature representations richly without increasing complexity. Then, a dual-correlate optimized coarse-fine strategy is proposed to match coarse feature sequences under low resolution, and the correlated fine feature sequences is optimized to refine pixel-wise matches based on coarse matching priors. Finally, the sequential coupling feature descriptor and dual-correlate optimization are merged into the Sequential Coupling Dual-Correlate Network (SeCo DC-Net) to produce high-quality matches. The evaluation is conducted on two public laparoscopic datasets: Scared and EndoSLAM, and the experimental results show the proposed network outperforms state-of-the-art methods in homography estimation, relative pose estimation, reprojection error, matching pairs number and inference runtime. The source code is publicly available at https://github.com/Iheckzza/FeatureMatching.

Multi-level knowledge-driven feature representation and triplet loss optimization network for image–text retrieval

Image–text retrieval plays a considerable role in associating vision and language. Existing mainstream approaches focus on fine-grained alignment while ignoring the influence of prior knowledge on model performance and the limitation of using a fixed margin in the triplet loss. In this paper, we propose a Multi-level Knowledge-driven feature representation and Triplet Loss Optimization Network (MKTLON) that exploits prior knowledge to enhance visual and textual feature representations and utilizes an adaptive margin of the triplet loss to optimize network training. Specifically, we first present an Enhanced feature Representation scheme based on the Self-Attention (ERSA) module, which incorporates the prior knowledge randomly initialized by uniform distribution into the matrices K and V in the self-attention mechanism. Subsequently, we adopt cascaded ERSA modules to encode images and texts to obtain multi-level visual and textual features with prior knowledge. Furthermore, we develop an adaptive margin optimization strategy that models the relevance scores of positive and negative samples as two independent Gaussian distributions, and obtain the optimized margin by minimizing the intersection of these two distributions. Extensive experiments on two benchmarks, Flickr30K (155,000 image–text pairs) and MSCOCO (616,435 image–text pairs), show the proposed MKTLON achieves 5.7% and 4.3% improvements on rSum, respectively, compared to the state-of-the-art method. The source code will be released at https://github.com/FlyCuteBird/MKTLON.

Facial expression recognition based on Feature Pyramid Network

Facial expression recognition with significant implications across fields such as psychology, computer science, and artificial intelligence. This paper proposes a combination of a Feature Pyramid Network (FPN) and a Residual Network (ResNet) to construct a recognition model. The main objective of the proposed model is to refine the multi-level feature representation of facial expressions. This approach aims to provide a more holistic understanding of the diverse and complex nature of facial expressions, recognizing the intricate interplay between macro and micro-expressions. Experimental results underscore the model's considerable superiority over traditional methods, particularly in terms of accuracy and adaptability to objects of varying sizes and complexities. This comprehensive approach to facial expression recognition showcases the potential of integrating different neural network architectures, furthering our understanding of the subtleties of facial expressions. The research, therefore, presents a significant contribution to the field of facial expression recognition, demonstrating the efficacy of integrating multi-scale feature extraction techniques to improve model performance. It sets the stage for future research directions in this domain, paving the way for more sophisticated emotion recognition systems that can be deployed in real-world applications.

High-quality super-resolution mapping using spatial deep learning

Super-resolution mapping (SRM) is a critical technology in remote sensing. Recently, several deep learning models have been developed for SRM. Most of these models, however, only use a single stream to process remote sensing images and mainly focus on capturing spectral features. This can undermine the quality of the resulting maps. To address this issue, we propose a soft information-constrained network (SCNet) for SRM that leverages spatial transition features represented by soft information as a spatial prior. Our network incorporates a separate branch to process prior spatial features for feature enhancement. SCNet can extract multi-level feature representations simultaneously from both remote sensing images and prior soft information and hierarchically incorporate features from soft information into image features. Experimental results on three datasets demonstrate that SCNet generates more complete spatial details in complex areas, providing an effective means for producing high-quality and high-resolution mapping products from remote sensing images.

EDPNet: An Encoding-Decoding Network with Pyramidal Representation for Semantic Image Segmentation.

This paper proposes an encoding-decoding network with a pyramidal representation module, which will be referred to as EDPNet, and is designed for efficient semantic image segmentation. On the one hand, during the encoding process of the proposed EDPNet, the enhancement of the Xception network, i.e., Xception+ is employed as a backbone to learn the discriminative feature maps. The obtained discriminative features are then fed into the pyramidal representation module, from which the context-augmented features are learned and optimized by leveraging a multi-level feature representation and aggregation process. On the other hand, during the image restoration decoding process, the encoded semantic-rich features are progressively recovered with the assistance of a simplified skip connection mechanism, which performs channel concatenation between high-level encoded features with rich semantic information and low-level features with spatial detail information. The proposed hybrid representation employing the proposed encoding-decoding and pyramidal structures has a global-aware perception and captures fine-grained contours of various geographical objects very well with high computational efficiency. The performance of the proposed EDPNet has been compared against PSPNet, DeepLabv3, and U-Net, employing four benchmark datasets, namely eTRIMS, Cityscapes, PASCAL VOC2012, and CamVid. EDPNet acquired the highest accuracy of 83.6% and 73.8% mIoUs on eTRIMS and PASCAL VOC2012 datasets, while its accuracy on the other two datasets was comparable to that of PSPNet, DeepLabv3, and U-Net models. EDPNet achieved the highest efficiency among the compared models on all datasets.

EANet: A Point Cloud Registration Network Based on EdgeConv with Dual Attention Mechanism

In machine vision, point cloud registration is one of the core elements, which has been applied to many fields such as robot localisation, medical image processing, and autonomous driving. The main problem solved by point cloud registration is to solve the rotation matrix and translation vectors from one point cloud to another. This paper proposes a point cloud registration network based on EdgeConv with spatial attention mechanism. EdgeConv can dynamically construct graph structure and build topological relationships within the point cloud, so that each point can obtain multi-level feature representation; the attention mechanism can capture contextual information and improve the accuracy of the registration. Experimental results show that EANet has higher registration accuracy, stronger generalisation ability and robustness compared to ICP, Go ICP, FGR, PCRNet and PointNetLK.

Simple and robust depth-wise cascaded network for polyp segmentation

The segmentation of the polyp region in colonoscopy images is considered difficult due to size, texture, and color variation. To segment polyps successfully, models based on convolutional neural networks (CNN), transformers, and their combinations have been developed. However, these methods are limited in that they can only model the local appearance of polyps or lack multi-level feature representation for spatial dependency in the decoding process. In this paper, we propose a simple, efficient yet powerful polyp segmentation framework that unifies the network with a multiscale cascaded path. The proposed MMS-Net utilizes multiscale and multipath convolutional operations in conjunction with multiple deep feature aggregation. The overall dense empowered features are sufficient for pixel-by-pixel detection of the polyp region. Extensive experiments on two popular benchmark datasets for polyp segmentation (Kvasir and CVC-Clinic DB) and two datasets of other medical applications (DRIVE and MC) are presented. The results show that our MMS-Net performs comparably to or better than other state-of-the-art methods despite having two or even three orders of magnitude fewer trainable parameters.

Latent Feature Pyramid Network for Object Detection

Object detection methods based on Convolution Neural Networks (CNN) usually utilize feature pyramid networks to detect objects with various scales. The state-of-the-art feature pyramid networks improve detection accuracy by enhancing multi-level feature representations. Fusing multi-level features is the most effective manner to enhance feature representations. However, the existing feature pyramid networks usually fuse multi-level features by element-wise operations. It leads to the lack of long-range dependencies in the feature fusion. To address this problem, we propose a simple yet efficient feature pyramid network named latent feature pyramid network (LFPN). LFPN can enhance feature representations by modeling inner-scale and cross-scale long-range dependencies through conducting inner-scale and cross-scale feature fusion in the latent space. Comprehensive experiments are performed on two challenge object detection datasets: MS COCO and Pascal VOC. The experimental results show consistent improvements on various feature pyramid networks, backbones, and object detectors, which demonstrates the effectiveness and generality of our LFPN. Our code and models will be made publicly available.

SegMarsViT: Lightweight Mars Terrain Segmentation Network for Autonomous Driving in Planetary Exploration

Planetary rover systems need to perform terrain segmentation to identify feasible driving areas and surround obstacles, which falls into the research area of semantic segmentation. Recently, deep learning (DL)-based methods were proposed and achieved great performance for semantic segmentation. However, due to the on-board processor platform’s strict comstraints on computational complexity and power consumption, existing DL approaches are almost impossible to be deployed on satellites under the burden of extensive computation and large model size. To fill this gap, this paper targeted studying effective and efficient Martian terrain segmentation solutions that are suitable for on-board satellites. In this article, we propose a lightweight ViT-based terrain segmentation method, namely, SegMarsViT. In the encoder part, the mobile vision transformer (MViT) block in the backbone extracts local–global spatial and captures multiscale contextual information concurrently. In the decoder part, the cross-scale feature fusion modules (CFF) further integrate hierarchical context information and the compact feature aggregation module (CFA) combines multi-level feature representation. Moreover, we evaluate the proposed method on three public datasets: AI4Mars, MSL-Seg, and S5Mars. Extensive experiments demonstrate that the proposed SegMarsViT was able to achieve 68.4%, 78.22%, and 67.28% mIoU on the AI4Mars-MSL, MSL-Seg, and S5Mars, respectively, under the speed of 69.52 FPS.

Hierarchical Graph Augmented Deep Collaborative Dictionary Learning for Classification

Recently, deep dictionary learning (DDL) has aroused attention due to its abilities of learning multiple different dictionaries and extracting multi-level abstract feature representations for samples. It has been applied to many intelligent recognition tasks, such as vehicle detection, traffic sign recognition and driver monitoring. Nevertheless, the off-the-shelf DDL-based methods ignore the essential structural information of data in multi-layer dictionary learning. The learned hierarchical data representations are less discriminative. To address this issue, we develop a new DDL framework, called the hierarchical graph augmented deep collaborative dictionary learning (HGDCDL). Firstly, we propose a new deep collaborative dictionary learning (DCDL) that applies collaborative representation to the deepest-level representation learning. Most importantly, equipped with a simple yet effective hierarchal graph construction mechanism, our HGDCDL uses the structure of data to regularize dictionary learning, and generates more informative dictionaries and discriminative representations at different levels. Extensive experiments show that our HGDCDL performs significantly better than the state-of-the-art shallow and deep representation learning methods for classification.

A multi-task driven and reconfigurable network for cloud detection in cloud-snow coexistence regions from very-high-resolution remote sensing images

Cloud detection is a crucial step in remote sensing image preprocessing. Deep learning (DL) methods are currently preferred for cloud detection, and the original images need to be cropped into smaller patches for DL training and testing. However, in very-high-resolution (VHR) remote sensing cloud-snow coexistence scenes, the cropped patches have low feature richness and high inter-class feature similarity. As a result, the common DL methods are prone to misclassifying clouds in complex regions. Therefore, based on the expanded cloud detection dataset of VHR cloud-snow coexistence regions, this paper proposes a multi-task driven and reconfigurable network (MTDR-Net) to better perform the cloud detection work in the scenes. There are four modules in MTDR-Net: the high-resolution backbone with multiple projection heads module (HRB-MPH), the re-parametrizable multi-scale feature fusion module (RP-MFF), the lightweight and adaptive feature fusion module (LAFF), and the multi-task gradient flow guidance module (MTGFG). HRB-MPH facilitates global information interaction, enhances the multi-level granularity feature representation, and supports feature sharing between pixel-level and superpixel-level segmentation tasks. RP-MFF is used to capture the local multi-scale cloud features in the training stage and simplify losslessly the structure in the testing stage. LAFF is employed to reconstruct meaningful cloud features. MTGFG can provide unbiased guidance on dividing gradient flow between the multi-tasks. The experimental results show that the MTDR-Net obtains the greatest accuracy performance due to its anti-interference capacity for confusing ground objects and its excellent cloud boundary and thin cloud extraction ability. In addition, MTDR-Net has the fewest parameters, forward-reasoning floating point operations, memory footprint, and memory access volume. With the coarse-to-fine strategy, MTDR-Net can be further sped up by avoiding inference resource consumption in cloud-free/all-cloud scenes.

Graph-based convolution feature aggregation for retinal vessel segmentation

Retinal vessel segmentation is a crucial task in computer assistant diagnosis of eye diseases. Instead of relying heavily on crafted features, high dimensional deep learning convolution features has provided better representation. However, the neglecting of inherent relation among multiple features is problematic. To explore non-local contextual dependencies, we proposed a graph-based convolution feature aggregation network (GCFAN) for segmenting retinal vessel and enhancing image non-vessel region simultaneously relying on graph to propagate and aggregate message of cross-level features. Specifically, the model consists of three modules: multi-level feature extraction module (MFEM), graph-based high level convolution feature aggregation module (GHFAM), graph-based low level convolution feature aggregation module (GLFAM). The multi-level feature representations are extracted from retinal image in MFEM. GHFAM utilizes more semantic information to reconstruct retinal image without vessel, which facilitates diagnosis. GLFAM utilizes more boundary information to segment vessel. Competitive experimental results on four retinal image datasets validate the efficacy of the proposed model, which achieves segmentation and reconstruct retinal image without vessel, indicating its potential clinical application. Finally, an IoT framework which integrates our algorithm is built to analyze image from various fundus cameras in different places and display results on PC and mobile phone simultaneously, which will facilitate doctor diagnose.

Detecting Personal Medication Intake in Twitter via Domain Attention-Based RNN with Multi-Level Features.

Personal medication intake detection aims to automatically detect tweets that show clear evidence of personal medication consumption. It is a research topic that has attracted considerable attention to drug safety surveillance. This task is inevitably dependent on medical domain information, and the current main model for this task does not explicitly consider domain information. To tackle this problem, we propose a domain attention mechanism for recurrent neural networks, LSTMs, with a multi-level feature representation of Twitter data. Specifically, we utilize character-level CNN to capture morphological features at the word level. Subsequently, we feed them with word embeddings into a BiLSTM to get the hidden representation of a tweet. An attention mechanism is introduced over the hidden state of the BiLSTM to attend to special medical information. Finally, a classification is performed on the weighted hidden representation of tweets. Experiments over a publicly available benchmark dataset show that our model can exploit a domain attention mechanism to consider medical information to improve performance. For example, our approach achieves a precision score of 0.708, a recall score of 0.694, and a F1 score of 0.697, which is significantly outperforming multiple strong and relevant baselines.

Multi-level aircraft feature representation and selection for aviation environmental impact analysis

A wholesome understanding of aviation’s environmental impacts is indispensable to the realization of a sustainable future of aviation. An accurate assessment of aircraft features that influence environmental impacts such as fuel burn, emissions, and noise is necessary to advance the current modeling capability and uncover the interactions between aircraft and their environmental impacts. Nevertheless, contemporary literature lacks a comprehensive feature selection study for aviation environmental impacts. This paper considers the problems of aircraft feature representation and selection for environmental impact analysis. First, aircraft features from different sources are integrated and filtered to obtain a comprehensive, yet essential feature set for representing an aircraft type. Then, a large-scale computer simulation is performed on over 200 unique aircraft types to obtain their environmental impacts during departure and arrival. Multiple supervised statistical learning approaches are utilized for the selection of critical aircraft features. In particular, single- and multi-target feature selections are performed on different levels of environmental impacts. The main results of this study include optimal subsets of features along with their relative importance scores for multiple levels of environmental impacts. A case study on an aircraft grouping and model substitution exercise shows that the improved aircraft representation using selected features and their importance information has the potential to better reflect an aircraft’s characteristics on the environmental impacts. Benefits and potential limitations of the proposed methodology are thoroughly discussed in the paper.