Efficient Feature Representation Research Articles

Efficient large-scale scene representation with a hybrid of high-resolution grid and plane features

Existing neural radiance fields (NeRF) methods for large-scale scene modeling require days of training using multiple GPUs, hindering their applications in scenarios with limited computing resources. Radiance field models based on explicit dense or hash grid features have been proposed for fast optimization, but their effectiveness has mainly been demonstrated in the context of object-scale scene representation. In this paper, we point out that the low feature resolution in explicit representation is the bottleneck for large-scale unbounded scene representation. To address this problem, we introduce a new and efficient hybrid feature representation for NeRF that fuses the 3D hash-grids and high-resolution 2D dense plane features. Compared with the dense-grid representation, the resolution of a dense 2D plane can be scaled up more efficiently. Based on this hybrid representation, we propose a fast optimization NeRF variant, called GP-NeRF, that achieves better rendering results while maintaining a compact model size. Extensive experiments on multiple large-scale unbounded scene datasets show that our model can converge in 1.5 h using a single GPU while achieving results comparable to or even better than the existing method that requires about one day’s training with 8 GPUs. Our code can be found at https://zyqz97.github.io/GP_NeRF/.

CDGT: Constructing diverse graph transformers for emotion recognition from facial videos

Recognizing expressions from dynamic facial videos can find more natural affect states of humans, and it becomes a more challenging task in real-world scenes due to pose variations of face, partial occlusions and subtle dynamic changes of emotion sequences. Existing transformer-based methods often focus on self-attention to model the global relations among spatial features or temporal features, which cannot well focus on important expression-related locality structures from both spatial and temporal features for the in-the-wild expression videos. To this end, we incorporate diverse graph structures into transformers and propose a CDGT method to construct diverse graph transformers for efficient emotion recognition from in-the-wild videos. Specifically, our method contains a spatial dual-graphs transformer and a temporal hyperbolic-graph transformer. The former deploys a dual-graph constrained attention to capture latent emotion-related graph geometry structures among local spatial tokens for efficient feature representation, especially for the video frames with pose variations and partial occlusions. The latter adopts a hyperbolic-graph constrained self-attention that explores important temporal graph structure information under hyperbolic space to model more subtle changes of dynamic emotion. Extensive experimental results on in-the-wild video-based facial expression databases show that our proposed CDGT outperforms other state-of-the-art methods.

Achieving Efficient Feature Representation for Modulation Signal: A Cooperative Contrast Learning Approach

Achieving Efficient Feature Representation for Modulation Signal: A Cooperative Contrast Learning Approach

DLformer: A Dynamic Length Transformer-Based Network for Efficient Feature Representation in Remaining Useful Life Prediction.

Representation learning-based remaining useful life (RUL) prediction plays a crucial role in improving the security and reducing the maintenance cost of complex systems. Despite the superior performance, the high computational cost of deep networks hinders deploying the models on low-compute platforms. A significant reason for the high cost is the computation of representing long sequences. In contrast to most RUL prediction methods that learn features of the same sequence length, we consider that each time series has its characteristics and the sequence length should be adjusted adaptively. Our motivation is that an "easy" sample with representative characteristics can be correctly predicted even when short feature representation is provided, while "hard" samples need complete feature representation. Therefore, we focus on sequence length and propose a dynamic length transformer (DLformer) that can adaptively learn sequence representation of different lengths. Then, a feature reuse mechanism is developed to utilize previously learned features to reduce redundant computation. Finally, in order to achieve dynamic feature representation, a particular confidence strategy is designed to calculate the confidence level for the prediction results. Regarding interpretability, the dynamic architecture can help human understand which part of the model is activated. Experiments on multiple datasets show that DLformer can increase up to 90% inference speed, with less than 5% degradation in model accuracy.

Transformer-CNN hybrid network for crowd counting

Efficient feature representation is the key to improving crowd counting performance. CNN and Transformer are the two commonly used feature extraction frameworks in the field of crowd counting. CNN excels at hierarchically extracting local features to obtain a multi-scale feature representation of the image, but it struggles with capturing global features. Transformer, on the other hand, could capture global feature representation by utilizing cascaded self-attention to capture remote dependency relationships, but it often overlooks local detail information. Therefore, relying solely on CNN or Transformer for crowd counting has certain limitations. In this paper, we propose the TCHNet crowd counting model by combining the CNN and Transformer frameworks. The model employs the CMT (CNNs Meet Vision Transformers) backbone network as the Feature Extraction Module (FEM) to hierarchically extract local and global features of the crowd using a combination of convolution and self-attention mechanisms. To obtain more comprehensive spatial local information, an improved Progressive Multi-scale Learning Process (PMLP) is introduced into the FEM, guiding the network to learn at different granularity levels. The features from these three different granularity levels are then fed into the Multi-scale Feature Aggregation Module (MFAM) for fusion. Finally, a Multi-Scale Regression Module (MSRM) is designed to handle the multi-scale fused features, resulting in crowd features rich in high-level semantics and low-level detail. Experimental results on five benchmark datasets demonstrate that TCHNet achieves highly competitive performance compared to some popular crowd counting methods.

PIS-Net: Efficient Medical Image Segmentation Network with Multivariate Downsampling for Point-of-Care.

Recently, with more portable diagnostic devices being moved to people anywhere, point-of-care (PoC) imaging has become more convenient and more popular than the traditional "bed imaging". Instant image segmentation, as an important technology of computer vision, is receiving more and more attention in PoC diagnosis. However, the image distortion caused by image preprocessing and the low resolution of medical images extracted by PoC devices are urgent problems that need to be solved. Moreover, more efficient feature representation is necessary in the design of instant image segmentation. In this paper, a new feature representation considering the relationships among local features with minimal parameters and a lower computational complexity is proposed. Since a feature window sliding along a diagonal can capture more pluralistic features, a Diagonal-Axial Multi-Layer Perceptron is designed to obtain the global correlation among local features for a more comprehensive feature representation. Additionally, a new multi-scale feature fusion is proposed to integrate nonlinear features with linear ones to obtain a more precise feature representation. Richer features are figured out. In order to improve the generalization of the models, a dynamic residual spatial pyramid pooling based on various receptive fields is constructed according to different sizes of images, which alleviates the influence of image distortion. The experimental results show that the proposed strategy has better performance on instant image segmentation. Notably, it yields an average improvement of 1.31% in Dice than existing strategies on the BUSI, ISIC2018 and MoNuSeg datasets.

Multimodal representation learning for tourism recommendation with two-tower architecture.

Personalized recommendation plays an important role in many online service fields. In the field of tourism recommendation, tourist attractions contain rich context and content information. These implicit features include not only text, but also images and videos. In order to make better use of these features, researchers usually introduce richer feature information or more efficient feature representation methods, but the unrestricted introduction of a large amount of feature information will undoubtedly reduce the performance of the recommendation system. We propose a novel heterogeneous multimodal representation learning method for tourism recommendation. The proposed model is based on two-tower architecture, in which the item tower handles multimodal latent features: Bidirectional Long Short-Term Memory (Bi-LSTM) is used to extract the text features of items, and an External Attention Transformer (EANet) is used to extract image features of items, and connect these feature vectors with item IDs to enrich the feature representation of items. In order to increase the expressiveness of the model, we introduce a deep fully connected stack layer to fuse multimodal feature vectors and capture the hidden relationship between them. The model is tested on the three different datasets, our model is better than the baseline models in NDCG and precision.

Research on an Enhanced Multimodal Network for Specific Emitter Identification

Specific emitter identification (SEI) refers to the task of distinguishing similar emitters, especially those of the same type and transmission parameters, which is one of the most critical tasks of electronic warfare. However, SEI is still a challenging task when a feature has low physical representation. Feature representation largely determines the recognition results. Therefore, this article expects to move toward robust feature representation for SEI. Efficient multimodal strategies have great potential for applications using multimodal data and can further improve the performance of SEI. In this research, we introduce a multimodal emitter identification method that explores the application of multimodal data, time-series radar signals, and feature vector data to an enhanced transformer, which employs a conformer block to embed the raw data and integrates an efficient multimodal feature representation module. Moreover, we employ self-knowledge distillation to mitigate overconfident predictions and reduce intra-class variations. Our study reveals that multimodal data provide sufficient information for specific emitter identification. Simultaneously, we propose the CV-CutMixOut method to augment the time-domain signal. Extensive experiments on real radar datasets indicate that the proposed method achieves more accurate identification results and higher feature discriminability.

Infrared Target Detection Based on Interval Sampling Weighting and 3D Attention Head in Complex Scenario

Thermal infrared detection technology can enable night vision and is robust in complex environments, making it highly advantageous for various fields. However, infrared images have low resolution and high noise, resulting in limited detailed information being available about the target object. This difficulty is further amplified when detecting small targets, which are prone to occlusion. In response to these challenges, we propose a model for infrared target detection designed to achieve efficient feature representation. Firstly, an interval sampling weighted (ISW) module is proposed, which strengthens the fusion network’s spatial relationship modeling, thereby elevating the model’s generalization capability across diverse target-density regions. Next, a detection head founded on 3D attention (TAHNet) is introduced, which helps the network more comprehensively understand the feature details of the target. This enhances the accuracy of the model in identifying the target object’s location, reduces false positives and false negatives, and optimizes the network’s performance. Furthermore, to our model, we introduce the C2f module to transfer gradient information across multiple branches. The features learned using diverse branches interact and fuse in subsequent stages, further enhancing the model’s representation ability and understanding of the target. Experimental outcomes validate the efficacy of the proposed model, showcasing state-of-the-art detection performance on FLIR and KAIST thermal infrared datasets and showing strong antiocclusion and robustness in complex scenes.

Unleashing the full potential of hyperspectral imaging: Decoupled image and frequency-domain spatial–spectral framework

Hyperspectral image classification (HSIC) is a rapidly developing field that utilizes deep learning methods. However, the reliance on convolutional neural networks (CNNs) for spectral–spatial feature extraction presents certain limitations. Specifically, the use of the fixed-position convolutional kernels in CNNs hinders their ability to capture fully the spectral information around the spatial central pixel, thereby overlooking critical differences between features. To address this issue, a decoupled image- and frequency-domain spectral–spatial framework for HSIC was developed in this study. This method incorporates image- and frequency-domain-based multiscale learnable convolutional attention to refine the differentiating features of the different feature distributions. Additionally, a novel frequency-domain information enhancement module was designed to extract the structural shape and texture details under the semantic constraints of the frequency phase, complementing the image domain to improve the extracted feature maps. Furthermore, a simple and efficient hierarchical feature representation module was introduced to extract both local and global information effectively from the fused features. The experimental results obtained using three open datasets and a practical hyperspectral image of the Gaofen-5 satellite demonstrate that the proposed method outperforms other state-of-the-art HSIC methods.

META-EEG: Meta-learning-based class-relevant EEG representation learning for zero-calibration brain–computer interfaces

Transfer learning for motor imagery-based brain–computer interfaces (MI-BCIs) struggles with inter-subject variability, hindering its generalization to new users. This paper proposes an advanced implicit transfer learning framework, META-EEG, designed to overcome the challenge arising from inter-subject variability. By incorporating gradient-based meta-learning with an intermittent freezing strategy, META-EEG ensures efficient feature representation learning, providing a robust zero-calibration solution.A comparative analysis reveals that META-EEG significantly outperforms all the baseline methods and competing methods on three different public datasets. Moreover, we demonstrate the efficiency of the proposed model through a neurophysiological and feature-representational analysis. With its robustness and superior performance on challenging datasets, META-EEG provides an effective solution for calibration-free MI-EEG classification, facilitating broader usability.

Manifold Explorer: Satellite Image Labelling and Clustering Tool with Using Deep Convolutional Autoencoders

We present a novel approach to providing greater insight into the characteristics of an unlabelled dataset, increasing the efficiency with which labelled datasets can be created. We leverage dimension-reduction techniques in combination with autoencoders to create an efficient feature representation for image tiles derived from remote sensing satellite imagery. The proposed methodology consists of two main stages. Firstly, an autoencoder network is utilised to reduce the high-dimensional image tile data into a compact and expressive latentfeature representation. Subsequently, features are further reduced to a two-dimensional embedding space using the manifold learning algorithm Uniform Manifold Approximation and Projection (UMAP) and t-distributed Stochastic Neighbour Embedding (t-SNE). This step enables the visualization of the image tile clusters in a 2D plot, providing an intuitive and interactive representation that can be used to aid rapid and geographically distributed image labelling. To facilitate the labelling process, our approach allows users to interact with the 2D visualization and label clusters based on their domain knowledge. In cases where certain classes are not effectively separated, users can re-apply dimension reduction to interactively refine subsets of clusters and achieve better class separation, enabling a comprehensively labelled dataset. We evaluate the proposed approach on real-world remote sensing satellite image datasets and demonstrate its effectiveness in achieving accurate and efficient image tile clustering and labelling. Users actively participate in the labelling process through our interactive approach, leading to enhanced relevance of the labelled data, by allowing domain experts to contribute their expertise and enrich the dataset for improved downstream analysis and applications.

Multimodal fusion for sensorimotor control in steering angle prediction

Efficient reasoning about the spatial and temporal structure of the environment is crucial for perception in autonomous driving, particularly in an end-to-end approach. Although different sensor modalities are employed to capture the complex nature of the environment, they each have their limitations. For example, frame-based RGB cameras are susceptible to variations in illumination conditions. However, these limitations at the sensor level can be addressed by complementing them with sensor fusion techniques, enabling the learning of efficient feature representations for end-to-end autonomous perception. In this study, we address the end-to-end perception problem by fusing a frame-based RGB camera with an event camera to improve the learned representation for predicting lateral control. To achieve this, we propose a convolutional encoder–decoder architecture called DRFuser. DRFuser encodes the features from both sensor modalities and leverages self-attention to fuse the frame-based RGB and event camera features in the encoder part. The decoder component unrolls the learned features to predict lateral control, specifically in the form of a steering angle. We extensively evaluate the proposed method on three datasets: our collected Dataset, Davis Driving dataset, and the EventScape dataset for simulation. The results demonstrate the generalization capability of our method on both real-world and simulated datasets. We observe qualitative and quantitative improvements in the performance of the proposed method for predicting lateral control by incorporating the event camera in fusion with the frame-based RGB camera. Notably, our method outperforms state-of-the-art techniques on the Davis Driving Dataset, achieving a 5.6% improvement in the root mean square error (RMSE) score.

Dense Papaya Target Detection in Natural Environment Based on Improved YOLOv5s

Due to the fact that the green features of papaya skin are the same colour as the leaves, the dense growth of fruits causes serious overlapping occlusion phenomenon between them, which increases the difficulty of target detection by the robot during the picking process. This study proposes an improved YOLOv5s-Papaya deep convolutional neural network for achieving dense multitarget papaya detection in natural orchard environments. The model is based on the YOLOv5s network architecture and incorporates the Ghost module to enhance its lightweight characteristics. The Ghost module employs a strategy of grouped convolutional layers and weighted fusion, allowing for more efficient feature representation and improved model performance. A coordinate attention module is introduced to improve the accuracy of identifying dense multitarget papayas. The fusion of bidirectional weighted feature pyramid networks in the PANet structure of the feature fusion layer enhances the performance of papaya detection at different scales. Moreover, the scaled intersection over union bounding box regression loss function is used rather than the complete intersection over union bounding box regression loss function to enhance the localisation accuracy of dense targets and expedite the convergence of the network model training. Experimental results show that the YOLOv5s-Papaya model achieves detection average precision, precision, and recall rates of 92.3%, 90.4%, and 83.4%, respectively. The model’s size, number of parameters, and floating-point operations are 11.5 MB, 6.2 M, and 12.8 G, respectively. Compared to the original YOLOv5s network model, the model detection average precision is improved by 3.6 percentage points, the precision is improved by 4.3 percentage points, the number of parameters is reduced by 11.4%, and the floating-point operations are decreased by 18.9%. The improved model has a lighter structure and better detection performance. This study provides the theoretical basis and technical support for intelligent picking recognition of overlapping and occluded dense papayas in natural environments.

Data-Efficient Image Quality Assessment with Attention-Panel Decoder

Blind Image Quality Assessment (BIQA) is a fundamental task in computer vision, which however remains unresolved due to the complex distortion conditions and diversified image contents. To confront this challenge, we in this paper propose a novel BIQA pipeline based on the Transformer architecture, which achieves an efficient quality-aware feature representation with much fewer data. More specifically, we consider the traditional fine-tuning in BIQA as an interpretation of the pre-trained model. In this way, we further introduce a Transformer decoder to refine the perceptual information of the CLS token from different perspectives. This enables our model to establish the quality-aware feature manifold efficiently while attaining a strong generalization capability. Meanwhile, inspired by the subjective evaluation behaviors of human, we introduce a novel attention panel mechanism, which improves the model performance and reduces the prediction uncertainty simultaneously. The proposed BIQA method maintains a light-weight design with only one layer of the decoder, yet extensive experiments on eight standard BIQA datasets (both synthetic and authentic) demonstrate its superior performance to the state-of-the-art BIQA methods, i.e., achieving the SRCC values of 0.875 (vs. 0.859 in LIVEC) and 0.980 (vs. 0.969 in LIVE). Checkpoints, logs and code will be available at https://github.com/narthchin/DEIQT.

AAGDN: Attention-Augmented Grasp Detection Network Based on Coordinate Attention and Effective Feature Fusion Method

High-precision robotic grasping is necessary for extensive grasping applications in the future. Most previous grasp detection methods fail to pay enough attention to learn grasp-related features and the detection accuracy is limited. In this letter, a novel attention-augmented grasp detection network (AAGDN) is presented to generate accurate grasp poses for unknown objects. The proposed AAGDN has three elaborate designs making it achieve higher accuracy than existing methods. First, we construct a coordinate attention residual module to extract positional information and improve the spatial sensitivity of features. Then, we propose an effective feature fusion module to bridge the resolution and semantic gaps of different-level features and obtain efficient feature representations. Lastly, a feature augmentation pyramid module is developed to enhance grasp- related features as needed and reduce the loss of information. Extensive experiments on three public datasets and various real-world scenes prove that the proposed AAGDN achieves better performance than current methods. Our model obtains the state-of-the-art 99.3% and 96.2% grasp detection accuracy on the Cornell and Jacquard dataset, respectively. Moreover, in physical grasping experiments, the AAGDN attains the 94.6% success rate for unseen objects in cluttered scenes, which further demonstrates the accuracy and robustness of our method to grasp novel objects.

End-to-end deep speaker embedding learning using multi-scale attentional fusion and graph neural networks

As an attractive research in biometric authentication, Text Independent Speaker Verification (TI-SV) problem aims to specify whether two given unconstrained utterances come from the same speaker or not. As state-of-the-art solutions, end-to-end approaches using deep neural networks seek to learn a highly discriminative speaker embedding space.In this paper, we propose a novel end-to-end approach for speaker embedding learning by focusing on two crucial factors: speaker embedder architecture and objective function. The proposed module in the speaker embedder is composed of an Efficient Multi-resolution feature Representation (EMR) block followed by a Multi-scale Channel Attention Fusion (MCAF) block. The EMR effectively addresses the issue of fixed resolution convolutional kernels which commonly used in most embedder architectures. Moreover, the MCAF significantly improves the simple summation-based feature fusion used in residual embedder networks. Regarding the objective function, we conduct the speaker embedding space towards learning the embedding-to-embedding relations, in addition to only embedding-to-training class relations employed by most previous methods. So, we propose to employ a dynamic graph attention network, on top of the proposed embedder to learn all informative relations between embeddings, and then learn both embedder and graph-based networks in an end-to-end manner.We conduct various experiments on a large-scale benchmark dataset called VoxCeleb1&2. The effectiveness of all proposed components is verified through an ablation study. We show the superior or competitive performances of the proposed approach compared to seven well-known embedding architectures and 32 SV systems, regarding two evaluation metrics, EER and minDCF, as well as the number of embedder parameters.

Deep learning for detecting and elucidating human T-cell leukemia virus type 1 integration in the human genome.

Human T-cell leukemia virus type 1 (HTLV-1), a retrovirus, is the causative agent for adult Tcell leukemia/lymphoma and many other human diseases. Accurate and high throughput detection of HTLV-1 virus integration sites (VISs) across the host genomes plays a crucial role in the prevention and treatment of HTLV-1-associated diseases. Here, we developed DeepHTLV, the first deep learning framework for VIS prediction de novo from genome sequence, motif discovery, and cis-regulatory factor identification. We demonstrated the high accuracy of DeepHTLV with more efficient and interpretive feature representations. Decoding the informative features captured by DeepHTLV resulted in eight representative clusters with consensus motifs for potential HTLV-1 integration. Furthermore, DeepHTLV revealed interesting cis-regulatory elements in regulation of VISs that have significant association with the detected motifs. Literature evidence demonstrated nearly half (34) of the predicted transcription factors enriched with VISs were involved in HTLV-1-associated diseases. DeepHTLV is freely available at https://github.com/bsml320/DeepHTLV.

Dual Contrastive Learning for Efficient Static Feature Representation in Sequential Recommendations

Dual Contrastive Learning for Efficient Static Feature Representation in Sequential Recommendations

Global Information-Assisted Fine-Grained Visual Categorization in Internet of Things

In fine-grained visual categorization (FGVC), most part-based frameworks do not work effectively in some extremely challenging scenarios such as partial occlusion. This limitation is due to the heavy disorder of local features extracted from such occluded targets. To address this issue, we propose a global information-assisted network (GIAN), where auxiliary global information can search the useful elements of local information and integrate with them for an efficient unified feature representation. In particular, in order to acquire the global information, we design a global attention-concentrated convolutional neural network (GAC-CNN) by extending a convolutional neural network with a nonlocal GCN module. Then, the unified feature representation is produced by two strategies. On the one hand, a global–local aggregation strategy is developed to selectively integrate global features with local features through consistency evaluation and reweighting method. On the other hand, an alternative knowledge distillation strategy is developed to help generate more powerful global and local features. Two strategies collaboratively make the unified features more robust and more discriminative than traditional part-based features. Experimental results show that the proposed GIAN can achieve accuracies of 92.8%, 93.8%, and 95.7% on CUB-200-2011, FGVC Aircraft, and Stanford Cars, respectively.