Pooling Operations Research Articles

DPNet: Dual-Path Network for Real-Time Object Detection With Lightweight Attention.

The recent advances in compressing high-accuracy convolutional neural networks (CNNs) have witnessed remarkable progress in real-time object detection. To accelerate detection speed, lightweight detectors always have few convolution layers using a single-path backbone. Single-path architecture, however, involves continuous pooling and downsampling operations, always resulting in coarse and inaccurate feature maps that are disadvantageous to locate objects. On the other hand, due to limited network capacity, recent lightweight networks are often weak in representing large-scale visual data. To address these problems, we present a dual-path network, named DPNet, with a lightweight attention scheme for real-time object detection. The dual-path architecture enables us to extract in parallel high-level semantic features and low-level object details. Although DPNet has a nearly duplicated shape with respect to single-path detectors, the computational costs and model size are not significantly increased. To enhance representation capability, a lightweight self-correlation module (LSCM) is designed to capture global interactions, with only a few computational overheads and network parameters. In the neck, LSCM is extended into a lightweight cross correlation module (LCCM), capturing mutual dependencies among neighboring scale features. We have conducted exhaustive experiments on MS COCO, Pascal VOC 2007, and ImageNet datasets. The experimental results demonstrate that DPNet achieves a state-of-the-art trade off between detection accuracy and implementation efficiency. More specifically, DPNet achieves 31.3% AP on MS COCO test-dev, 82.7% mAP on Pascal VOC 2007 test set, and 41.6% mAP on ImageNet validation set, together with nearly 2.5M model size, 1.04 GFLOPs, and 164 and 196 frames/s (FPS) FPS for [Formula: see text] input images of three datasets.

ResGEM: Multi-scale Graph Embedding Network for Residual Mesh Denoising.

Mesh denoising is a crucial technology that aims to recover a high-fidelity 3D mesh from a noise-corrupted one. Deep learning methods, particularly graph convolutional networks (GCNs) based mesh denoisers, have demonstrated their effectiveness in removing various complex real-world noises while preserving authentic geometry. However, it is still a quite challenging work to faithfully regress uncontaminated normals and vertices on meshes with irregular topology. In this paper, we propose a novel pipeline that incorporates two parallel normal-aware and vertex-aware branches to achieve a balance between smoothness and geometric details while maintaining the flexibility of surface topology. We introduce ResGEM, a new GCN, with multi-scale embedding modules and residual decoding structures to facilitate normal regression and vertex modification for mesh denoising. To effectively extract multi-scale surface features while avoiding the loss of topological information caused by graph pooling or coarsening operations, we encode the noisy normal and vertex graphs using four edge-conditioned embedding modules (EEMs) at different scales. This allows us to obtain favorable feature representations with multiple receptive field sizes. Formulating the denoising problem into a residual learning problem, the decoder incorporates residual blocks to accurately predict true normals and vertex offsets from the embedded feature space. Moreover, we propose novel regularization terms in the loss function that enhance the smoothing and generalization ability of our network by imposing constraints on normal consistency. Comprehensive experiments have been conducted to demonstrate the superiority of our method over the state-of-the-art on both synthetic and real-scanned datasets.

Learning Attention in the Frequency Domain for Flexible Real Photograph Denoising.

Recent advancements in deep learning techniques have pushed forward the frontiers of real photograph denoising. However, due to the inherent pooling operations in the spatial domain, current CNN-based denoisers are biased towards focusing on low-frequency representations, while discarding the high-frequency components. This will induce a problem for suboptimal visual quality as the image denoising tasks target completely eliminating the complex noises and recovering all fine-scale and salient information. In this work, we tackle this challenge from the frequency perspective and present a new solution pipeline, coined as frequency attention denoising network (FADNet). Our key idea is to build a learning-based frequency attention framework, where the feature correlations on a broader frequency spectrum can be fully characterized, thus enhancing the representational power of the network across multiple frequency channels. Based on this, we design a cascade of adaptive instance residual modules (AIRMs). In each AIRM, we first transform the spatial-domain features into the frequency space. Then, a learning-based frequency attention framework is devised to explore the feature inter-dependencies converted in the frequency domain. Besides this, we introduce an adaptive layer by leveraging the guidance of the estimated noise map and intermediate features to meet the challenges of model generalization in the noise discrepancy. The effectiveness of our method is demonstrated on several real camera benchmark datasets, with superior denoising performance, generalization capability, and efficiency versus the state-of-the-art.

Multi-objective pedestrian tracking method based on YOLOv8 and improved DeepSORT.

A multi-objective pedestrian tracking method based on you only look once-v8 (YOLOv8) and the improved simple online and real time tracking with a deep association metric (DeepSORT) was proposed with the purpose of coping with the issues of local occlusion and ID dynamic transformation that frequently arise when tracking target pedestrians in real complex traffic scenarios. To begin with, in order to enhance the feature extraction network's capacity to learn target feature information in busy traffic situations, the detector implemented the YOLOv8 method with a high level of small-scale feature expression. In addition, the omni-scale network (OSNet) feature extraction network was then put on top of DeepSORT in order to accomplish real-time synchronized target tracking. This increases the effectiveness of picture edge recognition by dynamically fusing the collected feature information at various scales. Furthermore, a new adaptive forgetting smoothing Kalman filtering algorithm (FSA) was created to adapt to the nonlinear condition of the pedestrian trajectory in the traffic scene in order to address the issue of poor prediction attributed to the linear state equation of Kalman filtering once more. Afterward, the original intersection over union (IOU) association matching algorithm of DeepSORT was replaced by the complete-intersection over union (CIOU) association matching algorithm to fundamentally reduce the target pedestrians' omission and misdetection situation and to improve the accuracy of data matching. Eventually, the generalized trajectory feature extractor model (GFModel) was developed to tightly merge the local and global information through the average pooling operation in order to get precise tracking results and further decrease the impact of numerous disturbances on target tracking. The fusion algorithm of YOLOv8 and improved DeepSORT method based on OSNet, FSA and GFModel was named YOFGD. According to the experimental findings, YOFGD's ultimate accuracy can reach 77.9% and its speed can reach 55.8 frames per second (FPS), which is more than enough to fulfill the demands of real-world scenarios.

Two‐dimensional medical image segmentation based on U‐shaped structure

AbstractWith rapid developments in convolutional neural networks for image processing, deep learning methods based on pixel classification have been extensively applied in medical image segmentation. One popular strategy for such tasks is the encoder‐decoder‐based U‐Net architecture and its variants. Most segmentation methods based on fully convolutional networks will cause the loss of spatial and contextual information due to continuous pooling operations or strided convolution when decreasing image resolution, and make less use of contextual information and global information under different receptive fields. To overcome this shortcoming, this paper proposes a novel structure called RAAU‐Net. In our proposed RAAU‐Net structure, which is a modified U‐shaped architecture, we aim to capture high‐level information while preserving spatial information and focusing on the regions of interest. RAAU‐Net comprises three main components: a feature encoder module that utilizes a pre‐trained ResNet‐18 model as a fixed feature extractor, a multi‐receptive field extraction module that we developed, and a feature decoder module. We have tested our method on several 2D medical image segmentation tasks such as retinal nerve, breast tumor, skin lesion, lung, gland, and polyp segmentation. All the indexes of the model reached the best in the dataset of skin lesions, in which Accuracy, Precision, IoU, Recall, and Dice Score were 3.26%, 5.42%, 9.92%, 6.52%, and 5.95% higher than UNet.

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning.

The spiking neural networks (SNNs) that efficiently encode temporal sequences have shown great potential in extracting audio-visual joint feature representations. However, coupling SNNs (binary spike sequences) with transformers (float-point sequences) to jointly explore the temporal-semantic information still facing challenges. In this paper, we introduce a novel Spiking Tucker Fusion Transformer (STFT) for audio-visual zero-shot learning (ZSL). The STFT leverage the temporal and semantic information from different time steps to generate robust representations. The time-step factor (TSF) is introduced to dynamically synthesis the subsequent inference information. To guide the formation of input membrane potentials and reduce the spike noise, we propose a global-local pooling (GLP) which combines the max and average pooling operations. Furthermore, the thresholds of the spiking neurons are dynamically adjusted based on semantic and temporal cues. Integrating the temporal and semantic information extracted by SNNs and Transformers are difficult due to the increased number of parameters in a straightforward bilinear model. To address this, we introduce a temporal-semantic Tucker fusion module, which achieves multi-scale fusion of SNN and Transformer outputs while maintaining full second-order interactions. Our experimental results demonstrate the effectiveness of the proposed approach in achieving state-of-the-art performance in three benchmark datasets. The harmonic mean (HM) improvement of VGGSound, UCF101 and ActivityNet are around 15.4%, 3.9%, and 14.9%, respectively.

Graph Reconfigurable Pooling for Graph Representation Learning

In recent years, graph neural networks have been widely used for tasks such as graph classification, link prediction, and node classification, and have achieved excellent results. In order to apply GNNs to graph classification tasks, recent works generate graph-level representations using node representations through a hierarchical pooling approach. Existing graph pooling methods such as DiffPool and EigenPool encourage adjacent nodes to be assigned to the same cluster, making the node assignment process similar to the graph partitioning process that ignores the role of nodes or some substructures (e.g., amino acids) in the process of composing a graph (e.g., proteins). In this paper, we propose a new pooling operator RecPool to capture the role played by nodes in the process of composing a graph. Specifically, we probabilistically model the feature distribution of the coarsened graph, construct the feature distribution of each cluster, resample the features of the coarsened graph into the original nodes according to the soft assignment matrix, reconstruct the original graph, and optimize the soft assignment matrix to divide the nodes that play the same role in the reconstruction process into the same cluster. The excellent performance of Recpool is demonstrated through experiments on four public benchmark dataset.

Pulmonary Nodule Detection from 3D CT Image with a Two-Stage Network

Early detection of lung nodules is an important means of reducing the lung cancer mortality rate. In this paper, we propose a three-dimensional CT image lung nodule detection method based on parallel pooling and dense blocks, which includes two parts, i.e., candidate nodule extraction and false positive suppression. First, a dense U-shaped backbone network with parallel pooling is proposed to obtain the candidate nodule probability map. The parallel pooling structure uses multiple pooling operations for downsampling to capture spatial information comprehensively and address the problem of information loss resulting from maximum and average pooling in the shallow layers. Then, a parasitic network with parallel pooling, dense blocks, and attention modules is designed to suppress false positive nodules. The parasitic network takes the multiscale feature maps of the backbone network as the input. The experimental results demonstrate that the proposed method significantly improves the accuracy of lung nodule detection, achieving a CPM score of 0.91, which outperforms many existing methods.

MSDANet: A multi-scale dilation attention network for medical image segmentation

Deep learning shows excellent performance in medical image segmentation. However, a pooling operation in its encoding stage leads to feature loss and the ability of multi-scale contextual information extraction is insufficient. In addition, there is a semantic gap between low-level features and high-level features in the encoder–decoder. Inspired by an U-shaped network (UNet), this work proposes a multi-scale dilation attention network (MSDANet) for medical image segmentation. Specifically, it is mainly composed of a parallel dilation pooling module (PD), a multi-scale channel attention mechanism (MSCA), a multilayer perceptron squeeze and excitation module (MSE), a big kernel convolution module (BC), and a skip feature pyramid module (SFP). The proposed PD reduces the loss of subtle features during downsampling. In order to improve the extraction of effective features, MSCA, MSE and BC are used in encoding and decoding stages. In the skip connection stage, SPF is used to reduce the difference between the low-level and high-level semantic information to accelerate the network learning. We validate the proposed model on two publicly available medical image datasets. The results show that MSDANet has superior performance on several evaluation metrics compared to state-of-the-art methods. Code is available at https://github.com/1999luan/MSDANet.

A fault diagnosis method based on feature-level fusion of multi-sensor information for rotating machinery

Traditionally, most fault diagnosis work on rotating machinery is carried out on single sensor datasets. However, the single feature source may suffer from missing or inaccurate features, which is especially sluggish for fault diagnosis tasks under noise interference. Feature-level fusion of multi-sensor information can obtain more comprehensive and abundant feature information, while improving the feature discrimination. Therefore, through feature-level fusion of multi-sensor information, a parallel multi-scale attentional convolutional neural network (PMSACNN) is proposed in this paper to achieve rotating machinery fault diagnosis. A dilated wide convolutional layer is designed to extract the short-time features of signals with noise by performing sparse sampling on them. The multi-scale structure is constructed to capture the diversity feature information of signals, and the feature-level stitching of multi-sensor information is realized by the parallel input mechanism. Feature fusion is achieved by adaptively correcting the importance of different channel features by using channel attention. The global averaging pooling operation is introduced to reduce the number of parameters and improve the efficiency of the model operation. The effectiveness of PMSACNN is verified by using the bearing dataset acquired from the mechanical comprehensive diagnosis simulation platform. The experimental results indicate that the proposed method outperforms the existing methods of this field in terms of fault diagnosis accuracy and noise immunity, which can improve the reliability and safety of rotating machinery.

Hallucinated-PQA: No reference point cloud quality assessment via injecting pseudo-reference features

Point clouds (PCs) have been increasingly applied in business and life, but a variety of PC transmission and generation systems inevitably produce various types of distortions. Therefore, it is of great significance to design an objective point cloud quality assessment (PCQA), particularly in a no-reference manner to evaluate the PC systems in the actual situation. However, due to the lack of original PCs, the existing no-reference (NR) PCQA metrics cannot perceive the feature changes caused by distortions, resulting in inaccurate quality prediction. In addition, dependent on the viewing habits of the subjects in the subjective evaluation experiment, a strong contextual correlation among intra- and inter-view local regions at different scales naturally exists, but the existing projection-based no-reference PCQA only using concatenation or average pooling operations cannot reflect this relationship of multi-view features fusion. Considering the above challenges, we propose a novel hallucination-guided NR PCQA framework, namely Hallucinated-PQA. Specifically, we introduce a distortion restoration network to correct multiple projected images in preprocessing to provide pseudo-reference information for NR PCQA. In the feature extraction of distorted PCs, we designed a hallucination injection block (HIB) by utilizing feature differences to assist the feature description of distorted PCs, and a multi-view and multi-scale context fusion (MMCF) module to construct the contextual correlation among intra- and inter-view local regions at different scales. Experimental results show that our Hallucinated-PQA can achieve comparable or better performance than state-of-the-art (SOTA) metrics on four open PCQA databases. The source code will be released at https://github.com/QSBAOYANGMU/Hallucinated-PQA.

Point Cloud Segmentation Network Based on Attention Mechanism and Dual Graph Convolution

To overcome the limitations of inadequate local feature representation and the underutilization of global information in dynamic graph convolutions, we propose a network that combines attention mechanisms with dual graph convolutions. Firstly, we construct a static graph based on the dynamic graph using the K-nearest neighbors algorithm and geometric distances of point clouds. This integration of dynamic and static graphs forms a dual graph structure, compensating for the underutilization of geometric positional relationships in the dynamic graph. Next, edge convolutions are applied to extract edge features from the dual graph structure. To further enhance the capturing ability of local features, we employ attention pooling, which combines max pooling and average pooling operations. Secondly, we introduce channel attention modules and spatial self-attention modules to improve the representation ability of global features and enhance semantic segmentation accuracy in our network. Experimental results on the S3DIS dataset demonstrate that compared to dynamic graph convolution alone, our proposed approach effectively utilizes both semantic and geometric relationships between point clouds using dual graph convolutions while addressing limitations related to insufficient local feature extraction. The introduction of attention mechanisms helps mitigate underutilization issues with global information, resulting in significant improvements in model performance.

LFEA-Net: semantic segmentation for urban point cloud scene via local feature extraction and aggregation

Considering the increasing prominence of 3D real city construction technology, 3D urban point cloud scene data merit further investigation. However, achieving finegrained semantic segmentation of urban scenes remains highly challenging due to the natural orderlessness and unstructured nature of acquired point clouds, along with their large-scale points and non-uniform distributions. In this study, we present LFEA-Net, a novel neural network specifically designed for semantic segmentation of large-scale urban point cloud scenes. The network comprises two main components: (1) The local feature extraction (LFE) module, which fully exploits local spatial, color and semantic information to enhance and preserve crucial information, consisting of bilateral feature encoding unit and multidimensional feature encoding unit. (2) The local feature aggregation (LFA) module, designed to bridge the semantic gap between local information and emphasize both local significant features and the entire local neighbor, consisting of soft cross operation and united pooling operation. We have evaluated the performance of LFEA-Net with state-of-the-art networks using the photogrammetric point cloud dataset SensatUrban, achieving 61.6 of mIoU score. The results demonstrate the superior efficacy of LFEA-Net in accurately segmenting and classifying large-scale urban point cloud scenes, highlighting its potential to advance environmental information perception.

MI-EEG: Generalized model based on mutual information for EEG emotion recognition without adversarial training

EEG-based emotion classification is a vital aspect of human–machine interfaces. However, inter-subject variability poses a challenge for accurate domain-agnostic EEG emotion recognition, often requiring individual model calibration with a robust base model for fine-tuning. To overcome this limitation and develop a generalized model, we propose a Generalized Model based on Mutual Information for EEG Emotion Recognition without Adversarial Training (MI-EEG). The MI-EEG model leverages disentanglement to extract shared features, wherein it separates EEG features into domain-invariant class-relevant features and other features. To avoid adversarial training, mutual information minimization is applied during the decoupling process. Additionally, mutual information maximization is used to enrich the features by strengthening the relationship between domain-invariant class-relevant features and emotion labels. Furthermore, the transformer-based feature extractor, which utilizes a multi-headed attention mechanism and pooling operations, enhances the feature quality in the time dimension. The experimental evaluation on two emotional EEG datasets demonstrates the superior performance of the proposed EEG-MI model compared to existing state-of-the-art methods.

A CNN-based network with attention mechanism for autonomous crack identification on building facade

ABSTRACT This paper presents a rapid and precise deep learning-based approach for measuring cracks in concrete structures. The proposed methodology involves data acquisition, pre-processing, model construction and post-processing. The Deeplabv3+ model is used for crack detection, and the use of Coordinate Attention is introduced to enhance its performance. And the implications of attention locations in the model are discussed. Insertion position after atrous spatial pyramid pooling (ASPP) operation is the most effective and accurate. The crack widths are obtained through post-processing, and the actual width is determined using the width conversion equation. The proposed method achieves an effective crack detection MIoU of 85.95 and a calculated width error of 6.3%, with a reduction of 2.5% compared to traditional models. Numerical experiments and real-world building experiments have demonstrated the feasibility and effectiveness of the proposed method. Overall, the proposed method presents a fast and dependable crack detection technique that has great potential for practical application.

M5UMCB: Prediction of RNA 5-methyluridine sites using multi-scale convolutional neural network with BiLSTM

As a prevalent RNA modification, 5-methyluridine (m5U) plays a critical role in diverse biological processes and disease pathogenesis. High-throughput identification of m5U typically relies on labor-intensive biochemical experiments using various sequencing-based techniques, which are not only time-consuming but also expensive. Consequently, there is a pressing need for more efficient and cost-effective computational methods to complement these high-throughput techniques. In this study, we present m5UMCB, a novel approach that harnesses a multi-scale convolutional neural network (CNN) in tandem with bidirectional long short-term memory (BiLSTM) to recognize m5U sites. Our method involves segmenting RNA sequences into smaller fragments based on a 3-mer length and subsequently mapping each fragment to a lower-dimensional vector representation using the global vectors for word representation (GloVe) technique. Through a series of multi-scale convolution and pooling operations, local features are extracted from RNA sequences and transformed into abstract, high-level features. The feature matrix is then inputted into a BiLSTM network, enabling the capture of contextual information and long-term dependencies within the sequence. Ultimately, a fully connected layer is employed to classify m5U sites. The validation results from 5-fold cross-validation (5-fold CV) test indicate that m5UMCB outperforms existing state-of-the-art predictive methods, demonstrating a 1.98% increase in the area under ROC curve (AUC) and significant improvements in relevant evaluation metrics. We are confident that m5UMCB will serve as a valuable tool for m5U prediction.

Regularized Optimal Transport Layers for Generalized Global Pooling Operations.

Global pooling is one of the most significant operations in many machine learning models and tasks, which works for information fusion and structured data (like sets and graphs) representation. However, without solid mathematical fundamentals, its practical implementations often depend on empirical mechanisms and thus lead to sub-optimal, even unsatisfactory performance. In this work, we develop a novel and generalized global pooling framework through the lens of optimal transport. The proposed framework is interpretable from the perspective of expectation-maximization. Essentially, it aims at learning an optimal transport across sample indices and feature dimensions, making the corresponding pooling operation maximize the conditional expectation of input data. We demonstrate that most existing pooling methods are equivalent to solving a regularized optimal transport (ROT) problem with different specializations, and more sophisticated pooling operations can be implemented by hierarchically solving multiple ROT problems. Making the parameters of the ROT problem learnable, we develop a family of regularized optimal transport pooling (ROTP) layers. We implement the ROTP layers as a new kind of deep implicit layer. Their model architectures correspond to different optimization algorithms. We test our ROTP layers in several representative set-level machine learning scenarios, including multi-instance learning (MIL), graph classification, graph set representation, and image classification. Experimental results show that applying our ROTP layers can reduce the difficulty of the design and selection of global pooling - our ROTP layers may either imitate some existing global pooling methods or lead to some new pooling layers fitting data better. The code is available at https://github.com/SDS-Lab/ROT-Pooling.

Fine-grained Species Recognition with Privileged Pooling: Better Sample Efficiency Through Supervised Attention.

We propose a scheme for supervised image classification that uses privileged information, in the form of keypoint annotations for the training data, to learn strong models from small and/or biased training sets. Our main motivation is the recognition of animal species for ecological applications such as biodiversity modelling, which is challenging because of long-tailed species distributions due to rare species, and strong dataset biases such as repetitive scene background in camera traps. To counteract these challenges, we propose a visual attention mechanism that is supervised via keypoint annotations that highlight important object parts. This privileged information, implemented as a novel privileged pooling operation, is only required during training and helps the model to focus on regions that are discriminative. In experiments with three different animal species datasets, we show that deep networks with privileged pooling can use small training sets more efficiently and generalize better.

Modelling local and general quantum mechanical properties with attention-based pooling

Atom-centred neural networks represent the state-of-the-art for approximating the quantum chemical properties of molecules, such as internal energies. While the design of machine learning architectures that respect chemical principles has continued to advance, the final atom pooling operation that is necessary to convert from atomic to molecular representations in most models remains relatively undeveloped. The most common choices, sum and average pooling, compute molecular representations that are naturally a good fit for many physical properties, while satisfying properties such as permutation invariance which are desirable from a geometric deep learning perspective. However, there are growing concerns that such simplistic functions might have limited representational power, while also being suboptimal for physical properties that are highly localised or intensive. Based on recent advances in graph representation learning, we investigate the use of a learnable pooling function that leverages an attention mechanism to model interactions between atom representations. The proposed pooling operation is a drop-in replacement requiring no changes to any of the other architectural components. Using SchNet and DimeNet++ as starting models, we demonstrate consistent uplifts in performance compared to sum and mean pooling and a recent physics-aware pooling operation designed specifically for orbital energies, on several datasets, properties, and levels of theory, with up to 85% improvements depending on the specific task.

Global routing between capsules

Current convolutional neural networks (CNNs) lack viewpoint equivariance. Hence, they perform poorly when dealing with viewpoints unseen during training procedures. CNN achieves invariance via pooling operations in image classification tasks. However, the pooling operation does not necessarily improve viewpoint generalization, rather relying on more data to achieve viewpoint equivariance. Capsule network (CapsNet) is proposed to tackle this issue, but it is inefficient and inaccurate when applied to complex datasets. We propose a novel CapsNet architecture called Global Routing CapsNet (GR-CapsNet) to solve this problem. Specifically, colored background in the input image can generate invalid background voting capsules to reduce the performance of CapsNet. Therefore, we first construct a dynamic linear unit (DLU), which avoids the generation of invalid background voting capsules. Then we present two extra learnable units: frequency domain unit (FDU) and spatial unit (SPU). The former is used to capture finer features in the frequency domain and aims to improve classification performance on complex datasets. The latter is applied to construct the spatial relationship between the voting capsules and component capsules and aims to enhance robustness to affine transformation. Finally, we propose a global routing mechanism to simplify the routing process for CapsNet, which obtains more feature information to improve the performance of CapsNet. Extensive experiments on nine datasets show that our method obtains better robustness and generalization and achieves SOTA performance compared to other related methods. And it has fewer the number of parameters and GPU memory consumption than these related methods. The source code is available on https://github.com/cwpl/GR-CapsNet.