Pooling Operations Research Articles

Combined Channel Attention and Spatial Attention Module Network for Chinese Herbal Slices Automated Recognition.

Chinese Herbal Slices (CHS) are critical components of Traditional Chinese Medicine (TCM); the accurate recognition of CHS is crucial for applying to medicine, production, and education. However, existing methods to recognize the CHS are mainly performed by experienced professionals, which may not meet vast CHS market demand due to time-consuming and the limited number of professionals. Although some automated CHS recognition approaches have been proposed, the performance still needs further improvement because they are primarily based on the traditional machine learning with hand-crafted features, resulting in relatively low accuracy. Additionally, few CHS datasets are available for research aimed at practical application. To comprehensively address these problems, we propose a combined channel attention and spatial attention module network (CCSM-Net) for efficiently recognizing CHS with 2-D images. The CCSM-Net integrates channel and spatial attentions, focusing on the most important information as well as the position of the information of CHS image. Especially, pairs of max-pooling and average pooling operations are used in the CA and SA module to aggregate the channel information of the feature map. Then, a dataset of 14,196 images with 182 categories of commonly used CHS is constructed. We evaluated our framework on the constructed dataset. Experimental results show that the proposed CCSM-Net indicates promising performance and outperforms other typical deep learning algorithms, achieving a recognition rate of 99.27%, a precision of 99.33%, a recall of 99.27%, and an F1-score of 99.26% with different numbers of CHS categories.

CDTNet: Improved Image Classification Method Using Standard, Dilated and Transposed Convolutions

Convolutional neural networks (CNNs) have achieved great success in image classification tasks. In the process of a convolutional operation, a larger input area can capture more context information. Stacking several convolutional layers can enlarge the receptive field, but this increases the parameters. Most CNN models use pooling layers to extract important features, but the pooling operations cause information loss. Transposed convolution can increase the spatial size of the feature maps to recover the lost low-resolution information. In this study, we used two branches with different dilated rates to obtain different size features. The dilated convolution can capture richer information, and the outputs from the two channels are concatenated together as input for the next block. The small size feature maps of the top blocks are transposed to increase the spatial size of the feature maps to recover low-resolution prediction maps. We evaluated the model on three image classification benchmark datasets (CIFAR-10, SVHN, and FMNIST) with four state-of-the-art models, namely, VGG16, VGG19, ResNeXt, and DenseNet. The experimental results show that CDTNet achieved lower loss, higher accuracy, and faster convergence speed in the training and test stages. The average test accuracy of CDTNet increased by 54.81% at most on SVHN with VGG19 and by 1.28% at least on FMNIST with VGG16, which proves that CDTNet has better performance and strong generalization abilities, as well as fewer parameters.

Global–Local Enhancement Network for Short Text Classification

Because of the limited context information, it is a challenging task to classify short texts. Most existing methods only focus on extracting high-quality local features or global features from text to construct text representations, which is not comprehensive enough. This article proposes a global–local enhancement network (GLEN), which can construct a high-quality text representation by integrating the global and local features of the text. To improve the quality of global and local features, a group-wise enhancement mechanism is introduced, which can effectively enhance the important features while weakening the unimportant features. For the information-loss problem of traditional pooling operation, we designed a global–local pooling mechanism, which can filter out local features that are more relevant to the whole information of the text. Seven benchmark datasets of text classification were selected to test the performance of the model, and GLEN obtained the best results on most datasets. The ablation experiment on GLEN shows that both the group-wise enhancement mechanism and the global–local pooling mechanism can effectively improve the performance of the model.

Deep Learning-Based Hexrep Neural Network for Convergence Free with Operator’s Efficacy in Hexagonal Image Processing

The field of hexagonal image processing is concerned with the creation of image processing systems that combine the advantages of biological model-based evolutionary motivated frameworks. The structure and functionality of artificial neural networks were inspired by biological processes. The fundamental framework of recording and output devices limits their present state of the art. Prior neural networks have used square or hexagonal style input to completely connected layers, which resulted in a high coherence problem between two adjacent hexagonal kernel layers due to pooling. Previous research does not design the self-data structure to support convolution to increase computational efficiency, so it violates the convolution and pooling operator, which greatly degrades the image process performance. This paper introduces a novel paradigm Proficient Deep Learning-based Hexrep Neural Network that overcomes major significant problems in image operations structure constraint, coherence problem, and violation of convolution and pooling operator and achieves hexagonal image processing.

Interest Evolution-driven Gated Neighborhood aggregation representation for dynamic recommendation in e-commerce

Recommender system as an effective method to reduce information overload has been widely used in the e-commerce field. Existing studies mainly capture semantic features by considering user-item interactions or behavioral history records, which ignores the sparsity of interactions and the drift of user preferences. To cope with these challenges, we introduce the recently popular Graph Neural Networks (GNN) and propose an Interest Evolution-driven Gated Neighborhood (IEGN) aggregation representation model which can capture accurate user representation and track the evolution of user interests. Specifically, in IEGN, we explicitly model the relational information between neighbor nodes by introducing the gated adaptive propagation mechanism. Then, a personalized time interval function is designed to track the evolution of user interests. In addition, a high-order convolutional pooling operation is used to capture the correlation among the short-term interaction sequence. The user preferences are predicted by the fusion of user dynamic preferences and short-term interaction features. Extensive experiments on Amazon and Alibaba datasets show that IEGN outperforms several state-of-the-art methods in recommendation tasks.

CCA-Net: Clinical-awareness attention network for nuclear cataract classification in AS-OCT

Nuclear cataract (NC) is the leading cause of vision impairment and blindness globally. NC patients can slow the opacity development with early intervention or recover vision with cataract surgery. Anterior segment optical coherence tomography (AS-OCT) images have been increasingly used for clinical NC diagnosis. Compared with other ophthalmic images, e.g., slit lamp images, AS-OCT images are vital for NC diagnosis due to their capability of clearly capturing the nucleus region. Moreover, clinical research has shown the high correlation and repeatability between NC severity levels and image features like mean, maximum, and standard deviation on AS-OCT images. This paper aims to incorporate the clinical features into convolutional neural networks (CNNs) to improve NC classification results and enhance the interpretation of the decision process. Thus, we propose a novel clinical-awareness attention network (CCA-Net) to classify NC severity levels automatically. In CCA-Net, we design a practical yet effective clinical-aware attention block, which not only uses the mixed pooling operator to extract clinical features from each channel but also applies the designed clinical integration operator to focus on salient channels. We conduct extensive experiments on one clinical AS-OCT image dataset and two publicly available ophthalmology datasets. The results demonstrate that the CCA-Net outperforms state-of-the-art attention-based CNNs and strong baselines. Moreover, we also provide in-depth analysis to explain the internal behaviors of our method, enhancing the interpretation ability of our method.

Birdsong classification based on ensemble multi-scale convolutional neural network

With the intensification of ecosystem damage, birds have become the symbolic species of the ecosystem. Ornithology with interdisciplinary technical research plays a great significance for protecting birds and evaluating ecosystem quality. Deep learning shows great progress for birdsongs recognition. However, as the number of network layers increases in traditional CNN, semantic information gradually becomes richer and detailed information disappears. Secondly, the global information carried by the entire input may be lost in convolution, pooling, or other operations, and these problems will weaken the performance of classification. In order to solve such problems, based on the feature spectrogram from the wavelet transform for the birdsongs, this paper explored the multi-scale convolution neural network (MSCNN) and proposed an ensemble multi-scale convolution neural network (EMSCNN) classification framework. The experiments compared the MSCNN and EMSCNN models with other CNN models including LeNet, VGG16, ResNet101, MobileNetV2, EfficientNetB7, Darknet53 and SPP-net. The results showed that the MSCNN model achieved an accuracy of 89.61%, and EMSCNN achieved an accuracy of 91.49%. In the experiments on the recognition of 30 species of birds, our models effectively improved the classification effect with high stability and efficiency, indicating that the models have better generalization ability and are suitable for birdsongs species recognition. It provides methodological and technical scheme reference for bird classification research.

Spatial Channel Attention for Deep Convolutional Neural Networks

Recently, the attention mechanism combining spatial and channel information has been widely used in various deep convolutional neural networks (CNNs), proving its great potential in improving model performance. However, this usually uses 2D global pooling operations to compress spatial information or scaling methods to reduce the computational overhead in channel attention. These methods will result in severe information loss. Therefore, we propose a Spatial channel attention mechanism that captures cross-dimensional interaction, which does not involve dimensionality reduction and brings significant performance improvement with negligible computational overhead. The proposed attention mechanism can be seamlessly integrated into any convolutional neural network since it is a lightweight general module. Our method achieves a performance improvement of 2.08% on ResNet and 1.02% on MobileNetV2 in top-one error rate on the ImageNet dataset.

DEEP FEATURE EXTRACTION BASED ON DYNAMIC GRAPH CONVOLUTIONAL NETWORKS FOR ACCELERATED HYPERSPECTRAL IMAGE CLASSIFICATION

Abstract. Deep learning has achieved impressive results on hyperspectral images (HSIs) classification. Among them, supervised learning convolutional neural networks (CNNs) and semi-supervised learning graph neural networks (GNNs) are the two main network frameworks. However, 1) the supervised learning CNN faces the problem of high model time complexity as the number of network layers deepens; 2) the semi-supervised learning GNN faces the problem of high spatial complexity due to the computation of adjacency relations. In this paper, a novel dynamic graph convolutional HSI classification method is proposed, which is called dynamic graph convolutional networks (DGCNet). We first obtain two classification features by implementing flattening and global average pooling operation on the results of the convolution layer, which fully exploits the spatial-spectral information contained in the hyperspectral data. Then the dynamic graph convolution module is applied to extract the intrinsic structural information of each patch. Finally, HSI is classified based on spatial, spectral and structural features. DGCNet uses three branches to process multiple features of HSI in parallel and is trained in a supervised learning manner. In addition, DropBlock and label smoothing regularization techniques are applied to further improve the generalization capability of the model. Comparative experiments show that our proposed algorithm is comparable with the state-of-the art supervised learning models in terms of accuracy while also significantly outperforming in terms of time.

Apple Disease Recognition Based on Convolutional Neural Networks With Modified Softmax.

Accurate and rapid identification of apple diseases is the basis for preventing and treating the apple diseases, and is very significant for assessing disease disaster. Apple disease recognition from its diseased leaf images is one of the interesting research areas in computer and agriculture field. An apple disease recognition method is proposed based on modified convolutional neural networks (MCNN). In MCNN, Inception is introduced into MCNN, global average pooling (GAP) operator is employed instead of several fully connected layers to speedup training model, and modified Softmax classifier is used in the output layer to improve the recognition performance. The modified Softmax classifier uses the modified linear element as the activation function in the hidden layer and adds the local response normalization in MCNN to avoid the gradient disappearance problem effectively. A series of experiments are conducted on two kinds of apple disease image datasets. The results show the feasibility of the algorithm.

Hierarchical Representation Learning in Graph Neural Networks With Node Decimation Pooling

In graph neural networks (GNNs), pooling operators compute local summaries of input graphs to capture their global properties, and they are fundamental for building deep GNNs that learn hierarchical representations. In this work, we propose the Node Decimation Pooling (NDP), a pooling operator for GNNs that generates coarser graphs while preserving the overall graph topology. During training, the GNN learns new node representations and fits them to a pyramid of coarsened graphs, which is computed offline in a preprocessing stage. NDP consists of three steps. First, a node decimation procedure selects the nodes belonging to one side of the partition identified by a spectral algorithm that approximates the MAXCUT solution. Afterward, the selected nodes are connected with Kron reduction to form the coarsened graph. Finally, since the resulting graph is very dense, we apply a sparsification procedure that prunes the adjacency matrix of the coarsened graph to reduce the computational cost in the GNN. Notably, we show that it is possible to remove many edges without significantly altering the graph structure. Experimental results show that NDP is more efficient compared to state-of-the-art graph pooling operators while reaching, at the same time, competitive performance on a significant variety of graph classification tasks.

Boundary-aware context neural network for medical image segmentation.

Medical image segmentation can provide a reliable basis for further clinical analysis and disease diagnosis. With the development of convolutional neural networks (CNNs), medical image segmentation performance has advanced significantly. However, most existing CNN-based methods often produce unsatisfactory segmentation masks without accurate object boundaries. This problem is caused by the limited context information and inadequate discriminative feature maps after consecutive pooling and convolution operations. Additionally, medical images are characterized by high intra-class variation, inter-class indistinction and noise, extracting powerful context and aggregating discriminative features for fine-grained segmentation remain challenging. In this study, we formulate a boundary-aware context neural network (BA-Net) for 2D medical image segmentation to capture richer context and preserve fine spatial information, which incorporates encoder-decoder architecture. In each stage of the encoder sub-network, a proposed pyramid edge extraction module first obtains multi-granularity edge information. Then a newly designed mini multi-task learning module for jointly learning segments the object masks and detects lesion boundaries, in which a new interactive attention layer is introduced to bridge the two tasks. In this way, information complementarity between different tasks is achieved, which effectively leverages the boundary information to offer strong cues for better segmentation prediction. Finally, a cross feature fusion module acts to selectively aggregate multi-level features from the entire encoder sub-network. By cascading these three modules, richer context and fine-grain features of each stage are encoded and then delivered to the decoder. The results of extensive experiments on five datasets show that the proposed BA-Net outperforms state-of-the-art techniques.

NaroNet: Discovery of tumor microenvironment elements from highly multiplexed images.

Understanding the spatial interactions between the elements of the tumor microenvironment -i.e. tumor cells. fibroblasts, immune cells- and how these interactions relate to the diagnosis or prognosis of a tumor is one of the goals of computational pathology. We present NaroNet, a deep learning framework that models the multi-scale tumor microenvironment from multiplex-stained cancer tissue images and provides patient-level interpretable predictions using a seamless end-to-end learning pipeline. Trained only with multiplex-stained tissue images and their corresponding patient-level clinical labels, NaroNet unsupervisedly learns which cell phenotypes, cell neighborhoods, and neighborhood interactions have the highest influence to predict the correct label. To this end, NaroNet incorporates several novel and state-of-the-art deep learning techniques, such as patch-level contrastive learning, multi-level graph embeddings, a novel max-sum pooling operation, or a metric that quantifies the relevance that each microenvironment element has in the individual predictions. We validate NaroNet using synthetic data simulating multiplex-immunostained images where a patient label is artificially associated to the -adjustable- probabilistic incidence of different microenvironment elements. We then apply our model to two sets of images of human cancer tissues: 336 seven-color multiplex-immunostained images from 12 high-grade endometrial cancer patients; and 382 35-plex mass cytometry images from 215 breast cancer patients. In both synthetic and real datasets, NaroNet provides outstanding predictions of relevant clinical information while associating those predictions to the presence of specific microenvironment elements.

A New Framework Containing Convolution and Pooling Circuits for Image Processing and Deep Learning Applications with Quantum Computing Implementation

The resource need for deep learning and quantum computers' high computing power potential encourage collaboration between the two fields. Today, variational quantum circuits are used to perform the convolution operation with quantum computing. However, the results produced by variational circuits do not show a direct resemblance to the classical convolution operation. Because classical data is encoded into quantum data with their exact values in value-encoded methods, in contrast to variational quantum circuits, arithmetical operations can be applied with high accuracy. In this study, value-encoded quantum circuits for convolution and pooling operations are proposed to apply deep learning in quantum computers in a traditional and proven way. To construct the convolution and pooling operations, some modules such as addition, multiplication, division, and comparison are created. In addition, a window-based framework for quantum image processing applications is proposed. The generated convolution and pooling circuits are simulated on the IBM QISKIT simulator in parallel thanks to the proposed framework. The obtained results are verified by the expected results. Due to the limitations of quantum simulators and computers in the NISQ era, the used grayscale images are resized to 8x8 and the resolution of the images is reduced to 3 qubits. With developing the quantum technologies, the proposed approach can be applied for bigger and higher resolution images. Although the proposed method causes more qubit usage and circuit depth compared to variational convolutional circuits, the results they produce are exactly the same as the classical convolution process.

Spatiotemporal tubelet feature aggregation and object linking for small object detection in videos

This paper addresses the problem of exploiting spatiotemporal information to improve small object detection precision in video. We propose a two-stage object detector called FANet based on short-term spatiotemporal feature aggregation and long-term object linking to refine object detections. First, we generate a set of short tubelet proposals. Then, we aggregate RoI pooled deep features throughout the tubelet using a new temporal pooling operator that summarizes the information with a fixed output size independent of the tubelet length. In addition, we define a double head implementation that we feed with spatiotemporal information for spatiotemporal classification and with spatial information for object localization and spatial classification. Finally, a long-term linking method builds long tubes with the previously calculated short tubelets to overcome detection errors. The association strategy addresses the generally low overlap between instances of small objects in consecutive frames by reducing the influence of the overlap in the final linking score. We evaluated our model in three different datasets with small objects, outperforming previous state-of-the-art spatiotemporal object detectors and our spatial baseline.

HGNN: Hyperedge-based graph neural network for MOOC Course Recommendation

Previous studies on Course Recommendation (CR) mainly focus on investigating the sequential relationships among courses (RNN is applied) and fail to learn the similarity relationships among learners. Moreover, existing RNN-based methods can only model courses’ short-term sequential patterns due to the inherent shortcomings of RNNs. In light of the above issues, we develop a hyperedge-based graph neural network, namely HGNN, for CR. Specifically, (1) to model the relationships among learners, we treat learners (i.e., hyperedges) as the sets of courses in a hypergraph, and convert the task of learning learners’ representations to induce the embeddings for hyperedges, where a hyperedge-based graph attention network is further proposed. (2) To simultaneously consider courses’ long-term and short-term sequential relationships, we first construct a course sequential graph across learners, and learn courses’ representations via a modified graph attention network. Then, we feed the learned representations into a GRU-based sequence encoder to infer their short-term patterns, and deem the last hidden state as the learned sequence-level learner embedding. After that, we obtain the learners’ final representations by a product pooling operation to retain features from different latent spaces, and optimize a cross-entropy loss to make recommendations. To evaluate our proposed solution HGNN, we conduct extensive experiments on two real-world datasets, XuetangX and MovieLens. We conduct experiments on MovieLens to prove the extensibility of our solution on other collections. From the experimental results, we can find that HGNN evidently outperforms other recent CR methods on both datasets, achieving 11.96% on P@20, 16.01% on NDCG@20, and 27.62% on MRR@20 on XuetangX, demonstrating the effectiveness of studying CR in a hypergraph, and the importance of considering both long-term and short-term sequential patterns of courses.

Brain tumor magnetic resonance image segmentation by a multiscale contextual attention module combined with a deep residual UNet (MCA-ResUNet)

Background and Objective. Automatic segmentation of MRI brain tumor area is a key step in the diagnosis and treatment of brain tumor. In recent years, the improved network based on UNet encoding and decoding structure has been widely used in brain tumor segmentation. However, due to continuous convolution and pooling operations, some spatial context information in existing networks will be discontinuous or even missing. It will affect the segmentation accuracy of the model. Therefore, the method proposed in this paper is to alleviate the lack of spatial context information and improve the accuracy of the model. Approach. This paper proposes a context attention module (multiscale contextual attention) to capture and filter out high-level features with spatial context information, which solves the problem of context information loss in feature extraction. The channel attention mechanism is introduced into the decoding structure to realize the fusion of high-level features and low-level features. The standard convolution block in the encoding and decoding structure is replaced by the pre-activated residual block to optimize the network training and improve the network performance. Results. This paper uses two public data sets (BraTs 2017 and BraTs 2019) to evaluate and verify the proposed method. Experimental results show that the proposed method can effectively alleviate the lack of spatial context information, and the segmentation performance is better than other existing methods. Significance. The method improves the segmentation performance of the model. It will assist doctors in making accurate diagnosis and provide reference basis for tumor resection. As a result, the proposed method will reduce the operation risk of patients and the postoperative recurrence rate.

Vision-Based Power Line Segmentation With an Attention Fusion Network

Automatic power transmission line detection plays the key role in smart grid which has been widely applied into the path planning and navigation of intelligent inspection platforms, such as Unmanned Aerial Vehicles (UAVs), climbing robots, hybrid inspection robots, etc. Nevertheless, the power lines are always against complex background environment and different illumination conditions. And the power lines occupy a minimal portion image pixels in the aerial images compared with backgrounds which causes the foreground-background class imbalance issue. Therefore, robust and accurate vision-based power line detection still faces a certain challenge. Recently, deep learning has got fast development on pixel-level object segmentation due to strong contextual feature expression ability, especially U-shape network (U-Net) and its variants. However, it still exists a certain shortcomings owing to insufficient process of local contextual features to affect the segmentation precision. Meanwhile, multiple pooling operations in deep convolutional neural networks (DCNNs) also will cause the information loss. To address these issues, with the encoder-decoder architecture, a novel vision-based power line detection network is proposed in this paper to construct an end-to-end detection scheme of power lines from aerial images. To make the segmentation network capture the global contexts and emphasize target regions of power transmission lines, an attention block is proposed to be embedded into the proposed power line detection network to address the class imbalance issue. Meanwhile, faced with the insufficient process of local contextual feature maps of DCNNs, an attention fusion block is proposed for multi-scale feature fusion to acquire more rich information and improve the segmentation precision. Experiments on power lines show that the proposed power line detection network shows a good segmentation performance on real power line environment compared with other advanced detection methods.

DeepLION: Deep Multi-Instance Learning Improves the Prediction of Cancer-Associated T Cell Receptors for Accurate Cancer Detection.

Recent studies highlight the potential of T cell receptor (TCR) repertoires in accurately detecting cancers via noninvasive sampling. Unfortunately, due to the complicated associations among cancer antigens and the possible induced T cell responses, currently, the practical strategy for identifying cancer-associated TCRs is the computational prediction based on TCR repertoire data. Several state-of-the-art methods were proposed in recent year or two; however, the prediction algorithms were still weakened by two major issues. To facilitate the computational processes, the algorithms prefer to decompose the original TCR sequences into length-fixed amino acid fragments, while the first dilemma comes as the lengths of cancer-associated motifs are suggested to be various. Moreover, the correlations among TCRs in the same repertoire should be further considered, which are often ignored by the existing methods. We here developed a deep multi-instance learning method, named DeepLION, to improve the prediction of cancer-associated TCRs by considering these issues. First, DeepLION introduced a deep learning framework with alternative convolution filters and 1-max pooling operations to handle the amino acid fragments with different lengths. Then, the multi-instance learning framework modeled the TCR correlations and assigned adjusted weights for each TCR sequence during the predicting process. To validate the performance of DeepLION, we conducted a series of experiments on several cohorts of patients from nine cancer types. Compared to the existing methods, DeepLION achieved, on most of the cohorts, higher prediction accuracies, sensitivities, specificities, and areas under the curve (AUCs), where the AUC reached notably 0.97 and 0.90 for thyroid and lung cancer cohorts, respectively. Thus, DeepLION may further support the detection of cancers from TCR repertoire data. DeepLION is publicly available on GitHub, at https://github.com/Bioinformatics7181/DeepLION, for academic usage only.

Cascaded context enhancement network for automatic skin lesion segmentation

Skin lesion segmentation is an important step for automatic melanoma diagnosis. Due to the non-negligible diversity of lesions from different patients, extracting powerful context for fine-grained semantic segmentation is still challenging today. Although the deep convolutional neural networks (CNNs) have made significant improvements on skin lesion segmentation, they often fail to reserve the spatial details and long-range dependencies context due to consecutive convolution striding and pooling operations inside CNNs. In this paper, we formulate a cascaded context enhancement neural network for automatic skin lesion segmentation. A new cascaded context aggregation (CCA) module with a gate-based information integration approach is proposed to sequentially and selectively aggregate original image and multi-level features from the encoder sub-network. The generated context is further utilized to guide discriminative features extraction by the designed context-guided local affinity (CGL) module. Furthermore, an auxiliary loss is added to the CCA module for refining the prediction. In our work, we evaluate our approach on four public skin dermoscopy image datasets. The proposed method achieves the Jaccard Index (JA) of 87.1%, 80.3%, 84.3%, and 86.6% on ISIC-2016, ISIC-2017, ISIC-2018, and PH2 datasets, which show highly competitive performance with other state-of-the-art models respectively.