Multiscale Representation Research Articles

MRMNet: Multi-scale residual multi-branch neural network for object detection

Neural networks in object detection methods are designed to handle the challenging task of multi-scale object detection in computer vision. However, small objects occupy small blocks of pixels, and feature information can easily be overwhelmed, which further leads to certain shortcomings and deficiencies in the network’s multi-scale representation abilities. To alleviate this problem, we propose a novel multi-scale residual multi-branch neural network (MRMNet) for object detection. Particularly, we first propose a multi-scale extension (MSE) module to structurally construct optimal solutions to enhance feature extraction and alleviate information loss, thus effectively improving multi-scale object detection performance. Furthermore, we propose a contextual feature refinement (CFR) module, which draws on the idea of residuals to introduce a multi-residual structure and the feature of partial convolution, which can effectively alleviate the problem of small objects being overwhelmed by conflicting semantic information while avoiding the degradation problem of deep feature networks in some sense. Finally, by combining our own MSE module and integrating it into the existing Modified CSP v6 backbone, we propose a new backbone network, the multi-scale residual (MSRes) network. Experimental results show that our MRMNet and MSRes backbone achieve a competitive detection performance of 43.1% AP on the MS-COCO 2017 dataset.

Multi-scale image recognition strategy based on convolutional neural network

The accurate recognition and interpretation of multi-scale visual information is a critical focus within contemporary computer vision research. To this end, this study explores and innovatively constructs a multi-scale image recognition strategy based on a Convolutional Neural Network (CNN) with a multi-level and multi-resolution perception domain. This strategy is embedded with an advanced multi-level convolutional operation mechanism, which enables the model to intelligently explore and learn the multi-scale feature representation space of images from tiny texture to grand structure, from shallow simple features to deep semantic abstraction. The core technology path of this paper is to design a deep separable convolutional architecture and combine pyramid pool technology to form a unique network module. This modular design not only ensures the computational efficiency of the model but also improves the ability of extracting and integrating multi-scale image features. Following intensive experimentation on an array of extensively recognized and substantial image datasets, the multi-scale image recognition approach introduced in our study has demonstrated marked enhancements in both recognition capability and stability, manifesting clear superiority compared to conventional, single-scale image recognition methodologies. This research not only enriches the theoretical framework of image recognition, but also provides a new and efficient solution for dealing with complex multi-scale image recognition challenges in practical applications, and further promotes the development of image understanding and recognition technology.

Robust Fault Detection in Monitoring Chemical Processes Using Multi-Scale PCA with KD Approach

Effective fault detection in chemical processes is of utmost importance to ensure operational safety, minimize environmental impact, and optimize production efficiency. To enhance the monitoring of chemical processes under noisy conditions, an innovative statistical approach has been introduced in this study. The proposed approach, called Multiscale Principal Component Analysis (PCA), combines the dimensionality reduction capabilities of PCA with the noise reduction capabilities of wavelet-based filtering. The integrated approach focuses on extracting features from the multiscale representation, balancing the need to retain important process information while minimizing the impact of noise. For fault detection, the Kantorovich distance (KD)-driven monitoring scheme is employed based on features extracted from Multiscale PCA to efficiently detect anomalies in multivariate data. Moreover, a nonparametric decision threshold is employed through kernel density estimation to enhance the flexibility of the proposed approach. The detection performance of the proposed approach is investigated using data collected from distillation columns and continuously stirred tank reactors (CSTRs) under various noisy conditions. Different types of faults, including bias, intermittent, and drift faults, are considered. The results reveal the superior performance of the proposed multiscale PCA-KD based approach compared to conventional PCA and multiscale PCA-based monitoring methods.

CDAM-Net: Channel shuffle dual attention based multi-scale CNN for efficient glaucoma detection using fundus images

Glaucoma is a typical eye disorder that induces damage to the optic nerve due to increased intraocular pressure. It ultimately leads to partial or complete blindness without clinical reversal. Hence, it is of utmost importance to screen and detect glaucoma at an early stage. Most of the earlier glaucoma diagnosis methods rely on manual feature engineering, which is time-consuming and requires domain experts. Although recent methods, particularly, convolutional neural networks (CNNs) facilitate learning high-level feature representations from fundus images, they need extensive parameters and pose overfitting issues due to insufficient training samples. Further, conventional CNNs often ignore the minute changes in the lesion region. To overcome these issues, we propose a lightweight multi-scale CNN architecture called as CDAM-Net for effective glaucoma identification from retinal fundus images. Additionally, we introduce an attention module called channel shuffle dual attention (CSDA), comprising of a channel attention block, a spatial attention block and a channel shuffle unit, to focus on important regions in the fundus images, thereby extracting class-specific features. The CDAM-Net mainly consists of multi-scale feature representation (MFR) blocks that enable the extraction of multi-scale features from fundus images. Each MFR block is followed by a CSDA module, which further helps enrich the feature representation. The CDAM-Net is evaluated on a retinal fundus image (RFI) dataset containing 1426 fundus images (837 glaucoma and 589 normal), and the results indicate that CDAM-Net yields promising classification performance compared to existing techniques. Also, ablation studies are carried out to test the effectiveness of each component of the CDAM-Net.

Gait coordination feature modeling and multi-scale gait representation for gait recognition

Gait coordination feature modeling and multi-scale gait representation for gait recognition

INDIAN SIGN LANGUAGE RECOGNITION USING CNN WITH SPATIAL PYRAMID AND GLOBAL AVERAGE POOLING

Sign language recognition plays a pivotal role in bridging communication gaps for the speech and hearing-impaired community. In this research, we present an innovative approach to enhance the accuracy and effectiveness of Indian sign language recognition through the integration of convolutional neural networks with spatial pyramid and global average pooling layers. The objective of this study is to address the intricate challenges by exploiting multi-scale feature representations and global context information. This architecture incorporates spatial pyramid layer to capture fine-grained spatial information across multiple scales. Additionally, global average pooling layer is introduced to consolidate context-aware features, further improving the model's discriminative power. Experiments are conducted using a standard dataset, encompassing a diverse range of gestures. The results demonstrate a significant improvement in recognition accuracy compared to conventional models. Incorporating spatial pyramid and global average pooling layers enables our model to effectively recognize the gestures, even in challenging scenarios with variations in lighting, complex background, and signer-dependent factors by achieving high accuracy. This research not only contributes to the advancement of ISL recognition technology but also holds promise for practical applications in real-world communication and assistive technologies.

Transformer-CNN hybrid network for crowd counting

Efficient feature representation is the key to improving crowd counting performance. CNN and Transformer are the two commonly used feature extraction frameworks in the field of crowd counting. CNN excels at hierarchically extracting local features to obtain a multi-scale feature representation of the image, but it struggles with capturing global features. Transformer, on the other hand, could capture global feature representation by utilizing cascaded self-attention to capture remote dependency relationships, but it often overlooks local detail information. Therefore, relying solely on CNN or Transformer for crowd counting has certain limitations. In this paper, we propose the TCHNet crowd counting model by combining the CNN and Transformer frameworks. The model employs the CMT (CNNs Meet Vision Transformers) backbone network as the Feature Extraction Module (FEM) to hierarchically extract local and global features of the crowd using a combination of convolution and self-attention mechanisms. To obtain more comprehensive spatial local information, an improved Progressive Multi-scale Learning Process (PMLP) is introduced into the FEM, guiding the network to learn at different granularity levels. The features from these three different granularity levels are then fed into the Multi-scale Feature Aggregation Module (MFAM) for fusion. Finally, a Multi-Scale Regression Module (MSRM) is designed to handle the multi-scale fused features, resulting in crowd features rich in high-level semantics and low-level detail. Experimental results on five benchmark datasets demonstrate that TCHNet achieves highly competitive performance compared to some popular crowd counting methods.

TL-YOLO: Foreign-Object Detection on Power Transmission Line Based on Improved Yolov8

Foreign objects on power transmission lines carry a significant risk of triggering large-scale power interruptions which may have serious consequences for daily life if they are not detected and handled in time. To accurately detect foreign objects on power transmission lines, this paper proposes a TL-Yolo method based on the Yolov8 framework. Firstly, we design a full-dimensional dynamic convolution (ODConv) module as a backbone network to enhance the feature extraction capability, thus retaining richer semantic content and important visual features. Secondly, we present a feature fusion framework combining a weighted bidirectional feature pyramid network (BiFPN) and multiscale attention (MSA) module to mitigate the degradation effect of multiscale feature representation in the fusion process, and efficiently capture the high-level feature information and the core visual elements. Thirdly, we utilize a lightweight GSConv cross-stage partial network (GSCSP) to facilitate efficient cross-level feature fusion, significantly reducing the complexity and computation of the model. Finally, we employ the adaptive training sample selection (ATSS) strategy to balance the positive and negative samples, and dynamically adjust the selection process of the training samples according to the current state and performance of the model, thus effectively reducing the object misdetection and omission. The experimental results show that the average detection accuracy of the TL-Yolo method reaches 91.30%, which is 4.20% higher than that of the Yolov8 method. Meanwhile, the precision and recall metrics of our method are 4.64% and 3.53% higher than those of Yolov8. The visualization results also show the superior detection performance of the TL-Yolo algorithm in real scenes. Compared with the state-of-the-art methods, our method achieves higher accuracy and speed in the detection of foreign objects on power transmission lines.

MRUNet-3D: A multi-stride residual 3D UNet for lung nodule segmentation

Obtaining an accurate segmentation of the pulmonary nodules in computed tomography (CT) images is challenging. This is due to: (1) the heterogeneous nature of the lung nodules; (2) comparable visual characteristics between the nodules and their surroundings. A robust multi-scale feature extraction mechanism that can effectively obtain multi-scale representations at a granular level can improve segmentation accuracy. As the most commonly used network in lung nodule segmentation, UNet, its variants, and other image segmentation methods lack this robust feature extraction mechanism. In this study, we propose a multi-stride residual 3D UNet (MRUNet-3D) to improve the segmentation accuracy of lung nodules in CT images. It incorporates a multi-slide Res2Net block (MSR), which replaces the simple sequence of convolution layers in each encoder stage to effectively extract multi-scale features at a granular level from different receptive fields and resolutions while conserving the strengths of 3D UNet. The proposed method has been extensively evaluated on the publicly available LUNA16 dataset. Experimental results show that it achieves competitive segmentation performance with an average dice similarity coefficient of 83.47 % and an average surface distance of 0.35 mm on the dataset. More notably, our method has proven to be robust to the heterogeneity of lung nodules. It has also proven to perform better at segmenting small lung nodules. Ablation studies have shown that the proposed MSR and RFIA modules are fundamental to improving the performance of the proposed model.

Transfer-Aware Graph U-Net with Cross-Level Interactions for PolSAR Image Semantic Segmentation

Although graph convolutional networks have found application in polarimetric synthetic aperture radar (PolSAR) image classification tasks, the available approaches cannot operate on multiple graphs, which hinders their potential to generalize effective feature representations across different datasets. To overcome this limitation and achieve robust PolSAR image classification, this paper proposes a novel end-to-end cross-level interaction graph U-Net (CLIGUNet), where weighted max-relative spatial convolution is proposed to enable simultaneous learning of latent features from batch input. Moreover, it integrates weighted adjacency matrices, derived from the symmetric revised Wishart distance, to encode polarimetric similarity into weighted max-relative spatial graph convolution. Employing end-to-end trainable residual transformers with multi-head attention, our proposed cross-level interactions enable the decoder to fuse multi-scale graph feature representations, enhancing effective features from various scales through a deep supervision strategy. Additionally, multi-scale dynamic graphs are introduced to expand the receptive field, enabling trainable adjacency matrices with refined connectivity relationships and edge weights within each resolution. Experiments undertaken on real PolSAR datasets show the superiority of our CLIGUNet with respect to state-of-the-art networks in classification accuracy and robustness in handling unknown imagery with similar land covers.

Cross-level and multiscale CNN-Transformer network for automatic building extraction from remote sensing imagery

ABSTRACT Automatic building detection from remote sensing images holds significant applications in various domains, including cartography, land-use change detection, urban planning, and 3D intelligent city construction. Driven by the innovation of deep learning technology, convolutional neural networks (CNNs) have emerged as a powerful framework for automatic interpretation, exhibiting hierarchical feature generation and robust representation for building extraction. However, the inherent computational limitations of convolutional operators hinder the acquisition of an effective global receptive field. Recent advancements in deep learning have witnessed the adoption of the multi-head self-attention (MHSA) mechanism by Transformer, presenting the superiority of global context construction. Nonetheless, this network structure encounters difficulties in effectively modelling two-dimensional spatial structure features with scale variation in semantic segmentation and often has huge parameters with high algorithm complexity. To address the above problems, this paper proposed a hybrid deep neural network synergistically harnessing the superiority of CNN and Transformer to extract buildings from remote sensing images. Meanwhile, an improved MHSA mechanism is developed to facilitate the construction of multiscale global representations, concurrently reducing computational complexity. Furthermore, the cross-level MHSA modules the semantic correlation between the deep and shallow-level features to refine spatial fine-grained information. The proposed method was comprehensively evaluated on two building datasets via quantitative analysis and visualization. The experimental results confirmed that the developed network and modules outperform other state-of-the-art methods and can effectively improve the accuracy (93.4% F1-score on the WHU dataset, 89.38% F1-score on the CAMB dataset) with high efficiency for building extraction.

Multi-scale spatial pyramid attention mechanism for image recognition: An effective approach

Attention mechanisms have gradually become necessary to enhance the representational power of convolutional neural networks (CNNs). Despite recent progress in attention mechanism research, some open problems still exist. Most existing methods ignore modeling multi-scale feature representations, structural information, and long-range channel dependencies, which are essential for delivering more discriminative attention maps. This study proposes a novel, low-overhead, high-performance attention mechanism with strong generalization ability for various networks and datasets. This mechanism is called Multi-Scale Spatial Pyramid Attention (MSPA) and can be used to solve the limitations of other attention methods. For the critical components of MSPA, we not only develop the Hierarchical-Phantom Convolution (HPC) module, which can extract multi-scale spatial information at a more granular level utilizing hierarchical residual-like connections, but also design the Spatial Pyramid Recalibration (SPR) module, which can integrate structural regularization and structural information in an adaptive combination mechanism, while employing the Softmax operation to build long-range channel dependencies. The proposed MSPA is a powerful tool that can be conveniently embedded into various CNNs as a plug-and-play component. Correspondingly, using MSPA to replace the 3 × 3 convolution in the bottleneck residual blocks of ResNets, we created a series of simple and efficient backbones named MSPANet, which naturally inherit the advantages of MSPA. Without bells and whistles, our method substantially outperforms other state-of-the-art counterparts in all evaluation metrics based on extensive experimental results from CIFAR-100 and ImageNet-1K image recognition. When applying MSPA to ResNet-50, our model achieves top-1 classification accuracy of 81.74% and 78.40% on the CIFAR-100 and ImageNet-1K benchmarks, exceeding the corresponding baselines by 3.95% and 2.27%, respectively. We also obtained promising performance improvements of 1.15% and 0.91% compared to the competitive EPSANet-50. In addition, empirical research results in autonomous driving engineering applications also demonstrate that our method can significantly improve the accuracy and real-time performance of image recognition with cheaper overhead. Our code is publicly available at https://github.com/ndsclark/MSPANet.

DECTNet: Dual Encoder Network combined convolution and Transformer architecture for medical image segmentation.

Automatic and accurate segmentation of medical images plays an essential role in disease diagnosis and treatment planning. Convolution neural networks have achieved remarkable results in medical image segmentation in the past decade. Meanwhile, deep learning models based on Transformer architecture also succeeded tremendously in this domain. However, due to the ambiguity of the medical image boundary and the high complexity of physical organization structures, implementing effective structure extraction and accurate segmentation remains a problem requiring a solution. In this paper, we propose a novel Dual Encoder Network named DECTNet to alleviate this problem. Specifically, the DECTNet embraces four components, which are a convolution-based encoder, a Transformer-based encoder, a feature fusion decoder, and a deep supervision module. The convolutional structure encoder can extract fine spatial contextual details in images. Meanwhile, the Transformer structure encoder is designed using a hierarchical Swin Transformer architecture to model global contextual information. The novel feature fusion decoder integrates the multi-scale representation from two encoders and selects features that focus on segmentation tasks by channel attention mechanism. Further, a deep supervision module is used to accelerate the convergence of the proposed method. Extensive experiments demonstrate that, compared to the other seven models, the proposed method achieves state-of-the-art results on four segmentation tasks: skin lesion segmentation, polyp segmentation, Covid-19 lesion segmentation, and MRI cardiac segmentation.

Dense Contrastive-based Federated Learning for Dense Prediction Tasks on Medical Images.

Deep learning (DL) models have achieved remarkable success in various domains. But training an accurate DL model requires large amounts of data, which can be challenging to obtain in medical settings due to privacy concerns. Recently, federated learning (FL) has emerged as a promising solution that shares local models instead of raw data. However, FL in medical settings faces challenges of client drift due to the data heterogeneity across dispersed institutions. Although there exist studies to address this challenge, they mainly focus on the classification tasks that learn global representation of an entire image. Few have been studied on the dense prediction tasks, such as object detection. In this study, we propose dense contrastive-based federated learning (DCFL) tailored for dense prediction tasks in FL settings. DCFL introduces dense contrastive learning to FL, which aligns the local optimization objectives towards the global objective by maximizing the agreement of representations between the global and local models. Moreover, to improve the performance of dense target prediction at each level, DCFL applies multi-scale contrastive representation by utilizing multi-scale representations with dense features in contrastive learning. We evaluated DCFL on a set of realistic datasets for pulmonary nodule detection. DCFL demonstrates an overall performance improvement compared with the other federated learning methods in heterogeneous settings-improving the mean average precision by 4.13% and testing recall by 6.07% in highly heterogeneous settings.

Learning Aligned Vertex Convolutional Networks for Graph Classification.

Graph convolutional networks (GCNs) are powerful tools for graph structure data analysis. One main drawback arising in most existing GCN models is that of the oversmoothing problem, i.e., the vertex features abstracted from the existing graph convolution operation have previously tended to be indistinguishable if the GCN model has many convolutional layers (e.g., more than two layers). To address this problem, in this article, we propose a family of aligned vertex convolutional network (AVCN) models that focus on learning multiscale features from local-level vertices for graph classification. This is done by adopting a transitive vertex alignment algorithm to transform arbitrary-sized graphs into fixed-size grid structures. Furthermore, we define a new aligned vertex convolution operation that can effectively learn multiscale vertex characteristics by gradually aggregating local-level neighboring aligned vertices residing on the original grid structures into a new packed aligned vertex. With the new vertex convolution operation to hand, we propose two architectures for the AVCN models to extract different hierarchical multiscale vertex feature representations for graph classification. We show that the proposed models can avoid iteratively propagating redundant information between specific neighboring vertices, restricting the notorious oversmoothing problem arising in most spatial-based GCN models. Experimental evaluations on benchmark datasets demonstrate the effectiveness.

Improving PTM Site Prediction by Coupling of Multi-Granularity Structure and Multi-Scale Sequence Representation

Protein post-translational modification (PTM) site prediction is a fundamental task in bioinformatics. Several computational methods have been developed to predict PTM sites. However, existing methods ignore the structure information and merely utilize protein sequences. Furthermore, designing a more fine-grained structure representation learning method is urgently needed as PTM is a biological event that occurs at the atom granularity. In this paper, we propose a PTM site prediction method by Coupling of Multi-Granularity structure and Multi-Scale sequence representation, PTM-CMGMS for brevity. Specifically, multigranularity structure-aware representation learning is designed to learn neighborhood structure representations at the amino acid, atom, and whole protein granularity from AlphaFold predicted structures, followed by utilizing contrastive learning to optimize the structure representations. Additionally, multi-scale sequence representation learning is used to extract context sequence information, and motif generated by aligning all context sequences of PTM sites assists the prediction. Extensive experiments on three datasets show that PTM-CMGMS outperforms the state-of-the-art methods. Source code can be found at https://github.com/LZY-HZAU/PTM-CMGMS.

DDViT: Double-Level Fusion Domain Adapter Vision Transformer (Student Abstract)

With the help of Vision transformers (ViTs), medical image segmentation was able to achieve outstanding performance. In particular, they overcome the limitation of convolutional neural networks (CNNs) which rely on local receptive fields. ViTs use self-attention mechanisms to consider relationships between all image pixels or patches simultaneously. However, they require large datasets for training and did not perform well on capturing low-level features. To that end, we propose DDViT, a novel ViT model that unites a CNN to alleviate data-hunger for medical image segmentation with two multi-scale feature representations. Significantly, our approach incorporates a ViT with a plug-in domain adapter (DA) with Double-Level Fusion (DLF) technique, complemented by a mutual knowledge distillation paradigm, facilitating the seamless exchange of knowledge between a universal network and specialized domain-specific network branches. The DLF framework plays a pivotal role in our encoder-decoder architecture, combining the innovation of the TransFuse module with a robust CNN-based encoder. Extensive experimentation across diverse medical image segmentation datasets underscores the remarkable efficacy of DDViT when compared to alternative approaches based on CNNs and Transformer-based models.

Rethinking Multi-Scale Representations in Deep Deraining Transformer

Existing Transformer-based image deraining methods depend mostly on fixed single-input single-output U-Net architecture. In fact, this not only neglects the potentially explicit information from multiple image scales, but also lacks the capability of exploring the complementary implicit information across different scales. In this work, we rethink the multi-scale representations and design an effective multi-input multi-output framework that constructs intra- and inter-scale hierarchical modulation to better facilitate rain removal and help image restoration. We observe that rain levels reduce dramatically in coarser image scales, thus proposing to restore rain-free results from the coarsest scale to the finest scale in image pyramid inputs, which also alleviates the difficulty of model learning. Specifically, we integrate a sparsity-compensated Transformer block and a frequency-enhanced convolutional block into a coupled representation module, in order to jointly learn the intra-scale content-aware features. To facilitate representations learned at different scales to communicate with each other, we leverage a gated fusion module to adaptively aggregate the inter-scale spatial-aware features, which are rich in correlated information of rain appearances, leading to high-quality results. Extensive experiments demonstrate that our model achieves consistent gains on five benchmarks.

Omni-Kernel Network for Image Restoration

Image restoration aims to reconstruct a high-quality image from a degraded low-quality observation. Recently, Transformer models have achieved promising performance on image restoration tasks due to their powerful ability to model long-range dependencies. However, the quadratically growing complexity with respect to the input size makes them inapplicable to practical applications. In this paper, we develop an efficient convolutional network for image restoration by enhancing multi-scale representation learning. To this end, we propose an omni-kernel module that consists of three branches, i.e., global, large, and local branches, to learn global-to-local feature representations efficiently. Specifically, the global branch achieves a global perceptive field via the dual-domain channel attention and frequency-gated mechanism. Furthermore, to provide multi-grained receptive fields, the large branch is formulated via different shapes of depth-wise convolutions with unusually large kernel sizes. Moreover, we complement local information using a point-wise depth-wise convolution. Finally, the proposed network, dubbed OKNet, is established by inserting the omni-kernel module into the bottleneck position for efficiency. Extensive experiments demonstrate that our network achieves state-of-the-art performance on 11 benchmark datasets for three representative image restoration tasks, including image dehazing, image desnowing, and image defocus deblurring. The code is available at https://github.com/c-yn/OKNet.

Multi-scale hash encoding based neural geometry representation

Recently, neural implicit function-based representation has attracted more and more attention, and has been widely used to represent surfaces using differentiable neural networks. However, surface reconstruction from point clouds or multi-view images using existing neural geometry representations still suffer from slow computation and poor accuracy. To alleviate these issues, we propose a multi-scale hash encoding-based neural geometry representation which effectively and efficiently represents the surface as a signed distance field. Our novel neural network structure carefully combines low-frequency Fourier position encoding with multi-scale hash encoding. The initialization of the geometry network and geometry features of the rendering module are accordingly redesigned. Our experiments demonstrate that the proposed representation is at least 10 times faster for reconstructing point clouds with millions of points. It also significantly improves speed and accuracy of multi-view reconstruction. Our code and models are available at https://github.com/Dengzhi-USTC/Neural-Geometry-Reconstruction.