Network Architecture Research Articles

Investigation of Topological properties of Butterfly Networks by Irregularity Invariants

In the modern era, networks play a crucial role in connecting individuals, organizations, and devices, enabling efficient communication, collaboration, and information exchange across the globe. They serve as the backbone of the digital age, supporting the growth of technologies such as the internet, cloud computing, IoT, and social media. The examination of Quantitative Structure Activity/Property Relationships (QSPR) to analyse the network architecture uses topological indices as its main instrument. The butterfly network is a type of interconnection network that follows a specific geometric structure. It consists of multiple stages, with each stage connecting nodes in a binary pattern. The connections between nodes form a butterfly-like shape, hence the name. The butterfly network provides efficient routing and communication paths between nodes, making it suitable for parallel computing and distributed systems. With the aid of sixteen irregularity indices, we will examine and compare three butterfly networks in this article. We will then plot the findings to see how they depend on the various parameters.

Fast-Vid2Vid++: Spatial-Temporal Distillation for Real-Time Video-to-Video Synthesis.

Video-to-Video synthesis (Vid2Vid) gains remarkable performance in generating a photo-realistic video from a sequence of semantic maps, such as segmentation, sketch and pose. However, this pipeline is heavily limited to high computational cost and long inference latency, mainly attributed to two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have been significantly reduced via more efficient network architectures. Existing methods mainly focus on slimming network architectures but ignore the size of the sequential data stream. Moreover, due to the lack of temporal coherence, image-based compression is not sufficient for the compression of the video task. In this paper, we present a spatial-temporal hybrid distillation compression framework, Fast-Vid2Vid++, which focuses on knowledge distillation of the teacher network and the data stream of generative models on both space and time. Fast-Vid2Vid++ makes the first attempt at time dimension to transfer hierarchical features and time coherence knowledge to reduce computational resources and accelerate inference. Specifically, we compress the data stream spatially and reduce the temporal redundancy. We distill the knowledge of the hierarchical features and the final response from the teacher network to the student network in high-resolution and full-time domains. We transfer the long-term dependencies of the features and video frames to the student model. After the proposed spatial-temporal hybrid knowledge distillation (Spatial-Temporal-HKD), our model can synthesize high-resolution key-frames using the low-resolution data stream. Finally, Fast-Vid2Vid++ interpolates intermediate frames by motion compensation with slight latency and generates full-length sequences with motion-aware inference (MAI). On standard benchmarks, Fast-Vid2Vid++ achieves a real-time performance of 30-59 FPS and saves 28-35× computational cost on a single V100 GPU.

EPMF: Efficient Perception-Aware Multi-Sensor Fusion for 3D Semantic Segmentation.

We study multi-sensor fusion for 3D semantic segmentation that is important to scene understanding for many applications, such as autonomous driving and robotics. Existing fusion-based methods, however, may not achieve promising performance due to the vast difference between the two modalities. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to effectively exploit perceptual information from two modalities, namely, appearance information from RGB images and spatio-depth information from point clouds. To this end, we project point clouds to the camera coordinate using perspective projection, and process both inputs from LiDAR and cameras in 2D space while preventing the information loss of RGB images. Then, we propose a two-stream network to extract features from the two modalities, separately. The extracted features are fused by effective residual-based fusion modules. Moreover, we introduce additional perception-aware losses to measure the perceptual difference between the two modalities. Last, we propose an improved version of PMF, i.e., EPMF, which is more efficient and effective by optimizing data pre-processing and network architecture under perspective projection. Specifically, we propose cross-modal alignment and cropping to obtain tight inputs and reduce unnecessary computational costs. We then explore more efficient contextual modules under perspective projection and fuse the LiDAR features into the camera stream to boost the performance of the two-stream network. Extensive experiments on benchmark data sets show the superiority of our method. For example, on nuScenes test set, our EPMF outperforms the state-of-the-art method, i.e., RangeFormer, by 0.9% in mIoU.

Pixel Distillation: Cost-Flexible Distillation Across Image Sizes and Heterogeneous Networks.

Previous knowledge distillation (KD) methods mostly focus on compressing network architectures, which is not thorough enough in deployment as some costs like transmission bandwidth and imaging equipment are related to the image size. Therefore, we propose Pixel Distillation that extends knowledge distillation into the input level while simultaneously breaking architecture constraints. Such a scheme can achieve flexible cost control for deployment, as it allows the system to adjust both network architecture and image quality according to the overall requirement of resources. Specifically, we first propose an input spatial representation distillation (ISRD) mechanism to transfer spatial knowledge from large images to student's input module, which can facilitate stable knowledge transfer between CNN and ViT. Then, a Teacher-Assistant-Student (TAS) framework is further established to disentangle pixel distillation into the model compression stage and input compression stage, which significantly reduces the overall complexity of pixel distillation and the difficulty of distilling intermediate knowledge. Finally, we adapt pixel distillation to object detection via an aligned feature for preservation (AFP) strategy for TAS, which aligns output dimensions of detectors at each stage by manipulating features and anchors of the assistant. Comprehensive experiments on image classification and object detection demonstrate the effectiveness of our method.

Weak Augmentation Guided Relational Self-Supervised Learning.

Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most methods mainly focus on the instance level information (i.e., the different augmented images of the same instance should have the same feature or cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduce a novel SSL paradigm, which we term as relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, our proposed method employs sharpened distribution of pairwise similarities among different instances as relation metric, which is thus utilized to match the feature embeddings of different augmentations. To boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. The designed asymmetric predictor head and an InfoNCE warm-up strategy enhance the robustness to hyper-parameters and benefit the resulting performance. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures, including various lightweight networks (e.g., EfficientNet and MobileNet).

Water Demand Forecasting in Multiple District Metered Areas Based on a Multi-Scale Correction Module Neural Network Architecture

Short-term water demand forecasting (STWDF) for multiple spatially and temporally correlated District Metering Areas (DMAs) is an essential foundation for achieving more refined management of urban water supply networks. However, due to the greater uncertainty associated with specific DMA demand compared to overall water usage, accurately predicting STWDF poses significant challenges. This study introduces an innovative network architecture—the multi-scale correction module neural network, built upon Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNN) enhanced with Attention mechanisms—for simultaneous STWDF with a temporal resolution of one hour over a week for 10 DMAs located in a single city in northern Italy. This framework utilizes multivariate corrections to refine and enhance the output accuracy. The results reveal that, in comparison to traditional Gated Recurrent Unit or LSTM models, the proposed model with integrated correction modules, particularly those that leverage inter-DMA correlations, improves performance across all evaluation metrics by an average of 5%-20% per DMA. Additionally, it consistently delivers superior accuracy across three scenarios: single DMA forecasting, total water demand, and extreme conditions, while maintaining stable performance throughout. Furthermore, the interpretability analysis underscores the feasibility of this innovative structure and highlights the contribution of meteorological features to the predictive model in some DMA-level STWDF. The unified input-output framework elegantly simplifies the STWDF process across multiple DMAs, providing new insights and methodologies for future research in this domain.

Pneumonia detection on x-ray image using improved depthwise separable convolutional neural networks

A single neural network model cannot capture intricate and diverse features due to its ability to learn only a finite set of patterns from the data. Additionally, training and utilising a single model can be computationally demanding. Experts propose incorporating multiple neural network models to address these constraints to extract complementary attributes. Previous research has highlighted challenges network models face, including difficulties in effectively capturing highly detailed spatial features, redundancy in network structure parameters, and restricted generalisation capabilities. This study introduces an innovative neural network architecture that combines the Xception module with the inverse residue structure to tackle these issues. Considering this, the paper presents a model for detecting pneumonia in X-ray images employing an improved depthwise separable convolutional network. This network architecture integrates the inverse residual structure from the MobileNetV2 model, using the rectified linear unit (ReLU) non-linear activation function throughout the entire network. The experimental results show an impressive recognition rate with a test accuracy of 97.24% on the chest x-ray dataset. This method can extract more profound and abstract image features while mitigating overfitting issues and enhancing the network's generalisation capacity.

An effective surrogate-assisted rank method for evolutionary neural architecture search

Evolutionary neural architecture search (ENAS) is able to automatically design high-performed architectures of deep neural networks (DNNs) for specific tasks. In recent years, surrogate models have gained significant traction because they can estimate the performance of neural architectures, avoiding excessive computational costs for training. However, most existing surrogate models primarily predict the performance of architectures directly or predict pairwise comparison relationships, which makes it challenging to obtain the rank of a group of architectures when training samples are limited. To address this problem, we propose TCMR-ENAS, an effective triple-competition model-assisted rank method for ENAS. TCMR-ENAS employs a novel triple-competition surrogate model combined with a score-based fitness evaluation method to predict group performance rank. Moreover, a progressive online learning method is proposed to enhance the predictive performance of the triple-competition surrogate model in the framework of modified genetic search. To validate the effectiveness of TCMR-ENAS, we conducted a series of experiments on NAS-Bench-101, NAS-Bench-201, NATS-Bench and NAS-Bench-301, respectively. Experimental results show that TCMR-ENAS can achieve better performance with lower computational resources. The accuracies of searched architectures achieve the best results compared with those of the state-of-the-art methods with limited training samples. In addition, the factors that may influence the effectiveness of TCMR-ENAS are explored in the ablation studies.

Broad Distributed Game Learning for intelligent classification in rolling bearing fault diagnosis

As a new Single Layer Feedforward Network (SLFN) architecture, Broad Learning System (BLS) has been widely used in the field of fault diagnosis because of its fast-training speed and high generalization capability. However, when features in different classes of signals are similar or weak, BLS generates a large number of redundant features that may be difficult to classify accurately. In view of this, a new Broad Distributed Game Learning (BDGL) method is proposed in this paper, which maps data into the game space by constructing two non-parallel game hyperplanes to achieve game and segmentation of different similar features, thereby making the data linearly differentiable in the game space. Meanwhile, a linear distribution constraint term is designed to reduce noise fitting and weak feature learning in training data learning by limiting the complexity of model parameters, thereby making the solution of the objective function simpler and faster. By comparing the Precision, Recall, F-score, Kappa and Accuracy of BDGL and the comparison methods on the two types of rolling bearing experimental data, the results show that BDGL has a high classification accuracy. In addition, the experimental results on small and noisy samples once again demonstrate the effectiveness of BDGL, which provides an efficient solution for rolling bearing fault diagnosis.

MPCNN: a novel approach for detecting human Monkeypox from skin lesion images leveraging deep neural network

The global healthcare scenario encounters a substantial challenge caused by the widespread outbreak of Monkeypox affecting over 65 countries. Limited availability of polymerase chain reaction (PCR) tests and biochemical assays necessitates alternative strategies. This study explores the viability of computer-aided identification of Monkeypox through the analysis of skin lesion images, offering a potential solution, particularly in resource-constrained settings. Employing data augmentation techniques, we augment the dataset to enhance its robustness. Subsequently, we utilize various pre-trained deep learning models, including EfficientNetB3, VGG16, ResNet50, AlexNet, and EfficientNet for classification tasks related to Monkeypox and other diseases. The achieved accuracies for these models are 98.48%, 69.19%, 91.41%, 78.38%, and 94.44%, respectively. We introduce a novel modified convolutional neural network (CNN) architecture named MPCNN to further improve performance. Our proposed MPCNN model demonstrates exceptional accuracy, precisely identifying Monkeypox patients with a remarkable precision of 99.49%. This technological advancement in disease identification holds significant promise for enhancing healthcare strategies and response mechanisms in the context of global health concerns.

Seeding precision: a mask region based convolutional neural networks classification approach for the classification of paddy seeds

The generation of sufficient training data that is accurately labelled for a deep neural network involves a significant amount of effort and frequently constitutes a bottleneck in the implementation process. For the purpose of this research, we are training a neural network model to perform instance segmentation and classification of crop seeds for various rice cultivars. Synthetically constructed dataset is used here. The concept of domain randomization, which offers a productive alternative to the laborious process of data annotation, serves as the basis for our methodology. We make use of the domain randomization technique in order to produce synthetic data, and the mask region-based convolutional neural network (Mask R-CNN) architecture is utilized in order to train our neural network models. A cultivar name is used to designate the seeds, and they are differentiated from one another using colors that are comparable to those used in the actual dataset of paddy cultivars. Our mission focuses on the identification and categorization of rice paddy varieties within automatically generated photographs. Farmers are able to accurately sort crop seeds from a variety of rice cultivars with the use of this approach, which is particularly useful for phenotyping and optimizing yields in laboratory settings.

Lightweight macro-pixel quality enhancement network for light field images compressed by versatile video coding

Previous research demonstrated that filtering Macro-Pixels (MPs) in a decoded Light Field Image (LFI) sequence can effectively enhances the quality of the corresponding Sub-Aperture Images (SAIs). In this paper, we propose a deep-learning-based quality enhancement model following the MP-wise processing approach tailored to LFIs encoded by the Versatile Video Coding (VVC) standard. The proposed novel Res2Net Quality Enhancement Convolutional Neural Network (R2NQE-CNN) architecture is both lightweight and powerful, in which the Res2Net modules are employed to perform LFI filtering for the first time, and are implemented with a novel improved 3D-feature-processing structure. The proposed method incorporates only 205K model parameters and achieves significant Y-BD-rate reductions over VVC of up to 32%, representing a relative improvement of up to 33% compared to the state-of-the-art method, which has more than three times the number of parameters of our proposed model.

Precise Localization for Anatomo-Physiological Hallmarks of the Cervical Spine by Using Neural Memory Ordinary Differential Equation.

In the evaluation of cervical spine disorders, precise positioning of anatomo-physiological hallmarks is fundamental for calculating diverse measurement metrics. Despite the fact that deep learning has achieved impressive results in the field of keypoint localization, there are still many limitations when facing medical image. First, these methods often encounter limitations when faced with the inherent variability in cervical spine datasets, arising from imaging factors. Second, predicting keypoints for only 4% of the entire X-ray image surface area poses a significant challenge. To tackle these issues, we propose a deep neural network architecture, NF-DEKR, specifically tailored for predicting keypoints in cervical spine physiological anatomy. Leveraging neural memory ordinary differential equation with its distinctive memory learning separation and convergence to a singular global attractor characteristic, our design effectively mitigates inherent data variability. Simultaneously, we introduce a Multi-Resolution Focus module to preprocess feature maps before entering the disentangled regression branch and the heatmap branch. Employing a differentiated strategy for feature maps of varying scales, this approach yields more accurate predictions of densely localized keypoints. We construct a medical dataset, SCUSpineXray, comprising X-ray images annotated by orthopedic specialists and conduct similar experiments on the publicly available UWSpineCT dataset. Experimental results demonstrate that compared to the baseline DEKR network, our proposed method enhances average precision by 2% to 3%, accompanied by a marginal increase in model parameters and the floating-point operations (FLOPs). The code (https://github.com/Zhxyi/NF-DEKR) is available.

Auxiliary diagnosis of primary bone tumors based on Machine learning model

ObjectiveResearch on auxiliary diagnosis of primary bone tumors can enhance diagnostic accuracy, facilitate early detection, and enable personalized treatment, thereby reducing misdiagnosis and missed cases, ultimately leading to improved patient prognosis and survival rates. In this study, we established a whole slide imaging (WSI) database comprising histopathological samples from all categories of bone tumors and integrated multiple neural network architectures for machine learning models. We then evaluated the accuracy of these models in diagnosing primary bone tumors. MethodsIn this paper, the machine learning model based on the deep convolutional neural network (DC-NN) method was combined with imaging omics analysis to analyze and discuss its clinical value in diagnosing primary bone tumors. In addition, this paper proposed a screening method for differentially expressed genes. Based on the paired T-test method, the process first estimated the tumor purity in the experimental data of each sample case, then assessed the actual gene expression value of the experimental data of each sample case, and finally calculated the optimized paired T-test statistics, and screened differentially expressed genes according to the threshold value. ResultsThe selected model demonstrated excellent diagnostic accuracy in distinguishing between normal and tumor images, with overall accuracy of (99.8 ± 0.4) % for five rounds of testing using the DCNN model and positive and negative predictive values of (100.0 ± 0.0) % and (99.6 ± 0.8) %, respectively. The mean area under each dataset’s curve (AUC) was (0.998 ± 0.004). Further, ten rounds of testing using the DCNN model showed an overall accuracy of (71.2 ± 1.6) % and a substantial positive predictive value of (91.9 ± 8.5) % in distinguishing benign from malignant bone tumors, with an average AUC of (0.62 ± 0.06) across datasets. ConclusionThe deep learning model accurately classifies bone tumor histopathology based on the degree of infiltration, achieving diagnostic performance comparable to that of senior pathologists. These findings affirm the feasibility and effectiveness of histopathological diagnosis in bone tumors, providing a theoretical foundation for the application and advancement of machine learning-assisted histopathological diagnosis in this field.

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition.

Vision Transformers have been the most popular network architecture in visual recognition recently due to the strong ability of encode global information. However, its high computational cost when processing high-resolution images limits the applications in downstream tasks. In this paper, we take a deep look at the internal structure of self-attention and present a simple Transformer style convolutional neural network (ConvNet) for visual recognition. By comparing the design principles of the recent ConvNets and Vision Transformers, we propose to simplify the self-attention by leveraging a convolutional modulation operation. We show that such a simple approach can better take advantage of the large kernels ( ≥ 7×7) nested in convolutional layers and we observe a consistent performance improvement when gradually increasing the kernel size from 5×5 to 21×21. We build a family of hierarchical ConvNets using the proposed convolutional modulation, termed Conv2Former. Our network is simple and easy to follow. Experiments show that our Conv2Former outperforms existent popular ConvNets and vision Transformers, like Swin Transformer and ConvNeXt in all ImageNet classification, COCO object detection and ADE20 k semantic segmentation.

Robust and energy-efficient RPL optimization algorithm with scalable deep reinforcement learning for IIoT

The increasing complexity and quantity of the Industrial Internet of Things (IIoT) pose new challenges to the traditional routing protocol for low-power and lossy networks (RPL) in terms of dynamic management, data transmission reliability, and energy efficiency optimization. This paper proposes a scalable deep reinforcement learning (DRL) algorithm with a multi-attention actor double critic model for routing optimization (MADC) to meet the requirements of IIoT for efficient and intelligent routing decisions while improving data transmission reliability and energy efficiency. Specifically, MADC employs the centralized training and decentralized execution (CTDE) learning paradigm to decouple the model’s training and inference tasks, which reduces the difficulty and computational cost of model learning and improves the training efficiency. In addition, a lightweight actor network based on multi-scale convolutional attention mechanism is designed in MADC, which can provide intelligent and real-time decision-making capabilities for resource-constrained nodes with low computational and storage complexities. Moreover, a scalable critic network utilizing multiple attention mechanisms is proposed. It is not only suitable for dynamic and changing network environments but also can more comprehensively and accurately evaluate local observation states, providing more accurate and efficient guidance for model optimization. Furthermore, MADC incorporates a double critic network architecture to mitigate potential overestimation issues during training, thereby ensuring the model’s robustness and reliability. Simulation results demonstrate that MADC outperforms existing RPL optimization algorithms in terms of energy efficiency, data transmission reliability, and adaptability.

Revitalizing Convolutional Network for Image Restoration.

Image restoration aims to reconstruct a high-quality image from its corrupted version, playing essential roles in many scenarios. Recent years have witnessed a paradigm shift in image restoration from convolutional neural networks (CNNs) to Transformer-based models due to their powerful ability to model long-range pixel interactions. In this paper, we explore the potential of CNNs for image restoration and show that the proposed simple convolutional network architecture, termed ConvIR, can perform on par with or better than the Transformer counterparts. By re-examing the characteristics of advanced image restoration algorithms, we discover several key factors leading to the performance improvement of restoration models. This motivates us to develop a novel network for image restoration based on cheap convolution operators. Comprehensive experiments demonstrate that our ConvIR delivers state-of-the-art performance with low computation complexity among 20 benchmark datasets on five representative image restoration tasks, including image dehazing, image motion/defocus deblurring, image deraining, and image desnowing.

MINN: Learning the Dynamics of Differential-Algebraic Equations and Application to Battery Modeling.

The concept of integrating physics-based and data-driven approaches has become popular for modeling sustainable energy systems. However, the existing literature mainly focuses on the data-driven surrogates generated to replace physics-based models. These models often trade accuracy for speed but lack the generalizability, adaptability, and interpretability inherent in physics-based models, which are often indispensable in modeling real-world dynamic systems for optimization and control purposes. We propose a novel machine learning architecture, termed model-integrated neural networks (MINN), that can learn the physics-based dynamics of general autonomous or non-autonomous systems consisting of partial differential-algebraic equations (PDAEs). The obtained architecture systematically solves an unsettled research problem in control-oriented modeling, i.e., how to obtain optimally simplified models that are physically insightful, numerically accurate, and computationally tractable simultaneously. We apply the proposed neural network architecture to model the electrochemical dynamics of lithium-ion batteries and show that MINN is extremely data-efficient to train while being sufficiently generalizable to previously unseen input data, owing to its underlying physical invariants. The MINN battery model has an accuracy comparable to the first principle-based model in predicting both the system outputs and any locally distributed electrochemical behaviors but achieves two orders of magnitude reduction in the solution time.

Insight mixed deep neural network architectures for molecular representation

Learning molecular representation is a crucial task in the field of drug discovery, particularly for various specific applications such as predicting molecular properties. Current methods are mainly based on deep neural network models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), graph neural networks (GNN) and their mixed models. However, these neural network models mentioned above do not provide detailed explanations for the ability to learn molecular representations and why such experimental results occurred. In this paper, we aim to compare the performance of these models in predicting molecular properties based on molecular representation, and give more insight into deep neural network architectures for specific molecular tasks. Our experimental results demonstrate that the graph neural network can obtain superior performance on the regression tasks, while the mixed deep neural network models show better performance on the classification tasks. Ablation study also gives more explanation and analysis to the experimental results.

Sharpness-Aware Lookahead for Accelerating Convergence and Improving Generalization.

Lookahead is a popular stochastic optimizer that can accelerate the training process of deep neural networks. However, the solutions found by Lookahead often generalize worse than those found by its base optimizers, such as SGD and Adam. To address this issue, we propose Sharpness-Aware Lookahead (SALA), a novel optimizer that aims to identify flat minima that generalize well. SALA divides the training process into two stages. In the first stage, the direction towards flat regions is determined by leveraging a quadratic approximation of the optimization trajectory, without incurring any extra computational overhead. In the second stage, however, it is determined by Sharpness-Aware Minimization (SAM), which is particularly effective in improving generalization at the terminal phase of training. In contrast to Lookahead, SALA retains the benefits of accelerated convergence while also enjoying superior generalization performance compared to the base optimizer. Theoretical analysis of the expected excess risk, as well as empirical results on canonical neural network architectures and datasets, demonstrate the advantages of SALA over Lookahead. It is noteworthy that with approximately 25% more computational overhead than the base optimizer, SALA can achieve the same generalization performance as SAM which requires twice the training budget of the base optimizer.