Network Architecture Research Articles

Nested Neural Networks: A Novel Approach to Flexible and Deep Learning Architectures

It’s easy to get lost in the complexities of sentences like, “Never will I have been in a place whose experiences will have been teaching me how amazing life really is till the beginning of every one of those marvelous whole new advenchallenges for more than 20 years of life by the end of my military service.” These kinds of nested and layered statements are not just the domain of advanced grammar—they have a parallel in cutting-edge technology as well. This paper introduces a novel neural network architecture, termed Nested Neural Networks (NNN), that incorporates nested layers within a more complex overarching structure. Much like the layered conditions in Future Perfect and Future Perfect Progressive tenses, NNNs utilize ‘nested’ conditions to bring flexibility and depth to data processing. This structure captures complex relationships, achieving high performance, computational efficiency, and generalization across tasks, much as understanding the nested clauses of a sentence can reveal deeper meanings. Our experiments on benchmark datasets demonstrate NNNs’ ability to handle intricate data patterns with enhanced memory efficiency and training adaptability, often outperforming traditional models. This fascinating intersection between language and machine learning showcases the powerful potential of structured complexity, whether in communication or computation.

Identification of CD44 as a key engager to hyaluronic acid-rich extracellular matrices for cell traction force generation and tumor invasion in 3D

Mechanical properties of the extracellular matrix (ECM) critically regulate a number of important cell functions including growth, differentiation and migration. Type I collagen and glycosaminoglycans (GAGs) are two primary components of ECMs that contribute to mammalian tissue mechanics, with the collagen fiber network sustaining tension, and GAGs withstanding compression. The architecture and stiffness of the collagen network are known to be important for cell-ECM mechanical interactions via integrin cell surface adhesion receptors. In contrast, studies of GAGs in modulating cell-ECM interactions are limited. Here, we present experimental studies on the roles of hyaluronic acid (HA) in single tumor cell traction force generation using a recently developed 3D cell traction force microscopy method. Our work reveals that CD44, a cell surface receptor to HA, is engaged in cell traction force generation in conjunction with β1-integrin. We find that HA significantly modifies the architecture and mechanics of the collagen fiber network, decreasing tumor cells’ propensity to remodel the collagen network, attenuating traction force generation, transmission distance, and tumor invasion. Our findings point to a novel role for CD44 in traction force generation, which can be a potential therapeutic target for diseases involving HA rich ECMs such as breast cancer and glioblastoma.

Point cloud segmentation neural network with same-type point cloud assistance

This paper proposes neural network architectures for point cloud segmentation, which leverage prior knowledge derived from same-type point clouds. The approach involves concurrent processing of two point clouds: a target point cloud necessitating segmentation and a labeled same-type point cloud. The labeled point cloud provides preliminary labeling information, assisting in segmenting the target point cloud. A feature combination module is proposed to identify and combine corresponding features across the point clouds. The module augments the feature representation of the target cloud and improves its capacity for object discrimination. Experiments on the ShapeNetPart and S3DIS datasets demonstrate that when integrated into classical network architectures, the proposed approach can achieve improved segmentation performance over the corresponding networks, significantly in some of them.

A protocol for trustworthy EEG decoding with neural networks

Deep learning solutions have rapidly emerged for EEG decoding, achieving state-of-the-art performance on a variety of decoding tasks. Despite their high performance, existing solutions do not fully address the challenge posed by the introduction of many hyperparameters, defining data pre-processing, network architecture, network training, and data augmentation. Automatic hyperparameter search is rarely performed and limited to network-related hyperparameters. Moreover, pipelines are highly sensitive to performance fluctuations due to random initialization, hindering their reliability. Here, we design a comprehensive protocol for EEG decoding that explores the hyperparameters characterizing the entire pipeline and that includes multi-seed initialization for providing robust performance estimates. Our protocol is validated on 9 datasets about motor imagery, P300, SSVEP, including 204 participants and 26 recording sessions, and on different deep learning models. We accompany our protocol with extensive experiments on the main aspects influencing it, such as the number of participants used for hyperparameter search, the split into sequential simpler searches (multi-step search), the use of informed vs. non-informed search algorithms, and the number of random seeds for obtaining stable performance. The best protocol included 2-step hyperparameter search via an informed search algorithm, with the final training and evaluation performed using 10 random initializations. The optimal trade-off between performance and computational time was achieved by using a subset of 3-5 participants for hyperparameter search. Our protocol consistently outperformed baseline state-of-the-art pipelines, widely across datasets and models, and could represent a standard approach for neuroscientists for decoding EEG in a trustworthy and reliable way.

Channel reconstruction through improvised deep learning architecture for high-speed networks

Efficient acquisition of channel state information (CSI) is quite complicated process but immensely essential to exploit probable benefits of massive multiple input multiple output (MIMO) systems. Therefore, a deep learningbased model is utilized to estimate channel feedback in a massive MIMO system. The proposed improvised deep learning-based channel estimation (IDLCE) model enhances channel reconstruction efficiency by using multiple convolutional layers and residual blocks. The proposed IDLCE model utilizes encoder network to compress CSI matrices where decoder network is used to downlink reconstruct CSI matrices. Here, an additional quantization block is incorporated to improve feedback reconstruction accuracy by reducing channel errors. A COST 2,100 model is adopted to analyse performance efficiency for both indoor and outdoor scenarios. Further, deep learning-based model is used to train thousands of parameter and correlation coefficients much faster and to minimize computational complexity. The proposed IDLCE model evaluate performance in terms of normalized mean square error (NMSE), correlation efficiency and reconstruction accuracy and compared against varied state-of-art-channel estimation techniques. Excellent performance results are obtained with large improvement in channel reconstruction accuracy

Multivariate probabilistic forecasting of electricity prices with trading applications

This study extends recently introduced neural networks approach, based on a regularized distributional multilayer perceptron (DMLP) technique for a multivariate case electricity price forecasting. The performance of a fully connected architecture and a LSTM architecture of neural networks are tested. Different from previous studies we incorporate dependence between multiple exchanges (EPEX and Nord Pool). The empirical data application analyzes two auctions in the day-ahead electricity market for the United Kingdom market. Along with statistical evaluation of probabilistic forecasts we develop a flexible bidding strategy based on risk-adjusted investor utility function. The trading application leverages the differences of the two exchanges by having long/short positions in both. Our findings demonstrate while DMLP shows similar performance compared to the benchmarks, the algorithm is considerably less computationally costly. LASSO Quantile Regression is better in terms if statistical evaluation of distributional fit, while DMLP outperforms in terms of Sharpe ration (by 18%) in the trading application.

Deep Learning and Handcrafted Features for Thyroid Nodule Classification

ABSTRACTIn this research, we present a refined image‐based computer‐aided diagnosis (CAD) system for thyroid cancer detection using ultrasound imagery. This system integrates a specialized convolutional neural network (CNN) architecture designed to address the unique aspects of thyroid image datasets. Additionally, it incorporates a novel statistical model that utilizes a two‐dimensional random coefficient autoregressive (2D‐RCA) method to precisely analyze the textural characteristics of thyroid images, thereby capturing essential texture‐related information. The classification framework relies on a composite feature vector that combines deep learning features from the CNN and handcrafted features from the 2D‐RCA model, processed through a support vector machine (SVM) algorithm. Our evaluation methodology is structured in three phases: initial assessment of the 2D‐RCA features, analysis of the CNN‐derived features, and a final evaluation of their combined effect on classification performance. Comparative analyses with well‐known networks such as VGG16, VGG19, ResNet50, and AlexNet highlight the superior performance of our approach. The outcomes indicate a significant enhancement in diagnostic accuracy, achieving a classification accuracy of 97.2%, a sensitivity of 84.42%, and a specificity of 95.23%. These results not only demonstrate a notable advancement in the classification of thyroid nodules but also establish a new standard in the efficiency of CAD systems.

Broad Distributed Game Learning for intelligent classification in rolling bearing fault diagnosis

As a new Single Layer Feedforward Network (SLFN) architecture, Broad Learning System (BLS) has been widely used in the field of fault diagnosis because of its fast-training speed and high generalization capability. However, when features in different classes of signals are similar or weak, BLS generates a large number of redundant features that may be difficult to classify accurately. In view of this, a new Broad Distributed Game Learning (BDGL) method is proposed in this paper, which maps data into the game space by constructing two non-parallel game hyperplanes to achieve game and segmentation of different similar features, thereby making the data linearly differentiable in the game space. Meanwhile, a linear distribution constraint term is designed to reduce noise fitting and weak feature learning in training data learning by limiting the complexity of model parameters, thereby making the solution of the objective function simpler and faster. By comparing the Precision, Recall, F-score, Kappa and Accuracy of BDGL and the comparison methods on the two types of rolling bearing experimental data, the results show that BDGL has a high classification accuracy. In addition, the experimental results on small and noisy samples once again demonstrate the effectiveness of BDGL, which provides an efficient solution for rolling bearing fault diagnosis.

Leveraging Edge Computing and Deep Learning for the Real-Time Identification of Bean Plant Pathologies

Beans are essential crops globally, standing out as one of the most consumed and nourishing legumes, thereby playing a significant role in human nutrition and food security. Their cultivation faces several challenges, such as pests, diseases, unpredictable weather patterns, and soil erosion. Of these challenges, diseases are recognized as a key challenge, resulting in a decline in both yield quality and quantity, and inflicting substantial financial losses on farmers.This work proposes a deep learning-based approach for precise in-field identification of diseases in bean plants. We evaluate image classification and object detection models using state-of-the-art Convolutional Neural Network (CNN) architectures to identify Angular Leaf Spot (ALS) and bean rust diseases, key bean diseases in Uganda and the region in general, from smartphone images of bean leaves collected in various districts of Uganda.The dataset employed to train these models is the Makerere University beans image dataset, comprising 15,335 images categorized into three (3) classes (ALS, bean rust, and healthy). To improve in-field performance, the dataset was expanded to include an additional class (unknown class) consisting of a diverse collection of 2,800 images to account for images unrelated to the three (3) predefined classes. Adversarial training was further employed to enhance model robustness in identifying the target classes. In addition, two (2) Out-of-Distribution (ODD) detection techniques, i.e., confidence thresholding and training with an auxiliary class (unknown class), were utilized to handle inputs unrelated to bean leaves.Our results show that our custom CNN, BeanWatchNet, achieved 90% accuracy when tested on unseen data for the classification of the three (3) target classes, i.e., ALS, bean rust and healthy. EfficientNet v2 B0 and BeanWatchNet demonstrated superior performance for the four-class (with unknown class) image classification task, achieving 91% and 90% accuracy, respectively, when evaluated on the test dataset. YOLO v8 exhibited superior performance for the object detection models, attaining mAP@50 of 87.6. The custom CNN model and YOLO v8 model were quantized and deployed across two (2) edge platforms: a smartphone (through a mobile application) and a Raspberry Pi 4B to facilitate in-field disease detection. The benchmarking code and best models are publicly available on GitHub.1

A novel Taguchi-based approach for optimizing neural network architectures: application to elastic short fiber composites

This study presents an innovative application of the Taguchi design of experiment method to optimize the structure of an Artificial Neural Network (ANN) model for the prediction of elastic properties of short fiber reinforced composites. The main goal is to minimize the computational effort required for hyperparameter optimization while enhancing the prediction accuracy. By utilizing a robust experimental design framework, the structure of an ANN model is optimized. This approach involves identifying a combination of hyperparameters that provides optimal predictive accuracy with the fewest algorithmic runs, thereby significantly reducing the required computational effort. The results suggested that the Taguchi-based developed ANN model with three hidden layers, 20 neurons in each hidden layer, elu activation function, Adam optimizer, and a learning rate of 0.001 can predict the anisotropic elastic properties of short fiber reinforced composites with a prediction accuracy of 97.71%. Then, external validation of the proposed ANN model was done using experimental data, and differences of less than 10% were obtained, indicating an appropriate predictive performance of the proposed ANN algorithm. Our findings demonstrate that the Taguchi method not only streamlines the hyperparameter tuning process but also substantially improves the algorithm's performance. These results highlight the potential of the Taguchi method as a powerful tool for optimizing machine learning algorithms, especially in scenarios where computational resources are limited. The implications of this study are far-reaching, offering insights for future research in the optimization of different algorithms for improved accuracies and computational efficiencies.

A novel automated neural network architecture search method of air target intent recognition

Modern air battlefield operations are characterized by flexibility and change, and the battlefield evolves rapidly and intricately. However, traditional air target intent recognition methods, which mainly rely on manually designed neural network models, find it difficult to maintain sustained and excellent performance in such a complex and changing environment. To address the problem of the adaptability of neural network models in complex environments, we propose a lightweight Transformer model (TransATIR) with a strong adaptive adjustment capability, based on the characteristics of air target intent recognition and the neural network architecture search technique. After conducting extensive experiments, it has been proved that TransATIR can efficiently extract the deep feature information from battlefield situation data by utilizing the neural architecture search algorithm, in order to quickly and accurately identify the real intention of the target. The experimental results indicate that TransATIR significantly improves recognition accuracy compared to the existing state-of-the-art methods, and also effectively reduces the computational complexity of the model.

From One-dimensional to Multidimensional Map Neural Networks

In the contemporary landscape, artificial neural networks transcend their traditional role of function approximation and have found diverse applications in fields such as image classification, machine translation, speech recognition, and natural language processing. In some datasets, traditional architectures exhibit low test and training accuracy, coupled with high loss and prolonged training times. This study aims to introduce innovative neural network architectures that outperform conventional models. The research presents a novel framework integrating one-dimensional and multidimensional map neural networks, consisting of three distinct architectures: 1D-Map, 2D-Map, and 3D-Map. A systematic performance comparison with traditional models is conducted following the implementation of these architectures. The evaluation spans four datasets, encompassing the domains of heat treatment of electroless Ni-P nano coatings, letter recognition, combined cycle power plants, and Seoul bike sharing demand. In the heat treatment dataset, the proposed 3D-Map architecture reached 0.055 higher average test accuracy than traditional MLP architecture. In the letter recognition dataset, the test accuracy of 3D-Map architecture was 0.0523 higher than the test accuracy of the LSTM architecture. In the combined cycle power plant dataset, the test accuracy of the 3D-Map architecture was 0.0612 more than the test accuracy of the MLP architecture. In the Seoul bike-sharing demand dataset, the test accuracy of the 2D-Map architecture was 0.0696 higher than the test accuracy of the LSTM architecture. The study's findings underscore the consistently superior performance of the proposed architectures compared to their traditional counterparts.

Probabilistic oil price forecasting with a variational mode decomposition-gated recurrent unit model incorporating pinball loss

Prediction methods have garnered significant attention in intelligent decision-making. Most existing approaches to predicting crude oil prices prioritize accuracy and stability while providing precise prediction intervals (PIs) that can offer valuable insights. Thus far, we introduce a novel hybrid model to forecast future crude oil prices. Our approach leverages the variational mode decomposition (VMD) to simplify the complexity of the original time series, yielding a set of subseries. These subseries are then modeled using a deep neural network architecture called a gated recurrent unit (GRU). To address the prediction uncertainty, we employed the pinball loss function rather than the mean square error (MSE) to guide the proposed VMD-GRU. This adaptation extends the traditional GRU-based point forecasting to probabilistic forecasting by estimating quantiles. We evaluated our proposed model on a well-established crude oil price series by conducting both single- and multi-step-ahead forecasting analyses. Our findings underscore the efficacy of the combined model, demonstrating its superior predictive performance compared to benchmark models.

A Novel Fully-Connected Beamforming Network Architecture With the QR Factorization Synthesis Method

A Novel Fully-Connected Beamforming Network Architecture With the QR Factorization Synthesis Method

2S-SGCN: A two-stage stratified graph convolutional network model for facial landmark detection on 3D data

Facial Landmark Detection (FLD) algorithms play a crucial role in numerous computer vision applications, particularly in tasks such as face recognition, head pose estimation, and facial expression analysis. While FLD on images has long been the focus, the emergence of 3D data has led to a surge of interest in FLD on it due to its potential applications in various fields, including medical research. However, automating FLD in this context presents significant challenges, such as selecting suitable network architectures, refining outputs for precise landmark localization and optimizing computational efficiency. In response, this paper presents a novel approach, the 2-Stage Stratified Graph Convolutional Network (2S-SGCN), which addresses these challenges comprehensively. The first stage aims to detect landmark regions using heatmap regression, which leverages both local and long-range dependencies through a stratified approach. In the second stage, 3D landmarks are precisely determined using a new post-processing technique, namely MSE-over-mesh. 2S-SGCN ensures both efficiency and suitability for resource-constrained devices. Experimental results on 3D scans from the public Facescape and Headspace datasets, as well as on point clouds derived from FLAME meshes collected in the DAD-3DHeads dataset, demonstrate that the proposed method achieves state-of-the-art performance across various conditions. Source code is accessible at https://github.com/gfacchi-dev/CVIU-2S-SGCN.

Multiclass Classification of Tomato Leaf Diseases Using Convolutional Neural Networks and Transfer Learning

ABSTRACTTomato (biological name: Solanum lycopersicum) is an important food crop worldwide. However, due to climatic changes and various diseases, the yield of tomatoes decreased significantly, being detrimental from an economic point of view. Various diseases infect the tomato leaves, such as bacterial and septorial leaf spots, early blight and mosaic virus, to name a few. If uncared, these tomato leaf diseases (TLDs) can spread to other leaves and the fruit. Hence it is vital to detect these diseases as early as possible. Leaf examination is one of the standard techniques to identify and control the spread of diseases. Big Data has made substantial progress, and with the help of computer vision and deep learning techniques to analyse data, we can identify the diseased leaves and help control the disease's spread further. This study used three lightweight midgeneration convolutional neural networks (CNNs) classification network architectures which has the scope to be deployed in IoT devices to help the agricultural community tackle TLDs. It also shows the efficacy of the models with and without geometric data augmentation. The model was trained on a Kaggle data set containing a more significant number of samples to make a robust model aware of broader data distribution and validated on the Plant Village dataset to test its efficacy. The results show that applying transfer learning using ImageNet weights to the MobileNet Architecture using geometrically augmented sample images yields a train and test accuracy of 99.71% and 99.49%, respectively.

In silico and functional analysis identifies key gene networks and novel gene candidates in obesity-linked human visceral fat.

Visceral adiposity is associated with increased proinflammatory activity, insulin resistance, diabetes risk, and mortality rate. Numerous individual genes have been associated with obesity, but studies investigating gene regulatory networks in human visceral obesity have been lacking. We analyzed gene regulatory networks in human visceral adipose tissue (VAT) from 48 and 11 Chinese patients with and without obesity, respectively, using gene coexpression and gene regulatory network construction from RNA-sequencing data. We also conducted RNA interference-based functional tests on selected genes for effects on adipocyte differentiation. A scale-free gene coexpression network was constructed from 360 differentially expressed genes between VAT samples from patients with and without obesity (absolute log fold change > 1, false discovery rate [FDR] < 0.05), with edge probability > 0.8. Gene regulatory network analysis identified candidate transcription factors associated with differentially expressed genes. A total of 15 subnetworks (communities) displayed altered connectivity patterns between obesity and nonobesity networks. Genes in proinflammatory pathways showed increased network connectivity in VAT samples with obesity, whereas the oxidative phosphorylation pathway displayed reduced connectivity (enrichment FDR < 0.05). Functional screening via RNA interference identified genes such as SOX30, SIRPB1, and OSBPL3 as potential network-derived candidates influencing adipocyte differentiation. This approach highlights the network architecture in human obesity, identifies novel candidate genes, and generates new hypotheses regarding network-assisted gene regulation in VAT.

Robust and energy-efficient RPL optimization algorithm with scalable deep reinforcement learning for IIoT

The increasing complexity and quantity of the Industrial Internet of Things (IIoT) pose new challenges to the traditional routing protocol for low-power and lossy networks (RPL) in terms of dynamic management, data transmission reliability, and energy efficiency optimization. This paper proposes a scalable deep reinforcement learning (DRL) algorithm with a multi-attention actor double critic model for routing optimization (MADC) to meet the requirements of IIoT for efficient and intelligent routing decisions while improving data transmission reliability and energy efficiency. Specifically, MADC employs the centralized training and decentralized execution (CTDE) learning paradigm to decouple the model’s training and inference tasks, which reduces the difficulty and computational cost of model learning and improves the training efficiency. In addition, a lightweight actor network based on multi-scale convolutional attention mechanism is designed in MADC, which can provide intelligent and real-time decision-making capabilities for resource-constrained nodes with low computational and storage complexities. Moreover, a scalable critic network utilizing multiple attention mechanisms is proposed. It is not only suitable for dynamic and changing network environments but also can more comprehensively and accurately evaluate local observation states, providing more accurate and efficient guidance for model optimization. Furthermore, MADC incorporates a double critic network architecture to mitigate potential overestimation issues during training, thereby ensuring the model’s robustness and reliability. Simulation results demonstrate that MADC outperforms existing RPL optimization algorithms in terms of energy efficiency, data transmission reliability, and adaptability.

Interactions of localized wave and dynamics analysis in the new generalized stochastic fractional potential-KdV equation.

In this paper, we investigate the new generalized stochastic fractional potential-Korteweg-de Vries equation, which describes nonlinear optical solitons and photon propagation in circuits and multicomponent plasmas. Inspired by Kolmogorov-Arnold network and our earlier work, we enhance the improved bilinear neural network method by using a large number of activation functions instead of neurons. This method incorporates the concept of simulating more complicated activation functions with fewer parameters, with more diverse activation functions to generate more complex and rare analytical solutions. On this basis, constraints are introduced into the method, reducing a significant amount of computational workload. We also construct neural network architectures, such as "2-3-1," "2-2-3-1," "2-3-3-1," and "2-3-2-1" using this method. Maple software is employed to obtain many exact analytical solutions by selecting appropriate parameters, such as the superposition of double-period lump solutions, lump-rogue wave solutions, and three interaction solutions. The results show that these solutions exhibit more complex waveforms than those obtained by conventional methods, which is of great significance for the electrical systems and multicomponent fluids to which the equation is applied. This novel method shows significant advantages when applied to fractional-order equations and is expected to be increasingly widely used in the study of nonlinear partial differential equations.

Beyond breathalyzers: AI-powered speech analysis for alcohol intoxication detection

Detecting potential alcohol inebriation or intoxication status holds paramount significance for social prevention and security. Beyond its association with long-term health effects, alcohol consumption can lead to immediate consequences, including reduced control over one’s actions, with traffic fatalities representing one of the most tragic outcomes.This study leveraged the Alcohol Language corpus, involving 162 subjects recorded both in sober and inebriated states. Participants provided 60 speech samples while sober and 30 when intoxicated, all within a realistic car setting using head-mounted microphones. Our research endeavors encompassed comprehensive stratified statistical tests to examine the impact of alcohol consumption on speech production while uncovering the influence of covariates such as age, gender, and drinking habits.Additionally, we introduced a speaker-neutral machine learning algorithm, based on the Domain-Adversarial Neural Network architecture. This approach aimed to overcome challenges posed by individual differences that often complicate intoxicated speech analysis. Notably, our findings highlighted the effectiveness of features like the RASTA-filtered auditory spectrum. Nevertheless, the results from statistical tests emphasized the need for techniques that minimize inter-subject variability.As for the automatic classification, the proposed architecture exhibited promising results, yielding a classification accuracy slightly exceeding 70% on an independent test set. Although preliminary, our research demonstrates the potential for detecting alcohol-induced speech changes, benefiting societal well-being and security. It also underscores the importance of developing strategies that account for individual differences while harnessing the power of automatic models to effectively distinguish between sober and intoxicated individuals.