Network Architecture Research Articles

One‐dimensional deep learning inversion of marine controlled‐source electromagnetic data

AbstractThis paper explores the application of machine learning techniques, specifically deep learning, to the inverse problem of marine controlled‐source electromagnetic data. A novel approach is proposed that combines the convolutional neural network and recurrent neural network architectures to reconstruct layered electrical resistivity variation beneath the seafloor from marine controlled‐source electromagnetic data. The approach leverages the strengths of both convolutional neural network and recurrent neural network, where convolutional neural network is used for recognizing and classifying features in the data, and recurrent neural network is used to capture the contextual information in the sequential data. We have built a large synthetic dataset based on one‐dimensional forward modelling of a large number of resistivity models with different levels of electromagnetic structural complexity. The combined learning of convolutional neural network and recurrent neural network is used to construct the mapping relationship between the marine controlled‐source electromagnetic data and the resistivity model. The trained network is then used to predict the distribution of resistivity in the model by feeding it with marine controlled‐source electromagnetic responses. The accuracy of the proposed approach is examined using several synthetic scenarios and applied to a field dataset. We explore the sensitivity of deep learning inversion to different electromagnetic responses produced by resistive targets distributed at different depths and with varying levels of noise. Results from both numerical simulations and field data processing consistently demonstrate that deep learning inversions reliably reconstruct the subsurface resistivity structures. Moreover, the proposed method significantly improves the efficiency of electromagnetic inversion and offers significant performance advantages over traditional electromagnetic inversion methods.

A review of neural networks in handwritten character recognition

Handwritten character recognition has been a significant research focus in the fields of pattern recognition and artificial intelligence over the past decade. With the advent of neural networks, particularly deep learning models, the accuracy and efficiency of offline handwritten character recognition have dramatically improved. This paper presents a comprehensive review of recent developments in applying neural networks to handwritten character recognition. The literature review covers studies conducted between 2016 and 2023, providing insights into the methodologies, data processing techniques, and evaluation metrics used. The review spans various neural network architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and hybrid models. It categorizes and compares their performance across multiple benchmark datasets, highlighting specific improvements in recognition accuracy and efficiency. Furthermore, the review discusses the challenges faced in large-scale datasets, such as the diversity of handwriting styles and computational cost constraints. Notably, CNNs have shown outstanding performance, but the integration of advanced techniques like transfer learning and Generative Adversarial Networks (GANs) is explored as a potential avenue to enhance future recognition systems.

Multi-Modal Pose Representations for 6-DOF Object Tracking

Pose estimation methods for robotics should return a distribution of poses rather than just a single pose estimate. Motivated by this, in this work we investigate multi-modal pose representations for reliable 6-DoF object tracking. A neural network architecture for simultaneous object segmentation and estimation of fiducial points of the object on RGB images is proposed. Given a priori probability distribution of object poses a particle filter is employed to estimate the posterior probability distribution of object poses. An advanced observation model relying on matching the projected 3D model with the segmented object and a distance transform-based object representation is used to weight samples representing the probability distribution. Afterwards, the object pose determined by the PnP algorithm is included in the probability distribution via replacing a particle with the smallest weight. Next, a k-means++ algorithm is executed to determine modes in a multi-modal probability distribution. A multi-swarm particle swarm optimization is then executed to determine the finest modes in the probability distribution. A subset of particles for final pose optimization is found in a multi-criteria analysis using the TOPSIS algorithm. They are verified using conflicting criteria that are determined on the basis of object keypoints, segmented object, and the distance transform. On the challenging YCB-Video dataset it outperforms recent algorithms for both object pose estimation and object pose tracking.

One-shot entorhinal maps enable flexible navigation in novel environments.

Animals must navigate changing environments to find food, shelter or mates. In mammals, grid cells in the medial entorhinal cortex construct a neural spatial map of the external environment1-5. However, how grid cell firing patterns rapidly adapt to novel or changing environmental features on a timescale relevant to behaviour remains unknown. Here, by recording over 15,000 grid cells in mice navigating virtual environments, we tracked the real-time state of the grid cell network. This allowed us to observe and predict how altering environmental features influenced grid cell firing patterns on a nearly instantaneous timescale. We found evidence that visual landmarks provide inputs to fixed points in the grid cell network. This resulted in stable grid cell firing patterns in novel and altered environments after a single exposure. Fixed visual landmark inputs also influenced the grid cell network such that altering landmarks induced distortions in grid cell firing patterns. Such distortions could be predicted by a computational model with a fixed landmark to grid cell network architecture. Finally, a medial entorhinal cortex-dependent task revealed that although grid cell firing patterns are distorted by landmark changes, behaviour can adapt via a downstream region implementing behavioural timescale synaptic plasticity6. Overall, our findings reveal how the navigational system of the brain constructs spatial maps that balance rapidity and accuracy. Fixed connections between landmarks and grid cells enable the brain to quickly generate stable spatial maps, essential for navigation in novel or changing environments. Conversely, plasticity in regions downstream from grid cells allows the spatial maps of the brain to more accurately mirror the external spatial environment. More generally, these findings raise the possibility of a broader neural principle: by allocating fixed and plastic connectivity across different networks, the brain can solve problems requiring both rapidity and representational accuracy.

Forecasting ETF Performance: A Comparative Study of Deep Learning Models and the Fama-French Three-Factor Model

The global financial landscape has witnessed a significant shift towards Exchange-Traded Funds (ETFs), with their market capitalization surpassing USD 10 trillion in 2023, due to advantages such as low management fees, high liquidity, and broad market exposure. As ETFs become increasingly central to investment strategies, accurately forecasting their performance has become crucial. This study addresses this need by comparing the efficacy of deep learning models against the traditional Fama-French three-factor model in predicting daily ETF returns. The methodology employs eight artificial neural network architectures, including ANN, LSTM, GRU, CNN, and their variants, implemented in Python and applied to data ranging from 2010 to 2020, while also exploring the impact of additional factors on forecast accuracy. Empirical results reveal that LSTM and the Fama-French three-factor model exhibit a superior performance in ETF return prediction. This study contributes to the literature on financial forecasting and offers practical insights into investment decision making. By leveraging advanced artificial intelligence techniques, this study aims to enhance the toolkit available for ETF performance analysis, potentially improving investment strategies in this dynamic market segment.

Framework for Cyber Risk Loss Distribution of Client‐Server Networks: A Bond Percolation Model and Industry Specific Case Studies

ABSTRACTCyber risk has emerged as a significant threat to businesses that have increasingly relied on new and existing information technologies (IT). Across various businesses in different industries and sectors, a distinct pattern of IT network architectures, such as the client‐server network architecture, may, in principle, expose those businesses, which share it, to similar cyber risks. That is why in this article, we propose a probabilistic structural framework for loss assessments of cyber risks on the class of client‐server network architectures with different client types. To our knowledge, there exist no theoretical models of an aggregate loss distribution for cyber risk in this setting. With this structural framework via the exact mean and variance of losses, we demonstrate how the changing cybersecurity environment of a business's IT network impacts the loss distribution. Furthermore, our framework provides insights into better investment strategies for cybersecurity protection on the client‐server network. Motivated by cyberattacks across industries, we apply our framework to four case studies that utilize the client‐server network architecture. Our first application is implantable medical devices in healthcare. Our second application is the smart buildings domain. Third, we present an application for ride‐sharing services such as Uber and Lyft. The fourth is the application of vehicle‐to‐vehicle cooperation in traffic management. The results are corresponding exact means and variances of cyber risk loss distributions parameterized by various cybersecurity parameters allowing for liability assessments and decisions in cybersecurity protection investments.

Neural Network Algorithm for Training a Mobile Robot in the Task of Following a Target

The article is devoted to the research, development and implementation of an artificial neural network with reinforcement for controlling a mobile robot with a differential drive to solve the task of following a target. The authors of the study present a detailed description of the architecture of the neural network, its training using a deep deterministic gradient policy algorithm, and integration with the robot control system. A distributed neural network structure was selected for specialized generation of controls, for which the authors chose the angular and linear speed of the robot. A system of rules has been synthesized that takes into account changes in the distance between the robot and the target object and tracking of the heading angle in real time. A mathematical model of a robot with a differential drive is considered, on the basis of which a simulation program is implemented for intermediate training of a neural network, as a result of which the initial weighting coefficients are formed. Using this program, you can get by with less energy and time costs at the initial stage of the study. Experiments to test the effectiveness of the developed neural network were carried out in the Gazebo simulation environment using the ROS2 communication interface. The article describes the process of integrating a neural network with a robot control system in a simulation environment, as well as test results and analysis of the obtained data. The results of experiments are presented for three scenarios in which the initial position of the robot is the same, the remaining parameters are generated according to the rules described by the authors. The effectiveness of using such a solution for trajectory planning has been confirmed. The research contributes to the field of autonomous systems and demonstrates the potential of artificial reinforcement neural networks in robotics.

Anomaly Detection in Time Series Data

This paper presents a novel unsupervised approach for anomaly detection in marine diesel engines using a Transformer Neural Network based autoencoder (TAE) and residual analysis with Sequential Probability Ratio Test (SPRT) and Sum of Squares of Normalized Residuals (SSNR). This approach adeptly captures temporal dependencies within normal time-series data without the necessity for labeled failure data. To assess the performance of the proposed methodology, a dataset containing faulty data is generated under the same operational profile as the normal training data. The model undergoes training using normal data, after which the faulty data is reconstructed utilizing the trained model. Subsequently, SPRT and SSNR are used to analyze the residuals from the observed and reconstructed faulty data. Significant deviations surpassing a predefined threshold are identified as anomalous behavior. Additionally, this study explores various architectures of Transformer neural networks and other types of neural networks to conduct a comprehensive comparative analysis of the performance of the proposed approach. Insights and recommendations derived from the performance analysis are also presented, offering valuable information for potential users to leverage. The experimental results demonstrate that the proposed approach can accurately and efficiently detect anomalies in marine diesel engines. Therefore, this approach can be considered as a promising solution for early anomaly detection, leading to timely maintenance and repair, and preventing costly downtime. Also, avoid accidents with probability severe consequences.

Real-time robust liver and gallbladder segmentation during laparoscopic cholecystectomy using convolutional neural networks: an analysis

Aim: Images in different laparoscopic cholecystectomy datasets are acquired using various camera models, parameters, and settings, with the annotation methods varying by institution. These factors result in inconsistent inference performance of the network model. This study aims to identify the optimal network model architecture for liver and gallbladder segmentation from several options. Then, the performance and robustness of the optimal network model are evaluated using an independent dataset that is not included in the training. Methods: The public dataset, CholecSeg8k, was utilized as the input for the network model training, validation, and testing. A local private dataset from KPJ Damansara Hospital, Selangor, Malaysia, was used for testing purposes only. For the implementation of liver and gallbladder segmentation, segmentation models, a public Python library was employed. Results: Among the experiments, highly accurate liver and gallbladder segmentation results were achieved using the feature pyramid network (FPN) architecture as the network model, with the Inception-ResNet-v2 architecture as the network backbone. The best-trained network model resulted in a loss of 0.070955, a mean intersection over union (IoU) score of 0.95896, and a mean F1-score of 0.9773 on the test set. However, visualized results for the private dataset contained considerable false-negative areas. Conclusion: The proposed automated technique has the potential to serve as an alternative to the conventional indocyanine green injection along with near-infrared fluorescence imaging (ICG-NIRF)-based method for liver and gallbladder segmentation during laparoscopic cholecystectomy. Future work will focus on enhancing the results of the private dataset. Additionally, a surgeon-assistant robotic arm that will use the liver and gallbladder segmentation results for camera steering will be analyzed.

Spatiotemporal Modeling of Mitochondrial Network Architecture

In many cell types, mitochondria undergo extensive fusion and fission to form dynamic, responsive network structures that contribute to a number of homeostatic, metabolic, and signaling functions. The relationship between the dynamic interactions of individual mitochondrial units and the cell-scale network architecture remains an open area of study. In this work, we use coarse-grained simulations and approximate analytic models to establish how the network morphology is governed by local mechanical and kinetic parameters. The transition between fragmented structures and extensive networks is controlled by local fusion-to-fission ratios, network density, and geometric constraints. Similar fusion rate constants are found to account for the very different structures formed by mammalian networks (poised at the percolation transition) and well-connected budding yeast networks. Over a broad parameter range, the simulated network structures can be described by effective mean-field association constants that exhibit a nonlinear dependence on the microscopic nonequilibrium fusion, fission, and transport rates. Intermediate fusion rate constants are shown to result in the highest rates of network remodeling, with mammalian mitochondrial networks situated in a regime of high turnover. This spatially resolved modeling and simulation framework helps elucidate the emergence of cellular scale network structures, and allows for the quantitative extraction of microscopic kinetic parameters from past and future experimental data. Published by the American Physical Society 2024

Deep learning with photonic neural cellular automata.

Rapid advancements in deep learning over the past decade have fueled an insatiable demand for efficient and scalable hardware. Photonics offers a promising solution by leveraging the unique properties of light. However, conventional neural network architectures, which typically require dense programmable connections, pose several practical challenges for photonic realizations. To overcome these limitations, we propose and experimentally demonstrate Photonic Neural Cellular Automata (PNCA) for photonic deep learning with sparse connectivity. PNCA harnesses the speed and interconnectivity of photonics, as well as the self-organizing nature of cellular automata through local interactions to achieve robust, reliable, and efficient processing. We utilize linear light interference and parametric nonlinear optics for all-optical computations in a time-multiplexed photonic network to experimentally perform self-organized image classification. We demonstrate binary (two-class) classification of images using as few as 3 programmable photonic parameters, achieving high experimental accuracy with the ability to also recognize out-of-distribution data. The proposed PNCA approach can be adapted to a wide range of existing photonic hardware and provides a compelling alternative to conventional photonic neural networks by maximizing the advantages of light-based computing whilst mitigating their practical challenges. Our results showcase the potential of PNCA in advancing photonic deep learning and highlights a path for next-generation photonic computers.

Improving neural-network-based prediction models for misalignment in off-axis three-mirror anastigmat telescopes

The performance of a telescope system heavily relies on the precise alignment of the mirrors. The off-axis three-mirror anastigmat (TMA) telescope presents unique challenges due to its complex optical design. Each optical element within the off-axis TMA telescope is inherently introduced with theoretical eccentricity and tilt. Furthermore, the incorporation of freeform surfaces and other optical elements with intricate surface features typically leads to low initial alignment accuracy of the optical path. With this low initial alignment accuracy and the noise of the measurements, the prediction of the misalignment of the telescope is getting harder. A fully connected neural network architecture is proposed as a misalignment calculation method for an off-axis TMA telescope system with a freeform surface. Random training data is created by using optical design software. The sensitivity of the mirrors to the wavefront error is quantified and incorporated into the loss function of the neural network to improve prediction accuracy. Adding the noisy measurement samples to the training data creates a noise-immune neural network model. Simulation results show that our model can successfully predict misalignment of the mirrors with noisy measurement values.

Advancing Remote Sensing: A Unified Deep Learning Approach with Pretrained and Custom Architectures for High-Precision Classification

Abstract Advancement of remote sensing is vital for accurate land cover mapping and ecological surveillance. This research proposes a cutting-edge deep learning framework that integrates a meticulously designed customary network architecture with the effectiveness of pre-trained models, namely GoogleNet, VGG16, and InceptionV3. Our approach captures the complex features of RGB satellite images throughout various land cover categories via the diverse EuroSAT dataset. Our system achieves an impressive 99.40% test accuracy by implementing feature extraction through effective models with a customised deep learning network in an intuitive manner. High class-wise accuracies are observed, ranging from 96.00% to 100.00%, whereas F1-scores, precision, and recall all converge at 99.40%. The aforementioned results demonstrate our approach’s perspective to significantly improve analysis while also confirming its superiority. These results not only confirm the performance of our algorithm, but also show that it could be used to redesign satellite visual analysis behaviour. These results open the window to further improve and accurate remote sensing systems, giving a helpful viewpoint for researchers as well as professionals.

VisualTorch: Streamlining Visualization for PyTorch Neural Network Architectures

VisualTorch: Streamlining Visualization for PyTorch Neural Network Architectures

A flexible generative algorithm for growing in silico placentas.

The placenta is crucial for a successful pregnancy, facilitating oxygen exchange and nutrient transport between mother and fetus. Complications like fetal growth restriction and pre-eclampsia are linked to placental vascular structure abnormalities, highlighting the need for early detection of placental health issues. Computational modelling offers insights into how vascular architecture correlates with flow and oxygenation in both healthy and dysfunctional placentas. These models use synthetic networks to represent the multiscale feto-placental vasculature, but current methods lack direct control over key morphological parameters like branching angles, essential for predicting placental dysfunction. We introduce a novel generative algorithm for creating in silico placentas, allowing user-controlled customisation of feto-placental vasculatures, both as individual components (placental shape, chorionic vessels, placentone) and as a complete structure. The algorithm is physiologically underpinned, following branching laws (i.e. Murray's Law), and is defined by four key morphometric statistics: vessel diameter, vessel length, branching angle and asymmetry. Our algorithm produces structures consistent with in vivo measurements and ex vivo observations. Our sensitivity analysis highlights how vessel length variations and branching angles play a pivotal role in defining the architecture of the placental vascular network. Moreover, our approach is stochastic in nature, yielding vascular structures with different topological metrics when imposing the same input settings. Unlike previous volume-filling algorithms, our approach allows direct control over key morphological parameters, generating vascular structures that closely resemble real vascular densities and allowing for the investigation of the impact of morphological parameters on placental function in upcoming studies.

Automated Recognition of Snow-Covered and Icy Road Surfaces Based on T-Net of Mount Tianshan

The Tianshan Expressway plays a crucial role in China’s “Belt and Road” strategy, yet the extreme climate of the Tianshan Mountains poses significant traffic safety risks, hindering local economic development. Efficient detection of hazardous road surface conditions (RSCs) is vital to address these challenges. The complexity and variability of RSCs in the region, exacerbated by harsh weather, make traditional surveillance methods inadequate for real-time monitoring. To overcome these limitations, a vision-based artificial intelligence approach is urgently needed to ensure effective, real-time detection of dangerous RSCs in the Tianshan road network. This paper analyzes the primary structures and architectures of mainstream neural networks and explores their performance for RSC recognition through a comprehensive set of experiments, filling a research gap. Additionally, T-Net, specifically designed for the Tianshan Expressway engineering project, is built upon the optimal architecture identified in this study. Leveraging the split-transform-merge structure paradigm and asymmetric convolution, the model excels in capturing detailed information by learning features across multiple dimensions and perspectives. Furthermore, the integration of channel, spatial, and multi-head attention modules enhances the weighting of key features, making the T-Net particularly effective in recognizing the characteristics of snow-covered and icy road surfaces. All models presented in this paper were trained on a custom RSC dataset, compiled from various sources. Experimental results indicate that the T-Net outperforms fourteen once state-of-the-art (SOTA) models and three models specifically designed for RSC recognition, with 97.44% accuracy and 9.79% loss on the validation set.

Deep-Millet: A Deep Learning Model for Pearl Millet Disease Identification to Envisage Precision Agriculture

Abstract Plants are integral to human sustenance, serving as fundamental sources of sustenance, materials, and energy, crucial for economic prosperity. However, their productivity and yield are increasingly threatened by pests and diseases, exacerbated by shifting climatic conditions. Pearl millet, a vital crop in Africa and Asia, is particularly susceptible to a range of diseases including downy mildew, rust, ergot, smut, and blast, posing significant risks to crop yield and quality. Timely and accurate disease identification is paramount for effective management strategies. Traditional methods of disease detection relying on visual identification are laborious, costly, and often require specialized expertise, presenting formidable challenges for farmers. In this study, we propose a novel mobile application integrating a robust Deep Learning (DL) model for the automated identification of pearl millet leaf diseases, employing advanced computer vision techniques. A Convolutional Neural Network (CNN) architecture, named Deep Millet, was trained on a comprehensive dataset comprising 3441 field images depicting pearl millet leaves in both healthy and diseased states. It consists of fewer but more effective layers, which are optimized to extract the most pertinent features from the RGB images Comparative analysis against pre-trained models, including AlexNet, ResNet50, InceptionV3, Xception, NasNet mobile, VGG16, and VGG19, was conducted to evaluate the performance of the proposed model. Results demonstrate that Deep Millet achieved superior accuracy, completing training in a mere 240 seconds and yielding an impressive accuracy rating of 98.86%, surpassing current state-of-the-art models.

Smart PV Monitoring and Maintenance: A Vision Transformer Approach within Urban 4.0

The advancement to Urban 4.0 requires urban digitization and predictive maintenance of infrastructure to improve efficiency, durability, and quality of life. This study aims to integrate intelligent technologies for the predictive maintenance of photovoltaic panel systems, which serve as essential smart city renewable energy sources. In addition, we employ vision transformers (ViT), a deep learning architecture devoted to evolving image analysis, to detect anomalies in PV systems. The ViT model is pre-trained on ImageNet to exploit a comprehensive set of relevant visual features from the PV images and classify the input PV panel. Furthermore, the developed system was integrated into a web application that allows users to upload PV images, automatically detect anomalies, and provide detailed panel information, such as PV panel type, defect probability, and anomaly status. A comparative study using several convolutional neural network architectures (VGG, ResNet, and AlexNet) and the ViT transformer was conducted. Therefore, the adopted ViT model performs excellently in anomaly detection, where the ViT achieves an AUC of 0.96. Finally, the proposed approach excels at the prompt identification of potential defects detection, reducing maintenance costs, advancing equipment lifetime, and optimizing PV system implementation.

Investigating deep learning strategies for fast denoising of 5D cardiac photon-counting micro-CT images


Photon-counting detectors (PCDs) for CT imaging use energy thresholds to simultaneously acquire projections at multiple energies, making them suitable for spectral imaging and material decomposition. Unfortunately, setting multiple energy thresholds results in noisy analytical reconstructions due to low photon counts in high-energy bins. Iterative reconstruction provides high quality photon-counting CT (PCCT) images but requires enormous computation time for 5D (3D + energy + time) in vivo cardiac imaging. 

Approach. 
We recently introduced UnetU, a deep learning (DL) approach that accurately denoises axial slices from 4D (3D + energy) PCCT reconstructions at various acquisition settings. In this study, we explore UnetU configurations for 5D cardiac PCCT denoising, focusing on singular value decomposition (SVD) modifications along the energy and time dimensions and alternate network architectures such as 3D U-net, FastDVDNet, and Swin Transformer UNet. We compare our networks to multi-energy non-local means (ME NLM), an established PCCT denoising algorithm. 

Main results. 
Our evaluation, using real mouse data and the digital MOBY phantom, revealed that all DL methods were more than 16 times faster than iterative reconstruction. DL denoising with SVD along the energy dimension was most effective, consistently providing low root mean square error and spatio-temporal reduced reference entropic difference, alongside strong qualitative agreement with iterative reconstruction. This superiority was attributed to lower effective rank along the energy dimension than the time dimension in 5D cardiac PCCT reconstructions. ME NLM sometimes outperformed DL with time SVD or time and energy SVD, but lagged behind iterative reconstruction and DL with energy SVD. Among alternate DL architectures with energy SVD, none consistently outperformed UnetU Energy (2D). 

Significance. 
Our study establishes UnetU Energy as an accurate and efficient method for 5D cardiac PCCT denoising, offering a 32-fold speed increase from iterative reconstruction. This advancement sets a new benchmark for DL applications in cardiovascular imaging.

Enhanced analog switching and neuromorphic performance of ZnO-based memristors with indium tin oxide electrodes for high-accuracy pattern recognition.

This study systematically investigates analog switching and neuromorphic characteristics in a ZnO-based memristor by varying the anodic top electrode (TE) materials [indium tin oxide (ITO), Ti, and Ta]. Compared with the TE materials (Ti and Ta), memristive devices with TEs made of ITO exhibit dual volatile and nonvolatile switching behavior and multistate switching characteristics assessed based on reset-stop voltage and current compliance (ICC) responses. The polycrystalline structure of the ZnO functional layer sandwiched between ITO electrodes was confirmed by high-resolution transmission electron microscopy analysis. The current transport mechanism in the ZnO-based memristor was dominated by Schottky emission, with the Schottky barrier height modulated from 0.26 to 0.4V by varying the reset-stop voltage under different ICC conditions. The long-term potentiation and long-term depression synaptic characteristics were successfully mimicked by modulating the pulse amplitudes. Furthermore, a 90.84% accuracy was achieved using a convolutional neural network architecture for Modified National Institute of Standards and Technology pattern categorization, as demonstrated by the confusion matrix. The results demonstrated that the ITO/ZnO/ITO/Si memristor device holds promise for high-performance electronic applications and effective ITO electrode modeling.