Deep Artificial Neural Networks Research Articles

Data-driven nonlinear parametric model order reduction framework using deep hierarchical variational autoencoder

A data-driven parametric model order reduction (MOR) method using a deep artificial neural network is proposed. The present network, a least-squares hierarchical variational autoencoder (LSH-VAE), is capable of performing nonlinear MOR for the parametric interpolation of a nonlinear dynamic system with a significant number of degrees of freedom. LSH-VAE differs from existing networks in two major respects: a deep hierarchical structure and a hybrid weighted, probabilistic loss function. The enhancements result in significantly improved accuracy and stability compared with conventional nonlinear MOR methods, autoencoders, and variational autoencoders. In LSH-VAE, the parametric MOR framework is based on the spherically linear interpolation of the latent manifold. The present framework is validated and evaluated on three nonlinear and multiphysics dynamic systems. First, the present framework is evaluated on the fluid–structure interaction benchmark problem to assess its efficiency and accuracy. Then, a highly nonlinear aeroelastic phenomenon, limit cycle oscillation, is analyzed. Finally, the framework is applied to three-dimensional fluid flow to demonstrate its capability to efficiently analyze a significantly large number of degrees of freedom. The superior performance of LSH-VAE is emphasized by comparing its results against those of widely used nonlinear MOR methods, a convolutional autoencoder, and β\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\beta $$\\end{document}-VAE. The proposed framework exhibits significantly enhanced accuracy compared with that of conventional methods, while the computational efficiency continues to remain significantly higher.

Spontaneous emergence of rudimentary music detectors in deep neural networks

Music exists in almost every society, has universal acoustic features, and is processed by distinct neural circuits in humans even with no experience of musical training. However, it remains unclear how these innate characteristics emerge and what functions they serve. Here, using an artificial deep neural network that models the auditory information processing of the brain, we show that units tuned to music can spontaneously emerge by learning natural sound detection, even without learning music. The music-selective units encoded the temporal structure of music in multiple timescales, following the population-level response characteristics observed in the brain. We found that the process of generalization is critical for the emergence of music-selectivity and that music-selectivity can work as a functional basis for the generalization of natural sound, thereby elucidating its origin. These findings suggest that evolutionary adaptation to process natural sounds can provide an initial blueprint for our sense of music.

Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks.

Developmental plasticity plays a prominent role in shaping the brain's structure during ongoing learning in response to dynamically changing environments. However, the existing network compression methods for deep artificial neural networks (ANNs) and spiking neural networks (SNNs) draw little inspiration from brain's developmental plasticity mechanisms, thus limiting their ability to learn efficiently, rapidly, and accurately. This paper proposed a developmental plasticity-inspired adaptive pruning (DPAP) method, with inspiration from the adaptive developmental pruning of dendritic spines, synapses, and neurons according to the "use it or lose it, gradually decay" principle. The proposed DPAP model considers multiple biologically realistic mechanisms (such as dendritic spine dynamic plasticity, activitydependent neural spiking trace, and local synaptic plasticity), with additional adaptive pruning strategy, so that the network structure can be dynamically optimized during learning without any pre-training and retraining. Extensive comparative experiments show consistent and remarkable performance and speed boost with the extremely compressed networks on a diverse set of benchmark tasks for deep ANNs and SNNs, especially the spatio-temporal joint pruning of SNNs in neuromorphic datasets. This work explores how developmental plasticity enables complex deep networks to gradually evolve into brain-like efficient and compact structures, eventually achieving state-of-the-art (SOTA) performance for biologically realistic SNNs.

ПРОГНОЗИРОВАНИЕ ЭКОНОМИКИ НАУЧНОГО СЕКТОРА НА МОДЕЛЯХ ДИСКРЕТНОГО ВЫБОРА С ИСПОЛЬЗОВАНИЕМ ИСКУССТВЕННЫХ НЕЙРОННЫХ СЕТЕЙ ПРИ МЕЖСТРАНОВОМ АНАЛИЗЕ

The article presents a study of the influence of artificial deep neural networks (AGNN) on increasing the accuracy and reliability of predictive regression models in the economics of the scientific sector of 61 countries, including Russia. We integrated the feedforward IGNS into the econometric discrete choice model (DCM), which represents a multinomial logit model (MNLR). AGNN in regression models built on statistical data with a large range of standard deviation lead to an increase in forecasting accuracy by at least 32% within 0,8 periods of the analyzed data. The use of integration of AGNN with DCM is limited only to heterogeneous data. The structures under consideration can effectively solve econometric problems and problems of managing organizational systems.

An adaptive image compression algorithm based on joint clustering algorithm and deep learning

AbstractIn recent years, deep artificial neural networks have attracted much attention and have been applied in various fields because they surpass the parameter fitting effect of traditional methods under the condition of data convergence. On the other hand, limited transmission bandwidth and storage capacity make image compression necessary in communication. Here, a compression algorithm that combines the K‐means clustering algorithm with the neural network algorithm is proposed. First, the pixel points of the image are clustered by K‐means algorithm in order to reduce the amount of data input to the neural network algorithm. Secondly, neural network is used to extract image features which realizes further compression. The experiment results show that the peak signal‐to‐noise ratio (PSNR) is 33.48 dB at most with compression ratio at 32:1. The ablation experiment shows that the run time speeds up 9.5% compared to the algorithm without K‐means clustering. Comprehensive comparison experiment shows that the average PSNR is 30.09 dB, which is larger than other baseline approaches. The proposed algorithm is an efficient solution for image compression.

Use of an unmanned aerial‐aquatic vehicle for acoustic sensing in freshwater ecosystems

AbstractFreshwater ecosystems are endangered, underfunded and understudied, making new methods such as passive acoustic monitoring (PAM) essential for improving the efficiency and effectiveness of data collection. However, many challenges are still to be addressed with PAM: difficulty in accessing research sites, the logistics of implementing large‐scale studies and the invasiveness of data collection. When combined with PAM and other sensing strategies, mobile robotics are a promising solution to directly address these challenges. In this paper, we integrate water surface and underwater acoustic monitoring equipment onto a prototype unmanned aerial‐aquatic vehicle (UAAV) capable of sailing and flight (SailMAV). Twelve autonomous sailing missions were run on Lake Vrana, Croatia, during which acoustic data were collected, and the ability of the UAAV to facilitate the collection of acoustic data demonstrated. Data were simultaneously collected using standard recording methods on buoys and banksides to provide a comparative approach. Acoustic indices were used to analyse the soundscape of underwater acoustic data and BirdNET (a deep artificial neural network) was used on water surface datasets to determine bird species composition. Results show higher species richness and call abundance from UAAV surveys and high site dissimilarity owing to turnover between stationary and UAAV methods. This highlights the success of the UAAV in detecting biodiversity and the complementarity of these methods in providing a broad picture of the biodiversity of freshwater ecosystems. Increased bird diversity and underwater acoustic activity in protected areas demonstrate the benefits of protecting freshwater ecosystems; however, site dissimilarity driven by turnover highlights the importance of protecting the entire ecosystem. We show how, by integrating PAM and a UAAV, we can overcome some of the current challenges in freshwater biodiversity monitoring, improving accessibility, increasing spatial scale and coverage, and reducing invasiveness.

Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions.

Models that predict brain responses to stimuli provide one measure of understanding of a sensory system and have many potential applications in science and engineering. Deep artificial neural networks have emerged as the leading such predictive models of the visual system but are less explored in audition. Prior work provided examples of audio-trained neural networks that produced good predictions of auditory cortical fMRI responses and exhibited correspondence between model stages and brain regions, but left it unclear whether these results generalize to other neural network models and, thus, how to further improve models in this domain. We evaluated model-brain correspondence for publicly available audio neural network models along with in-house models trained on 4 different tasks. Most tested models outpredicted standard spectromporal filter-bank models of auditory cortex and exhibited systematic model-brain correspondence: Middle stages best predicted primary auditory cortex, while deep stages best predicted non-primary cortex. However, some state-of-the-art models produced substantially worse brain predictions. Models trained to recognize speech in background noise produced better brain predictions than models trained to recognize speech in quiet, potentially because hearing in noise imposes constraints on biological auditory representations. The training task influenced the prediction quality for specific cortical tuning properties, with best overall predictions resulting from models trained on multiple tasks. The results generally support the promise of deep neural networks as models of audition, though they also indicate that current models do not explain auditory cortical responses in their entirety.

Selection of an algorithm for classifying data quoted on the Day Ahead Market of TGE S.A. in MATLAB and Simulink using Deep Learning Toolbox

The article contains an analysis leading to the selection of an algorithm for classifying data listed on the Day-Ahead Market of TGE S.A. in MATLAB and Simulink using Deep Learning Toolbox. In this regard, an introduction to deep learning methods, classification methods, and classification algorithms is provided first. Particular attention was paid to the essence of three important deep learningmethods in the classification, i.e. the methods called: Stochastic Gradient Descent Momentum, Root Mean Square Prop and Adaptive Moment Estimation. Then, three architectures of artificial neural networks used in deep learning were characterized, i.e.: Deep Belief Network, Convolutional Neural Network and Recurrent Neural Network. Attention was paid to the selection parameters of algorithms for learning deep artificial neural networks that can be used in classification, such as: accuracy, information losses and learning time. Practical aspects of research experiments were also shown, including selected results of research conducted on volume and fixing 1 data quoted on the TGE S.A. Day-Ahead Market. After analyzing the obtained test results for the hourly system, it was noted that the least suitable algorithm for classification purposes was the Stochastic Gradient Descent Momentum algorithm, which in each case had worse results than the other two algorithms, i.e. the Adaptive Moment Estimation algorithm and the Root Mean algorithm Square Prop. However, the best algorithm turned out to be the Adaptive Moment Estimation algorithm, which obtained the highest accuracy, which was at a level comparable to the Root Mean Square Prop algorithm, with the latter algorithm having larger losses.

Application of transfer learning to predict drug-induced human in vivo gene expression changes using rat in vitro and in vivo data.

The liver is the primary site for the metabolism and detoxification of many compounds, including pharmaceuticals. Consequently, it is also the primary location for many adverse reactions. As the liver is not readily accessible for sampling in humans; rodent or cell line models are often used to evaluate potential toxic effects of a novel compound or candidate drug. However, relating the results of animal and in vitro studies to relevant clinical outcomes for the human in vivo situation still proves challenging. In this study, we incorporate principles of transfer learning within a deep artificial neural network allowing us to leverage the relative abundance of rat in vitro and in vivo exposure data from the Open TG-GATEs data set to train a model to predict the expected pattern of human in vivo gene expression following an exposure given measured human in vitro gene expression. We show that domain adaptation has been successfully achieved, with the rat and human in vitro data no longer being separable in the common latent space generated by the network. The network produces physiologically plausible predictions of human in vivo gene expression pattern following an exposure to a previously unseen compound. Moreover, we show the integration of the human in vitro data in the training of the domain adaptation network significantly improves the temporal accuracy of the predicted rat in vivo gene expression pattern following an exposure to a previously unseen compound. In this way, we demonstrate the improvements in prediction accuracy that can be achieved by combining data from distinct domains.

Modified state activation functions of deep learning-based SC-FDMA channel equalization system

The most important function of the deep learning (DL) channel equalization and symbol detection systems is the ability to predict the user’s original transmitted data. Generally, the behavior and performance of the deep artificial neural networks (DANNs) rely on three main aspects: the network structure, the learning algorithms, and the activation functions (AFs) used in each node in the network. Long short-term memory (LSTM) recurrent neural networks have shown some success in channel equalization and symbol detection. The AFs used in the DANN play a significant role in how the learning algorithms converge. Our article shows how modifying the AFs used in the tanh units (block input and output) of the LSTM units can significantly boost the DL equalizer's performance. Additionally, the learning process of the DL model was optimized with the help of two distinct error-measuring functions: default (cross-entropy) and sum of squared error (SSE). The DL model's performance with different AFs is compared. This comparison is conducted using three distinct learning algorithms: Adam, RMSProp, and SGdm. The findings clearly demonstrate that the most frequently used AFs (sigmoid and hyperbolic tangent functions) do not really make a significant contribution to perfect network behaviors in channel equalization. On the other hand, there are a lot of non-common AFs that can outperform the frequently employed ones. Furthermore, the outcomes demonstrate that the recommended loss functions (SSE) exhibit superior performance in addressing the channel equalization challenge compared to the default loss functions (cross-entropy).

Improved deep artificial neural network-powered prediction of extreme mechanical performances of fractal architectures with high hierarchical rank

This work explores the mechanical performances, including anisotropy, wave propagation, and buckling resistance capability of the Menger sponge and Jerusalem fractal structures, with the high rank, using an improved deep artificial neural network (ANN) model inserted by a loop with multiple conditions. Where, the effective mechanical attributes, including Young’s modulus, shear modulus, and Poisson’s ratio of the Menger sponge structure with rank 4 are attained using high-performance computer cluster (HPCC) that are then used to train and also validate the ANN model. The accuracy of the deep ANN model trained by data of fractal structures with and without rank 4 is verified by the numerical model and experiment with the good agreement found or the maximum percent difference being 15.6% where the numerical and experimental data don’t belong to its training samples. Remarkably, the deep ANN model demonstrates an extremely low cost, but very fast compared with the numerical and experimental works. Namely, the improved deep ANN model implemented by a regular computer can accurately predict the mechanical attributes of Menger sponge fractal structures with full ranks in 3 min. While it takes a week for numerical work using HPCC or experiment method to produce the same mechanical properties of the Menger sponge with only rank 4, displaying at least thousands of times slower than the ANN model. The ANN results indicate that the onset of anisotropy of considered fractal structures is from rank 1, and the reversal of anisotropy will take place upon an increase in the rank while such reversal is not seen for the case of lattice structure. The anisotropy reversal could be one of main reasons why the fractal structure tends to have a high rank in nature. In addition, the buckling stress and phase wave velocity are observed to decrease with rising rank, with a nonlinear pattern.

Deep photonic reservoir computer based on frequency multiplexing with fully analog connection between layers

Reservoir computers (RCs) are randomized recurrent neural networks well adapted to process time series, performing tasks such as nonlinear distortion compensation or prediction of chaotic dynamics. Deep reservoir computers (deep-RCs), in which the output of one reservoir is used as the input for another one, can lead to improved performance because, as in other deep artificial neural networks, the successive layers represent the data in more and more abstract ways. We present a fiber-based photonic implementation of a two-layer deep-RC based on frequency multiplexing. The two RC layers are encoded in two frequency combs propagating in the same experimental setup. The connection between the layers is fully analog and does not require any digital processing. We find that the deep-RC outperforms a traditional RC by up to two orders of magnitude on two benchmark tasks. This work paves the way towards using fully analog photonic neuromorphic computing for complex processing of time series, while avoiding costly analog-to-digital and digital-to-analog conversions.

Spiking neural networks fine-tuning for brain image segmentation.

The field of machine learning has undergone a significant transformation with the progress of deep artificial neural networks (ANNs) and the growing accessibility of annotated data. ANNs usually require substantial power and memory usage to achieve optimal performance. Spiking neural networks (SNNs) have recently emerged as a low-power alternative to ANNs due to their sparsity nature. Despite their energy efficiency, SNNs are generally more difficult to be trained than ANNs. In this study, we propose a novel three-stage SNN training scheme designed specifically for segmenting human hippocampi from magnetic resonance images. Our training pipeline starts with optimizing an ANN to its maximum capacity, then employs a quick ANN-SNN conversion to initialize the corresponding spiking network. This is followed by spike-based backpropagation to fine-tune the converted SNN. In order to understand the reason behind performance decline in the converted SNNs, we conduct a set of experiments to investigate the output scaling issue. Furthermore, we explore the impact of binary and ternary representations in SNN networks and conduct an empirical evaluation of their performance through image classification and segmentation tasks. By employing our hybrid training scheme, we observe significant advantages over both ANN-SNN conversion and direct SNN training solutions in terms of segmentation accuracy and training efficiency. Experimental results demonstrate the effectiveness of our model in achieving our design goals.

Application of Artificial Neural Networks for Identification of Lithofacies by Processing of Core Drilling Data

Identifying lithofacies types from core drilling data presents significant challenges, especially given the limited number of physical drilling characteristics available for analysis. Traditional machine learning methods often face issues with poor training and testing due to these limitations. Addressing this, we propose a new method for processing core drilling data to improve the accuracy of deep artificial neural networks (DANNs) in lithofacies recognition. Our approach transforms torque, weight on bit (WOB), and rotational speed data into three square matrices, creating a novel three-channel lithofacies image. This method allows for the application of DANNs by converting the complex lithofacies recognition task into a more standard image recognition problem. The developed method dramatically increases the input vector dimensions, enhancing the richness of the data input. The validation of results revealed that the DANN model trained for merely 3000 iterations successfully predicted lithofacies types of all eight testing samples in a mere 2.85 ms, showcasing superior accuracy. The innovative drilling data processing method proposed in this study enables DANNs to identify lithofacies with increased speed and accuracy. This offers a new direction for other DANNs utilizing drilling data.

DeepMDFC: A deep learning based android malware detection and family classification method

AbstractUnprecedented growth and prevalent adoption of the Android Operating System (OS) have triggered a substantial transformation, not only within the smartphone industry but across various categories of intelligent devices. These intelligent devices store a wealth of sensitive data, making them enticing targets for malicious individuals who create harmful Android applications to steal this data for malicious purposes. While numerous Android malware detection methods have been proposed, the exponential growth in sophisticated and malicious Android apps presents an unprecedented challenge to existing detection techniques. Some of the researchers have attempted to classify malicious Android applications into families through static analysis of applications but most of them are evaluated on applications of previous API levels. This paper introduces a novel dataset compromising of 2019 to 2021 applications and proposes a Deep Learning based Malware Detection and Family Classification method (DeepMDFC) to detect and classify emerging malicious Android applications through static analysis and deep Artificial Neural Networks. Experimental findings indicate that DeepMDFC surpasses standard machine learning algorithms, achieving accuracy rates of 99.3% and 96.7% for Android malware detection and classification, respectively, with a limited size feature set. The performance of DeepMDFC is also assessed using the benchmark dataset (DREBIN) and results showed that DeepMDFC surpasses these methods in terms of performance. Furthermore, it leverages the proposed dataset to construct a prediction model that adeptly identifies malicious Android applications from both the years 2022 and 2023. This process the potency and resilience of DeepMDFC against emerging Android applications.

Img2fmri: a python package for predicting group-level fMRI responses to visual stimuli using deep neural networks.

Here we introduce a new python package, img2fmri, to predict group-level fMRI responses to individual images. This prediction model uses an artificial deep neural network (DNN), as DNNs have been successful at predicting cortical responses in the human visual cortex when trained on real world visual categorization tasks. To validate our model, we predict fMRI responses to images our model has not previously seen from a new dataset. We then show how our frame-by-frame prediction model can be extended to a continuous visual stimulus by predicting an fMRI response to Pixar Animation Studio's short film Partly Cloudy. In analyzing the timepoint-timepoint similarity of our predicted fMRI response around human-annotated event boundaries in the movie, we find that our model outperforms the baseline model in describing the dynamics of the real fMRI response around these event boundaries, particularly in the timepoints just before and at an event. These analyses suggest that in visual areas of the brain, at least some of the temporal dynamics we see in the brain's processing of continuous, naturalistic stimuli can be explained by dynamics in the stimulus itself, since they can be predicted from our frame-by-frame model. All code, analyses, tutorials, and installation instructions can be found at https://github.com/dpmlab/img2fmri.

A mechanics-informed deep learning framework for data-driven nonlinear viscoelasticity

A mechanics-informed, data-driven framework for learning the constitutive law of a nonlinear viscoelastic material from stress–strain data using deep artificial neural networks (ANNs) is proposed. In this framework, each ANN architecture is designed and its input and output are selected such that fundamental laws of mechanics and thermodynamics relevant to nonlinear viscoelasticity are strongly satisfied by the resulting data-driven constitutive model. The framework itself extends a previously developed mechanics-informed methodology for the data-driven constitutive modeling of nonlinear elasticity to account for the dependence of the stress on the entire strain history. It is grounded in an additive decomposition of the strain energy density and an internal variable approach to viscoelasticity. Collectively, these are characterized by three potentials whose ANN representations are taught the second law of thermodynamics, various stability conditions, consistency (preservation of rigid body modes), and the recovery of elastic limits. The proposed mechanics-informed deep learning framework for data-driven nonlinear viscoelasticity is illustrated and its performance is assessed for one academic problem where synthetic data is generated by a nonlinear convolution; a coupon-level application associated with a multi-scale homogenization of a viscoelastic woven fabric material; and two representative finite element analyses performed using the material law generated during the previous application, to highlight different features of the framework.

Artificial deep neural network enables one-size-fits-all electric vehicle user behavior prediction framework

As greener mobility becomes the norm with the advent of electric vehicles (EVs), a natural question arises: how big of a change are we seeing in terms of the stochastic energy demands imposed by EVs? There have been considerable difficulties in analyzing the adoption of all types of EV infrastructure due to the lack of publicly available individual EV user data across various service providers, such as distribution network operators, EV aggregators, EV users, utilities, regulators, and academics. In this study, we introduce the JETPANN (Joint EV energy consumption and charging duration Training Prediction using Artificial Neural Networks), a novel jointly trainable artificial deep neural network framework for predicting stochastic EV user behavior by predicting their stay/charging duration and energy consumption simultaneously. We used a large-scale dataset of individual EV users’ charging transactions collected at a multi-site UCLA campus, the Los Angeles Department of Water and Power (LADWP), the Port of LA, the City of Santa Monica, and the City of Pasadena over a five-year period. This includes more than 50 EVSEs (Electric Vehicle Supply Equipment) and 216 EV charging points. Over 50,000 real-time transactions were recorded in our database with 341 distinct EV users. In this research, essential attributes of each charging session were identified, extracted from our database, and then fed into our JETPANN network. Using a jointly trained framework, our AI-enabled network predicts the stay/charging duration and the energy consumption of all the recorded EV users with an accuracy of above 99 percent. The proposed technique was tested and validated using all the collected historical charging data from individual EV users via realizing mean-absolute errors of training loss versus validation loss. Utilizing hyperparameter tuning and semi-grid search, the JETPANN with semi-optimized hyperparameters was achieved with the lowest mean-absolute errors of 0.927 and 0.068 in predicting stay/charging durations and energy consumptions, respectively, in a jointly trainable framework. This study demonstrates the potential of the JETPANN framework in the prediction of EV users’ behavior by using large-scale real-world data collected from a diverse pool of EV users, including faculty and students, occasional and frequent users, and early and late adopters.

A Novel Approach for Individual Design Perception Based on Fuzzy Inference System Training with YUKI Algorithm

This paper presents a novel approach for individual design perception modeling using the YUKI algorithm-trained Fuzzy Inference System. The study focuses on understanding how individuals perceive design based on personality traits, particularly openness to experience, using the YUKI algorithm and Fuzzy C-means clustering algorithm. The approach generates several Sugeno-type Fuzzy Inference System models to predict design perception, to minimize the Root Mean Squared Error between the model prediction and the actual design perception of participants. The results demonstrate that the suggested method offers more accurate predictions compared to the traditional Fuzzy C-means Fuzzy Inference System and Deep Artificial Neural Networks, and the Root Mean Square deviation for individual design perceptions falls within a satisfactory range of 0.84 to 1.32. The YUKI algorithm-trained Fuzzy Inference System proves effective in clustering individuals based on their level of openness, providing insights into how personality traits influence design perception.

An exact mapping from ReLU networks to spiking neural networks

Deep spiking neural networks (SNNs) offer the promise of low-power artificial intelligence. However, training deep SNNs from scratch or converting deep artificial neural networks to SNNs without loss of performance has been a challenge. Here we propose an exact mapping from a network with Rectified Linear Units (ReLUs) to an SNN that fires exactly one spike per neuron. For our constructive proof, we assume that an arbitrary multi-layer ReLU network with or without convolutional layers, batch normalization and max pooling layers was trained to high performance on some training set. Furthermore, we assume that we have access to a representative example of input data used during training and to the exact parameters (weights and biases) of the trained ReLU network. The mapping from deep ReLU networks to SNNs causes zero percent drop in accuracy on CIFAR10, CIFAR100 and the ImageNet-like data sets Places365 and PASS. More generally our work shows that an arbitrary deep ReLU network can be replaced by an energy-efficient single-spike neural network without any loss of performance.