Constructive Neural Network Research Articles

The Method of Construction of Logical Neural Networks on the Basis of Variable-Valued Logical Functions

In this paper, we suggest a method of constructing logical neural networks on the basis of variable-valued logical functions. We prove the theorem on a possibility of representation of any logical function as a logical neural network. The proof given in the paper contains an algorithm of the construction of the logical neural network. We point out the possibility of the generalization of the result obtained to the case of fuzzy logic.

Machine learning for accelerating macroscopic parameters prediction for poroelasticity problem in stochastic media

In this paper, we consider a coarse grid approximation (numerical homogenization and multiscale finite element method) for the poroelasticity problem with stochastic properties. The proposed method is based on the construction of deep neural network for fast calculation of macroscopic parameters for a coarse grid approximation of the problem. We train neural networks on a set of selected realizations of local microscale stochastic fields and macroscale characteristics (effective property tensor or local matrix). We construct a deep learning method through a convolutional neural network (CNN) to learn a map between stochastic fields and macroscopic parameters. Numerical results are presented for two and three-dimensional model problems and show that the proposed method provides fast and accurate effective property predictions.

Deep neural networks for predicting restricted mean survival times.

Restricted mean survival time (RMST) is a useful summary measurement of the time-to-event data, and it has attracted great attention for its straightforward clinical interpretation. In this article, I propose a deep neural network model that directly relates the RMST to its baseline covariates for simultaneous prediction of RSMT at multiple times. Each subject's survival time is transformed into a series of jackknife pseudo observations and then used as quantitative response variables in a deep neural network model. By using the pseudo values, a complex survival analysis is reduced to a standard regression problem, which greatly simplifies the neural network construction. By jointly modeling RMST at multiple times, the neural network model gains prediction accuracy by information sharing across times. The proposed network model was evaluated by extensive simulation studies and was further illustrated on three real datasets. In real data analyses, I also used methods to open the blackbox by identifying subject-specific predictors and their importance in contributing to the risk prediction. The source code is freely available at http://github.com/lilizhaoUM/DnnRMST. Supplementary data are available at Bioinformatics online.

МОДЕЛЮВАННЯ ВПЛИВУ СОЦІАЛЬНО-ЕКОНОМІЧНИХ КРИЗОВИХ ЯВИЩ НА МІГРАЦІЙНІ ПРОЦЕСИ

Migration movements, in particular external migration, are common to all countries of the world, and Ukraine is not an exception. The peculiarity of the role of movement between states is the ability to achieve a balance between the quantitative and qualitative composition of the labor force, so it is important to understand the nature of the migration process and be able to predict it for decision-making at the state level. Modelling of the external migration process are taking into account the influence of economic factors, so using of machine learning methods is proposed. This choice of methodology is explained by the fact that machine learning allows to obtain high values of accuracy in forecasting of socio-economic phenomena. Indicators of socio-economic crises, as well as factors characterizing the level of development of the state were used in the selection of factors influencing the external migration process. The migration process was predicted using the "random forest" method. The selection of the model was based on minimizing the deviations of the predicted data from the actual ones. Еhe model was trained on statistical information from 15 countries, covering a period of 20 years. Сountries were selected according to the size of migration flows with Ukraine, taking into account both the flow of departure and the flow of arrival. Net migration rate, counted by dividing the number of immigrants by the number of emigrants, was used as the indicator of migration movement. To predict the importance of influencing factors, time series forecasting methods were used, in particular the ARIMA model, the choice of the best model was based on minimizing the Akaike criterion. The obtained results were used to forecast the external migration process for Ukraine for 2020–2021. In the course of the work the methods of correlation and regression analysis, theories of statistics, ensemble methods of machine learning, methods of forecasting time series and construction of neural networks were used. The model presented in the article can be used to predict the nature of migration between states, taking into account the influence of various factors.

Sustainable digital technologies in the management of infrastructure property complexes

The existing procedure for regulating property relations, built on the basis of the concept of market equilibrium, requires updating on the basis of a mechanism for preparing and making fair decisions, taking into account the wide opportunities for analysis through the use of artificial intelligence. We continue and develop the presented research by describing the implementation of the decision-making preparation mechanism based on the creation of a financial digital twin of the marine infrastructure property complex. The purpose of this study is to consider options and digital mechanisms for making management decisions on the development of infrastructure property complexes that are justified from an economic point of view, as well as from the point of view of public efficiency. The following tasks are solved: define a regional marine property complex as a type of infrastructure property complex; create criteria for evaluating the effectiveness of the regional marine infrastructure property complex; describe the implementation of the mechanism for preparing for decision-making based on the creation of a financial digital double of the research object. Mechanisms of support for managerial decision-making using artificial intelligence based on neural network construction are considered. The scientific novelty lies in the fact that the study offers an effective mechanism for preparing and making decisions, implemented on the basis of creating a financial digital double of the marine infrastructure property complex.

Development of a neuron model based on DNAzyme regulation.

Neural networks based on DNA molecular circuits play an important role in molecular information processing and artificial intelligence systems. In fact, some DNA molecular systems can become dynamic units with the assistance of DNAzymes. The complex DNA circuits can spontaneously induce corresponding feedback behaviors when their inputs changed. However, most of the reported DNA neural networks have been implemented by the toehold-mediated strand displacement (TMSD) method. Therefore, it was important to develop a method to build a neural network utilizing the TMSD mechanism and adding a mechanism to account for modulation by DNAzymes. In this study, we designed a model of a DNA neuron controlled by DNAzymes. We proposed an approach based on the DNAzyme modulation of neuronal function, combing two reaction mechanisms: DNAzyme digestion and TMSD. Using the DNAzyme adjustment, each component simulating the characteristics of neurons was constructed. By altering the input and weight of the neuron model, we verified the correctness of the computational function of the neurons. Furthermore, in order to verify the application potential of the neurons in specific functions, a voting machine was successfully implemented. The proposed neuron model regulated by DNAzymes was simple to construct and possesses strong scalability, having great potential for use in the construction of large neural networks.

Signal processing algorithm for neural networks with integrodifferential splines as an activation function and its particular case of image classification

Classic neural networks suppose trainable parameters to include just weights of neurons. This paper proposes parabolic integrodifferential splines (ID-splines), developed by author, as a new kind of activation function (AF) for neural networks, where ID-splines coefficients are also trainable parameters. Parameters of ID-spline AF together with weights of neurons are vary during the training in order to minimize the loss function thus reducing the training time and increasing the operation speed of the neural network. The newly developed algorithm enables software implementation of the ID-spline AF as a tool for neural networks construction, training and operation. It is proposed to use the same ID-spline AF for neurons in the same layer, but different for different layers. In this case, the parameters of the ID-spline AF for a particular layer change during the training process independently of the activation functions (AFs) of other network layers. In order to comply with the continuity condition for the derivative of the parabolic ID-spline on the interval (x x0, n) , its parameters fi (i= 0,...,n) should be calculated using the tridiagonal system of linear algebraic equations: To solve the system it is necessary to use two more equations arising from the boundary conditions for specific problems. For exam- ple the values of the grid function (if they are known) in the points (x x0, n) may be used for solving the system above: f f x0 = ( 0) , f f xn = ( n) . The parameters Iii+1 (i= 0,...,n−1 ) are used as trainable parameters of neural networks. The grid boundaries and spacing of the nodes of ID-spline AF are best chosen experimentally. The optimal selection of grid nodes allows improving the quality of results produced by the neural network. The formula for a parabolic ID-spline is such that the complexity of the calculations does not depend on whether the grid of nodes is uniform or non-uniform. An experimental comparison of the results of image classification from the popular FashionMNIST dataset by convolutional neural 0, x< 0 networks with the ID-spline AFs and the well-known ReLUx( ) =AF was carried out. The results reveal that the usage x x, ≥ 0 of the ID-spline AFs provides better accuracy of neural network operation than the ReLU AF. The training time for two convolutional layers network with two ID-spline AFs is just about 2 times longer than with two instances of ReLU AF. Doubling of the training time due to complexity of the ID-spline formula is the acceptable price for significantly better accuracy of the network. Wherein the difference of an operation speed of the networks with ID-spline and ReLU AFs will be negligible. The use of trainable ID-spline AFs makes it possible to simplify the architecture of neural networks without losing their efficiency. The modification of the well-known neural networks (ResNet etc.) by replacing traditional AFs with ID-spline AFs is a promising approach to increase the neural network operation accuracy. In a majority of cases, such a substitution does not require to train the network from scratch because it allows to use pre-trained on large datasets neuron weights supplied by standard software libraries for neural network construction thus substantially shortening training time.

OPTIMAL GENETIC ALGORITHM SELECTION FOR DEEP NEURAL NETWORK SETTINGS

The problem of construction of deep neural networks with the use of genetic algorithms is considered. The problem of structural-parametric synthesis with creation of neural networks is defined. The main purpose of the study is to find a deep neural network that is optimal for solving urgent problems. The classification problem is chosen as the urgent problem to solve. Also the classification of genetic algorithms is given, which are used as a basis for establishing the parameters of deep neural networks A system for the optimal tuning of the parameters of deep neural networks is proposed, which includes a two-stage algorithm. At the first stage of the algorithm, a multicriteria genetic algorithm is selected from a set of possible ones (genetic algorithm of vector estimation, genetic algorithm of Fonseca and Fleming, genetic algorithm of Pareto approximation with niche, genetic algorithm of sorting without dominance, genetic algorithm of Pareto force, genetic algorithm of Pareto-2 force ) that best fits the given training sample. At the second step, the problem of structural-parametric synthesis of a neural network is solved according to the criteria of accuracy and complexity. As a result of training, the values of the neural network parameters are found, such as: the number of layers, the number of neurons in each layer, the values of the weight coefficients.The modeling of the proposed system is carried out. The results of modeling, comparison of results with similar software packages are presented. The obtained results show the possibilities of wide use.

Electrical Properties and Biological Synaptic Simulation of Ag/MXene/SiO2/Pt RRAM Devices

Utilizing electronic devices to emulate biological synapses for the construction of artificial neural networks has provided a feasible research approach for the future development of artificial intelligence systems. Until now, different kinds of electronic devices have been proposed in the realization of biological synapse functions. However, the device stability and the power consumption are major challenges for future industrialization applications. Herein, an electronic synapse of MXene/SiO2 structure-based resistive random-access memory (RRAM) devices has been designed and fabricated by taking advantage of the desirable properties of SiO2 and 2D MXene material. The proposed RRAM devices, Ag/MXene/SiO2/Pt, exhibit the resistance switching characteristics where both the volatile and nonvolatile behaviors coexist in a single device. These intriguing features of the Ag/MXene/SiO2/Pt devices make them more applicable for emulating biological synaptic plasticity. Additionally, the conductive mechanisms of the Ag/MXene/SiO2/Pt RRAM devices have been discussed on the basis of our experimental results.

The Northern Extragalactic WISE × Pan-STARRS (NEWS) catalogue

This study involves two photometric catalogues, AllWISE and Pan-STARRS Data Release 1, which were cross-matched to identify extragalactic objects among the common sources of these catalogues. To separate galaxies and quasars from stars, we created a machine-learning model that is trained on photometric (in fact, colour-based) information from the optical and infrared wavelength ranges. The model is based on three important procedures: the construction of the autoencoder artificial neural network, separation of galaxies and quasars from stars with a support vector machine (SVM) classifier, and cleaning of the AllWISE × PS1 sample to remove sources with abnormal colour indices using a one-class SVM. As a training sample, we employed a set of spectroscopically confirmed sources from the Sloan Digital Sky Survey Data Release 14. Having applied the classification model to the data of crossing the AllWISE and Pan-STARRS DR1 samples, we created the Northern Extragalactic WISE × Pan-STARRS (NEWS) catalogue, containing 40 million extragalactic objects and covering 3/4 of celestial sphere up tog = 23m. Several independent classification quality tests, namely, the astrometric test along with others based on the use of data from spectroscopic surveys show similar results and indicate a high purity (∼98.0%) and completeness (> 98%) for the NEWS catalogue within the magnitude range of 19.0m < g < 22.5m. The classification quality still retains quite acceptable levels of 70% for purity and 97% for completeness for the brightest and faintest objects from this magnitude range. In addition, validation with external data sets has demonstrated the need for using only those sources in the NEWS catalogue that are outside the zone with the enhanced extinction. We show that the number of quasars from the NEWS catalogue identified inGaiaDR2 exceeds the number of quasars previously identified inGaiaDR2 with the use of the AllWISEAGN catalogue. These quasars may be used in future as an additional sample for testing and anchoring theGaiaCelestial Reference Frame.

SciANN: A Keras/TensorFlow wrapper for scientific computations and physics-informed deep learning using artificial neural networks

In this paper, we introduce SciANN, a Python package for scientific computing and physics-informed deep learning using artificial neural networks. SciANN uses the widely used deep-learning packages TensorFlow and Keras to build deep neural networks and optimization models, thus inheriting many of Keras’s functionalities, such as batch optimization and model reuse for transfer learning. SciANN is designed to abstract neural network construction for scientific computations and solution and discovery of partial differential equations (PDE) using the physics-informed neural networks (PINN) architecture, therefore providing the flexibility to set up complex functional forms. We illustrate, in a series of examples, how the framework can be used for curve fitting on discrete data, and for solution and discovery of PDEs in strong and weak forms. We summarize the features currently available in SciANN, and also outline ongoing and future developments.

Deep Neural Networks for Survival Analysis Using Pseudo Values.

There has been increasing interest in modelling survival data using deep learning methods in medical research. Current approaches have focused on designing special cost functions to handle censored survival data. We propose a very different method with two simple steps. In the first step, we transform each subject's survival time into a series of jackknife pseudo conditional survival probabilities and then use these pseudo probabilities as a quantitative response variable in the deep neural network model. By using the pseudo values, we reduce a complex survival analysis to a standard regression problem, which greatly simplifies the neural network construction. Our two-step approach is simple, yet very flexible in making risk predictions for survival data, which is very appealing from the practice point of view. The source code is freely available at http://github.com/lilizhaoUM/DNNSurv.

Fake metabolomics chromatogram generation for facilitating deep learning of peak-picking neural networks

Finding peaks in chromatograms and determining their start and end points (peak picking) is a core task in chromatography based biotechnology. Construction of peak-picking neural networks by deep learning was, however, hampered from the preparation of exact peak-picked or "labeled" chromatograms since the exact start and end points were often unclear in overlapping peaks in real chromatograms. We present a design of a fake chromatogram generator, along with a method for deep learning of peak-picking neural networks. Fake chromatograms were generated by generation of fake peaks, random sampling of peak positions from feature distributions, and merging with real blank sample chromatograms. Information on the exact start and end points, as labeled on the fake chromatograms, were effective for training and evaluating peak-picking neural networks. The peak-picking neural networks constructed herein outperformed conventional peak-picking software and showed comparable performance with that of experienced operators for processing the widely targeted metabolome data. Results of this study indicate that generation of fake chromatograms would be crucial for developing peak-picking neural networks and a key technology for further improvement of peak picking neural networks.

Back propagation neural network rainfall prediction model based on particle swarm optimization

It is a feasible method to use neural network construction to predict rainfall in the region. However, the error of this method is bigger, and its error is from the neural network structure itself. In view of the problem that the rainfall prediction precision based on BP neural network construction is low, this thesis proposes to optimize BP neural network rainfall prediction model with particle swarm optimization (PSO) algorithm, and then use the same training samples and testing samples to conduct a simulation experiment to the forecast model before and after optimization. The results show that the experimental testing result of the forecast model after optimization is more in line with the actual value.

Construction of feedforward neural networks with simple architectures and approximation abilities

The universal approximation property of feedforward neural networks (FNNs) is the basis for all FNNs applications. In almost all existing FNN approximation studies, it is always assumed that the activation function of the network satisfies certain conditions, such as sigmoid or bounded continuity. This paper focuses on building simple architecture FNNs with approximation ability by constructing an activation function. First, for any continuous function defined on [0, 1] and an arbitrary accuracy ϵ > 0, an FNN is constructed, which has only one neuron and a fixed weight 1 and can approximate the function with an accuracy of ϵ. Thereafter, we study the representation of FNNs for polynomial functions by constructing a proper activation function. It is proved that any algebraic polynomial with the degree n can be represented by an FNN. Further, a piecewise function is constructed as an activation function of an FNN such that the FNN represents the famous Bernstein polynomials.

Detecting composite orders in layered models via machine learning

Determining the phase diagram of systems consisting of smaller subsystems ‘connected’ via a tunable coupling is a challenging task relevant for a variety of physical settings. A general question is whether new phases, not present in the uncoupled limit, may arise. We use machine learning and a suitable quasidistance between different points of the phase diagram to study layered spin models, in which the spin variables constituting each of the uncoupled systems (to which we refer as layers) are coupled to each other via an interlayer coupling. In such systems, in general, composite order parameters involving spins of different layers may emerge as a consequence of the interlayer coupling. We focus on the layered Ising and Ashkin–Teller models as a paradigmatic case study, determining their phase diagram via the application of a machine learning algorithm to the Monte Carlo data. Remarkably our technique is able to correctly characterize all the system phases also in the case of hidden order parameters, i.e. order parameters whose expression in terms of the microscopic configurations would require additional preprocessing of the data fed to the algorithm. We correctly retrieve the three known phases of the Ashkin–Teller model with ferromagnetic couplings, including the phase described by a composite order parameter. For the bilayer and trilayer Ising models the phases we find are only the ferromagnetic and the paramagnetic ones. Within the approach we introduce, owing to the construction of convolutional neural networks, naturally suitable for layered image-like data with arbitrary number of layers, no preprocessing of the Monte Carlo data is needed, also with regard to its spatial structure. The physical meaning of our results is discussed and compared with analytical data, where available. Yet, the method can be used without any a priori knowledge of the phases one seeks to find and can be applied to other models and structures.

CNNs on surfaces using rotation-equivariant features

This paper is concerned with a fundamental problem in geometric deep learning that arises in the construction of convolutional neural networks on surfaces. Due to curvature, the transport of filter kernels on surfaces results in a rotational ambiguity, which prevents a uniform alignment of these kernels on the surface. We propose a network architecture for surfaces that consists of vector-valued, rotation-equivariant features. The equivariance property makes it possible to locally align features, which were computed in arbitrary coordinate systems, when aggregating features in a convolution layer. The resulting network is agnostic to the choices of coordinate systems for the tangent spaces on the surface. We implement our approach for triangle meshes. Based on circular harmonic functions, we introduce convolution filters for meshes that are rotation-equivariant at the discrete level. We evaluate the resulting networks on shape correspondence and shape classifications tasks and compare their performance to other approaches.

심층신경망 모형을 이용한 서울시 도시공원 및 녹지공간의 열섬저감효과 분석

The Urban Heat Island (UHI) Effect has intensified due to urbanization and heat management at the urban level is treated as an important issue. Green space improvement projects and environmental policies are being implemented as a way to alleviate Urban Heat Islands. Several studies have been conducted to analyze the correlation between urban green areas and heat with linear regression models. However, linear regression models have limitations explaining the correlation between heat and the multitude of variables as heat is a result of a combination of non-linear factors. This study evaluated the Heat Island alleviating effects in Seoul during the summer by using a deep neural network model methodology, which has strengths in areas where it is difficult to analyze data with existing statistical analysis methods due to variable factors and a large amount of data. Wide-area data was acquired using Landsat 8. Seoul was divided into a grid (30m x 30m) and the heat island reduction variables were enter in each grid space to create a data structure that is needed for the construction of a deep neural network using ArcGIS 10.7 and Python3.7 with Keras. This deep neural network was used to analyze the correlation between land surface temperature and the variables. We confirmed that the deep neural network model has high explanatory accuracy. It was found that the cooling effect by NDVI was the greatest, and cooling effects due to the park size and green space proximity were also shown. Previous studies showed that the cooling effects related to park size was 2°C-3°C, and the proximity effect was found to lower the temperature 0.3°C-2.3°C. There is a possibility of overestimation of the results of previous studies. The results of this study can provide objective information for the justification and more effective formation of new urban green areas to alleviate the Urban Heat Island phenomenon in the future.

Cascade neural network algorithm with analytical connection weights determination for modelling operations and energy applications

The performance and learning speed of the Cascade Correlation neural network (CasCor) may not be optimal because of redundant hidden units’ in the cascade architecture and the tuning of connection weights. This study explores the limitations of CasCor and its variants and proposes a novel constructive neural network (CNN). The basic idea is to compute the input connection weights by generating linearly independent hidden units from the orthogonal linear transformation, and the output connection weights by connecting hidden units in a linear relationship to the output units. The work is unique in that few attempts have been made to analytically determine the connection weights on both sides of the network. Experimental work on real energy application problems such as predicting powerplant electrical energy, predicting seismic hazards to prevent fatal accidents and reducing energy consumption by predicting building occupancy detection shows that analytically calculating the connection weights and generating non-redundant hidden units improves the convergence of the network. The proposed CNN is compared with that of the state-of-the-art machine learning algorithms. The work demonstrates that proposed CNN predicts a wide range of applications better than other methods.

Ecological monitoring of landscape sites using regularized artificial neural networks

The problem of recognition and analysis of landscape objects cartographic images for localization of pollution areas,monitoring of changes of images contours under the influence of natural and anthropogenic factors is considered. Thearchitecture of a deep neural network is proposed that uses the technique of regularization of artificial neural networksto prevent its overfitting. The results of the construction, training and application of a convolutional neural network withthe Dropout elimination procedure for problems of ecological landscape monitoring are presented. The prospects ofusing the proposed approach for image recognition of problematic landscape areas are determined.