Standard Feedforward Networks Research Articles

Solving the Dirichlet problem for the Monge–Ampère equation using neural networks

The Monge–Ampère equation is a full y nonlinear partial differential equation (PDE) of fundamental importance in analysis, geometry and in the applied sciences. In this paper we solve the Dirichlet problem associated with the Monge–Ampère equation using neural networks and we show that an ansatz using deep input convex neural networks can be used to find the unique convex solution. As part of our analysis we study the effect of singularities, discontinuities and noise in the source function, we consider nontrivial domains, and we investigate how the method performs in higher dimensions. We investigate the convergence numerically and present error estimates based on a stability result. We also compare this method to an alternative approach in which standard feed-forward networks are used together with a loss function which penalizes lack of convexity.

No-Transaction Band Network: A Neural Network Architecture for Efficient Deep Hedging

Deep hedging is a versatile framework for computing the optimal hedging strategy of derivatives in incomplete markets. However, it is subject to the action-dependence problem impeding efficient training because the appropriate hedging action at the next step depends on the current action. To overcome this issue, the authors leverage a no-transaction band strategy, an existing technique that provides optimal hedging strategies for European options and exponential utility. The authors theoretically argue this strategy to be optimal for a wider class of utilities and derivatives, including exotics. Based on the result, the authors propose a no-transaction band network, namely, a neural network architecture that facilitates fast training and precise evaluation of the optimal hedging strategy. Moreover, the authors experimentally demonstrate that, for European and lookback options, their architecture rapidly attains a better hedging strategy compared with a standard feed-forward network. The findings thus have important implications for the practical applications of deep hedging.

Experiments on Gaussian Dropout and Adadelta for Hepatocellular Carcinoma

Abstract: Hepatocellular carcinoma (HCC) is the most common type of primary liver cancer. Hepatocellular carcinoma occurs most often in people with chronic liver diseases, such as cirrhosis caused by hepatitis B or hepatitis C infection. Early detection and accurate predictive analysis play a pivotal role in the totality of the human population and are of extreme importance for enhanced life expectancy. With the advent of computation, there are well-defined publicly available datasets that can be leveraged for an accurate and temporarily efficient understanding of HCC. There exists preliminary work on these data samples that leverage classical machine learning algorithms, however, the state of the art is heavily skewed towards the deep neural networks. To improve the existing approaches, this paper seeks to leverage Gaussian Dropout, a variant of the standard dropout, for its remedial action on overfitting and related qualities. The pipeline is also tested and experimented with Adadelta, to obtain the applicability of these additions to a standard feed-forward network. These experiments and the methodologies considered for appendage to the baseline network are thoroughly assessed and validated by using the accepted metrics on an iteratively imputed dataset on multiple train-test data distributions.

Recurrent Connections in the Primate Ventral Visual Stream Mediate a Trade-Off Between Task Performance and Network Size During Core Object Recognition.

The computational role of the abundant feedback connections in the ventral visual stream is unclear, enabling humans and nonhuman primates to effortlessly recognize objects across a multitude of viewing conditions. Prior studies have augmented feedforward convolutional neural networks (CNNs) with recurrent connections to study their role in visual processing; however, often these recurrent networks are optimized directly on neural data or the comparative metrics used are undefined for standard feedforward networks that lack these connections. In this work, we develop task-optimized convolutional recurrent (ConvRNN) network models that more correctly mimic the timing and gross neuroanatomy of the ventral pathway. Properly chosen intermediate-depth ConvRNN circuit architectures, which incorporate mechanisms of feedforward bypassing and recurrent gating, can achieve high performance on a core recognition task, comparable to that of much deeper feedforward networks. We then develop methods that allow us to compare both CNNs and ConvRNNs to finely grained measurements of primate categorization behavior and neural response trajectories across thousands of stimuli. We find that high-performing ConvRNNs provide a better match to these data than feedforward networks of any depth, predicting the precise timings at which each stimulus is behaviorally decoded from neural activation patterns. Moreover, these ConvRNN circuits consistently produce quantitatively accurate predictions of neural dynamics from V4 and IT across the entire stimulus presentation. In fact, we find that the highest-performing ConvRNNs, which best match neural and behavioral data, also achieve a strong Pareto trade-off between task performance and overall network size. Taken together, our results suggest the functional purpose of recurrence in the ventral pathway is to fit a high-performing network in cortex, attaining computational power through temporal rather than spatial complexity.

Optimizing Neural Network Parameters For Effective Classification of Benign and Malicious Websites

Bio-inspired algorithms are the most powerful way to solve optimization problems. The objective of this paper is to use optimized network parameters for website classification and the efficiency of Neural Network is improved by optimized network parameters. The network parameters are optimized using two bio-inspired algorithms: Particle Swarm Optimization (PSO) and Cuckoo Search (CS) and the optimized parameter values are used with two neural network models, a standard Multi-Layer Feed Forward Network with Backpropagation (BPN) and Radial Basis Function (RBF) Network. Security is one of the major concerns in this digital era. There are numerous websites, which are potentially risky in spreading malicious files. It is difficult to detect such websites. In this work, Neural Network is used to classify the websites as benign and malicious. The proposed neural network models are tested with URL dataset. The experimental results are assessed in terms of Error reduction, training time and classification accuracy. The experimental result shows that the optimized network parameters have given good improvement in classification with faster convergence.

Evolutionary Neural Architecture Search for High-Dimensional Skip-Connection Structures on DenseNet Style Networks

Convolutional neural networks hold state-of-the-art results for image classification, and many neural architecture search algorithms have been proposed to discover high performance convolutional neural networks. However, the use of neural architecture search for the discovery of skip-connection structures, an important element in modern convolutional neural networks, is limited within the literature. Furthermore, while many neural architecture search algorithms utilize performance estimation techniques to reduce computation time, empirical evaluations of these performance estimation techniques remain limited. This work focuses on utilizing evolutionary neural architecture search to examine the search space of networks, which follow a fundamental DenseNet structure, but have no fixed skip connections. In particular, a genetic algorithm is designed, which searches the space consisting of all networks between a standard feedforward network and the corresponding DenseNet. To design the algorithm, lower fidelity performance estimation of this class of networks is examined and presented. The final algorithm finds networks that are more accurate than DenseNets on CIFAR10 and CIFAR100, and have fewer trainable parameters. The structures found by the algorithm are examined to shed light on the importance of different types of skip-connection structures in convolutional neural networks, including the discovery of a simple skip-connection removal, which improves DenseNet performance on CIFAR10.

No-Transaction Band Network: A Neural Network Architecture for Efficient Deep Hedging

Deep hedging (Buehler et al. 2019) is a versatile framework to compute the optimal hedging strategy of derivatives in incomplete markets. However, this optimal strategy is hard to train due to action dependence, that is, the appropriate hedging action at the next step depends on the current action. To overcome this issue, we leverage the idea of a no-transaction band strategy, which is an existing technique that gives optimal hedging strategies for European options and the exponential utility. We theoretically prove that this strategy is also optimal for a wider class of utilities and derivatives including exotics. Based on this result, we propose a no-transaction band network, a neural network architecture that facilitates fast training and precise evaluation of the optimal hedging strategy. We experimentally demonstrate that for European and lookback options, our architecture quickly attains a better hedging strategy in comparison to a standard feed-forward network.

Robust Learning with Implicit Residual Networks

In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to the improved stability of both forward and backward propagations, has a favorable impact on the generalization power, and allows for control the robustness of the network with only a few hyperparameters. In addition, the proposed reformulation of ResNet does not introduce new parameters and can potentially lead to a reduction in the number of required layers due to improved forward stability. Finally, we derive the memory-efficient training algorithm, propose a stochastic regularization technique, and provide numerical results in support of our findings.

Representing Uncertainty in Property Valuation Through a Bayesian Deep Learning Approach

Abstract Although deep learning-based valuation models are spreading throughout the real estate industry following the artificial intelligence boom, property owners and investors continue to doubt the accuracy of the results. In this study, we specify a neural network for predicting house prices. We suggest a standard feed-forward network with two hidden layers, and show that it is sufficiently reasonable to apply its prediction to real-world projects such as property valuation. In addition, we propose a Bayesian neural network for describing uncertainty in house price predictions while providing a means to quantify uncertainty for each prediction. We choose Gangnam-gu, Seoul for the analysis, and predict house prices in the area using both networks. Although the Bayesian neural network did not perform better than the conventional network, it could provide a tool to measure the uncertainty inherent in predicted prices. The findings of this study show that a Bayesian approach can model uncertainty in property valuation, thereby promoting the adoption of deep learning tools in the real estate industry.

Design of adaptive Elman networks for credit risk assessment

The banks' need of quantitative approaches for credit risk assessment is becoming more and more evident, due to the introduction of the Basel agreements. To this extent, we define an Elman network approach to determine the insolvency of a borrower, and compare its performances with classical neural networks approaches for bankruptcy prediction. Then, we devise an adaptive procedure to select the best network topology, by performing a multi-objective analysis to take into account different compromises between conflicting criteria. We apply our procedure to different real and partly-artificial-case scenarios composed of Italian SMEs by using predictors coming from balance-sheet ratios, credit-history ratios and macro-economic indicators; then, we compare our approach to other ones proposed in the literature and to a standard logistic regression tool used by practitioners; last, given the recent research interest towards the use of qualitative predictors in credit risk assessment, we also apply our approaches on qualitative data. The results show that the Elman networks are effective in assessing credit risk and robust with respect to criteria and data, confirming the applicability of neural networks to bankruptcy prediction. Our contribution adds to the discussion of the ongoing debate about comparing neural networks to standard techniques: in particular, we find Elman Networks to lead to lower classification errors than standard feed-forward networks, whilst results from the comparison to logistic regression vary with respect to the error class considered. As for the data, we remark that the use of macro-economics indicators does not lead to particular improvement in classification accuracy, except when used to improve results coming from the use of qualitative variables only.

Unsupervised learning by competing hidden units

It is widely believed that end-to-end training with the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility and which is motivated by Hebb's idea that change of the synapse strength should be local-i.e., should depend only on the activities of the pre- and postsynaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer and is capable of learning early feature detectors in a completely unsupervised way. These learned lower-layer feature detectors can be used to train higher-layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm on simple tasks.

Piecewise Polynomial Activation Functions for Feedforward Neural Networks

Since the origins of artificial neural network research, many models of feedforward networks have been proposed. This paper presents an algorithm which adapts the shape of the activation function to the training data, so that it is learned along with the connection weights. The activation function is interpreted as a piecewise polynomial approximation to the distribution function of the argument of the activation function. An online learning procedure is given, and it is formally proved that it makes the training error decrease or stay the same except for extreme cases. Moreover, the model is computationally simpler than standard feedforward networks, so that it is suitable for implementation on FPGAs and microcontrollers. However, our present proposal is limited to two-layer, one-output-neuron architectures due to the lack of differentiability of the learned activation functions with respect to the node locations. Experimental results are provided, which show the performance of the proposal algorithm for classification and regression applications.

Exploring global sentence representation for graph-based dependency parsing using BLSTM-SCNN

Deep Learning has been widely applied for dependency parsing in recent years. In this paper, we propose an effective deep neural network model for graph-based dependency parsing. In our model, first, a special feature extraction layer is elaborately designed by combining the bidirectional Long Short-Term Memory (BLSTM) and the segment-based Convolutional Neural Network (SCNN), which is able to capture rich contextual information of the sentence for parsing. Then, the features learnt in feature extraction layer are fed into the standard feed-forward network, which is trained with max-margin criteria and makes predictions for dependency labels. Finally, to search the best dependency structure for the sentence from the dependency graph, the classical dynamic programming algorithm is used. In our experiment, we test the proposed model on 14 different languages including English, Chinese and German, whose results show that the proposed model achieves competitive accuracies in unlabeled attachment scores and labeled attachment scores compared with state-of-the-art dependency parsers. What's more, the model shows better ability in recovering long-distance dependencies compared with common neural network models.

APPLICATION OF ARTIFICIAL NEURAL NETWORK FOR PREDICTION OF SUPERPLASTIC BEHAVIOUR IN NiAl ALLOYS

Chemical composition, grain size, and processing conditions such as temperature and strain rate have important influence on superplasticity of NiAl alloys, which would allow the optimization of these parameters in order to achieve the desired combination of properties. In this work, the optimal superplastic deformation conditions of NiAl alloys were studied by using artificial neural networks (ANN). The standard multilayer feedforward networks were trained and tested using comprehensive datasets from previous experimentally works on the as-extruded NiAl, NiAl-25Cr, NiAl-20Fe-Y(Ce), NiAl-30Fe-Y, NiAl-9Mo, NiAl-P alloys. Different effects are modeled, including the influence of the alloying elements on the superplastic, and the influence of deformation temperature, strain rate and grain size on the elongations during the superplastic tensile tests. The artificial neural network models are combined with computer programmers for optimization of the inputs in order to achieve desirable combination of outputs. Good performances of the neural networks are achieved. Results of this research propose a range of strain rate and temperature within which the NiAl alloy possesses superplasticity with larger elongations, although the deformation temperature and strain rate of superplastic alloys alternately influence each other within the range. These models are convenient and powerful tools for practical applications in superplastic prediction in NiAl alloys.

Groundwater modeling using hybrid of artificial neural network with genetic algorithm

Accurate estimates of groundwater level have a valuable effect in improving decision support systems of groundwater resources exploitation. The present study investigates the ability of a hybrid model of artificial neural network (ANN) and genetic algorithm (GA) in forecasting groundwater level in an individual well (target well). A standard feed forward networks (FFN) and recurrent neural networks (RNN) are utilized for performing the prediction task. Moreover, GA is used in order to determine the optimal structure of ANN (that is, number of neurons for each hidden layer). Air temperature, rainfall depth and groundwater levels in neighboring wells in Kerman plain (Kerman, Iran) were used as input data of the hybrid model. This study indicates that the ANN-GA model can be used successfully to forecast groundwater levels of individual wells. In addition, a comparative study of both hybrid models indicates that the feed forward networks performed better than the recurrent neural networks. Key words: Artificial neural network, feed forward networks, recurrent neural networks, genetic algorithm, groundwater level.

An Intelligent System for Hepatitis B Disease Diagnosis

In spite of all standardization efforts, medical diagnosis is still considered as an art owing to its uncertainty that today’s computing machinery cannot cope with. This research is performed for the search of substitutes to the resolution of complex medical diagnosis where human knowledge is detained in a general fashion. Artificial neural network algorithms provide a reasonably high-quality solution for medical problems and this paper is concerned with the diagnosis of hepatitis B. In this paper, we present an intelligent system for hepatitis B diagnosis using a standard feed-forward network with multilayer perceptron (MLP) structure and back propagation algorithm. The results obtained reveal that neural networks can be successfully used for the diagnosis of hepatitis B. Also, a comparative analysis is performed with a number of other knowledge-based systems designed for hepatitis B diagnosis, to prove the effectiveness of the proposed intelligent system based on MLP.

A transmission line fault locator based on Elman recurrent networks

In this paper, a transmission line fault location model which is based on an Elman recurrent network (ERN) has been presented for balanced and unbalanced short circuit faults. All fault situations with different inception times are implemented on a 380-kV prototype power system. Wavelet transform (WT) is used for selecting distinctive features about the faulty signals. The system has the advantages of utilizing single-end measurements, using both voltage and current signals. ERN is able to determine the fault location occurred on transmission line rapidly and correctly as an important alternative to standard feedforward back propagation networks (FFNs) and radial basis functions (RBFs) neural networks.

A neural network approach for credit risk evaluation

The Basel Committee on Banking Supervision proposes a capital adequacy framework that allows banks to calculate capital requirement for their banking books using internal assessments of key risk drivers. Hence the need for systems to assess credit risk. Among the new methods, artificial neural networks have shown promising results. In this work, we describe the case of a successful application of neural networks to credit risk assessment. We developed two neural network systems, one with a standard feedforward network, while the other with a special purpose architecture. The application is tested on real-world data, related to Italian small businesses. We show that neural networks can be very successful in learning and estimating the in bonis/ default tendency of a borrower, provided that careful data analysis, data pre-processing and training are performed.

Training a feed-forward network with incomplete data due to missing input variables

Abstract Data available for training a neural network may be deficient not only in quantity of data but entire independent variables with their data may be missing such as is often the situation for software engineering data. This may cause the relation based on the available data to exhibit the property of one-to-many (o-m) valuedness or almost o-m valuedness. Multiplayer perceptrons or feed-forward network however are generally trained to represent functions or m-o mappings. The solution consists of adding another input to the standard feed-forward network and of modifying the training algorithm to allow for determination of this input for which no training data is available. If the values for the additional input are restricted to a discrete set then they may be perceived as cluster identifiers and the training method may be perceived as another form of clustering or segmentation of the input. If the missing input variable is assumed to have a finite number of values the method proposed here may be compared to mixture of experts and mixture density networks except that the proposed method is more direct solution. The modified feed-forward network and training method has been successfully applied to several examples.

An Adaptive Connectionist Model of Cognitive Dissonance

This article proposes an adaptive connectionist model that implements an attributional account of cognitive dissonance. The model represents an attitude as the connection between the attitude object and behavioral-affective outcomes. Dissonance arises when circumstantial constraints induce a mismatch between the model's (mental) prediction and discrepant behavior or affect. Reduction of dissonance by attitude change is accomplished through long-lasting changes in the connection weights using the error-correcting delta learning algorithm. The model can explain both the typical effects predicted by dissonance theory as well as some atypical effects (i.e., reinforcement effect), using this principle of weight changes and by giving a prominent role to affective experiences. The model was implemented in a standard feedforward connectionist network. Computer simulations showed an adequate fit with several classical dissonance paradigms (inhibition, initiation, forced compliance, free choice, & misattribution), as well as novel studies that underscore the role of affect. A comparison with an earlier constraint satisfaction approach (Shultz & Lepper, 1996) indicates that the feedforward implementation provides a similar fit with these shortcomings of this previous model.