Arbitrary Neural Networks Research Articles

Svetlana a supervised segmentation classifier for Napari

We present Svetlana (SuperVised sEgmenTation cLAssifier for NapAri), an open-source Napari plugin dedicated to the manual or automatic classification of segmentation results. A few recent software tools have made it possible to automatically segment complex 2D and 3D objects such as cells in biology with unrivaled performance. However, the subsequent analysis of the results is oftentimes inaccessible to non-specialists. The Svetlana plugin aims at going one step further, by allowing end-users to label the segmented objects and to pick, train and run arbitrary neural network classifiers. The resulting network can then be used for the quantitative analysis of biophysical phenoma. We showcase its performance through challenging problems in 2D and 3D and provide a comprehensive discussion on its strengths and limits.

Size and Depth of Monotone Neural Networks: Interpolation and Approximation.

We study monotone neural networks with threshold gates where all the weights (other than the biases) are nonnegative. We focus on the expressive power and efficiency of the representation of such networks. Our first result establishes that every monotone function over [0,1]d can be approximated within arbitrarily small additive error by a depth-4 monotone network. When , we improve upon the previous best-known construction, which has a depth of d+1 . Our proof goes by solving the monotone interpolation problem for monotone datasets using a depth-4 monotone threshold network. In our second main result, we compare size bounds between monotone and arbitrary neural networks with threshold gates. We find that there are monotone real functions that can be computed efficiently by networks with no restriction on the gates, whereas monotone networks approximating these functions need exponential size in the dimension.

Solving partial differential equation for atmospheric dispersion of radioactive material using physics-informed neural network

The governing equations of atmospheric dispersion most often taking the form of a second-order partial differential equation (PDE). Currently, typical computational codes for predicting atmospheric dispersion use the Gaussian plume model that is an analytic solution. A Gaussian model is simple and enables rapid simulations, but it can be difficult to apply to situations with complex model parameters. Recently, a method of solving PDEs using artificial neural networks called physics-informed neural network (PINN) has been proposed. The PINN assumes the latent (hidden) solution of a PDE as an arbitrary neural network model and approximates the solution by optimizing the model. Unlike a Gaussian model, the PINN is intuitive in that it does not require special assumptions and uses the original equation without modifications. In this paper, we describe an approach to atmospheric dispersion modeling using the PINN and show its applicability through simple case studies. The results are compared with analytic and fundamental numerical methods to assess the accuracy and other features. The proposed PINN approximates the solution with reasonable accuracy. Considering that its procedure is divided into training and prediction steps, the PINN also offers the advantage of rapid simulations once the training is over.

Layer-wise regularized adversarial training using layers sustainability analysis framework

Deep neural network models are used today in various applications of artificial intelligence, the strengthening of which, in the face of adversarial attacks is of particular importance. An appropriate solution to adversarial attacks is adversarial training, which reaches a trade-off between robustness and generalization. This paper introduces a novel framework (Layer Sustainability Analysis (LSA)) for the analysis of layer vulnerability in an arbitrary neural network in the scenario of adversarial attacks. LSA can be a helpful toolkit to assess deep neural networks and to extend the adversarial training approaches towards improving the sustainability of model layers via layer monitoring and analysis. The LSA framework identifies a list of Most Vulnerable Layers (MVL list) of the given network. The relative error, as a comparison measure, is used to evaluate representation sustainability of each layer against adversarial inputs. The proposed approach for obtaining robust neural networks to fend off adversarial attacks is based on a layer-wise regularization (LR) over LSA proposal(s) for adversarial training (AT). This means that the AT-LR procedure could be used with any benchmark adversarial attack to reduce the vulnerability of network layers and to improve conventional adversarial training approaches. The proposed idea performs well theoretically and experimentally for state-of-the-art multilayer perceptron and convolutional neural network architectures. Additionally, a measure named robustness and generalization score or R&G score is defined to better evaluate each adversarially trained model over a variety of significant perturbations. Compared with the AT-LR and its corresponding base adversarial training, the R&G score on Moon, MNIST, and CIFAR-10 benchmark datasets was increased by 56.52%, 75.82%, and 6.54%, respectively for more significant perturbations. The LSA framework is available and published at https://github.com/khalooei/LSA.

Feature alignment as a generative process.

Reversibility in artificial neural networks allows us to retrieve the input given an output. We present feature alignment, a method for approximating reversibility in arbitrary neural networks. We train a network by minimizing the distance between the output of a data point and the random output with respect to a random input. We applied the technique to the MNIST, CIFAR-10, CelebA, and STL-10 image datasets. We demonstrate that this method can roughly recover images from just their latent representation without the need of a decoder. By utilizing the formulation of variational autoencoders, we demonstrate that it is possible to produce new images that are statistically comparable to the training data. Furthermore, we demonstrate that the quality of the images can be improved by coupling a generator and a discriminator together. In addition, we show how this method, with a few minor modifications, can be used to train networks locally, which has the potential to save computational memory resources.

Codebase release 3.4 for NetKet

We introduce version 3 of NetKet, the machine learning toolbox for many-body quantum physics. NetKet is built around neural quantum states and provides efficient algorithms for their evaluation and optimization. This new version is built on top of JAX, a differentiable programming and accelerated linear algebra framework for the Python programming language. The most significant new feature is the possibility to define arbitrary neural network ansätze in pure Python code using the concise notation of machine-learning frameworks, which allows for just-in-time compilation as well as the implicit generation of gradients thanks to automatic differentiation. NetKet 3 also comes with support for GPU and TPU accelerators, advanced support for discrete symmetry groups, chunking to scale up to thousands of degrees of freedom, drivers for quantum dynamics applications, and improved modularity, allowing users to use only parts of the toolbox as a foundation for their own code.

NetKet 3: Machine Learning Toolbox for Many-Body Quantum Systems

We introduce version 3 of NetKet, the machine learning toolbox for many-body quantum physics. NetKet is built around neural quantum states and provides efficient algorithms for their evaluation and optimization. This new version is built on top of JAX, a differentiable programming and accelerated linear algebra framework for the Python programming language. The most significant new feature is the possibility to define arbitrary neural network ansätze in pure Python code using the concise notation of machine-learning frameworks, which allows for just-in-time compilation as well as the implicit generation of gradients thanks to automatic differentiation. NetKet 3 also comes with support for GPU and TPU accelerators, advanced support for discrete symmetry groups, chunking to scale up to thousands of degrees of freedom, drivers for quantum dynamics applications, and improved modularity, allowing users to use only parts of the toolbox as a foundation for their own code.

Machine Learning-Based 5G-and-Beyond Channel Estimation for MIMO-OFDM Communication Systems.

Channel estimation plays a critical role in the system performance of wireless networks. In addition, deep learning has demonstrated significant improvements in enhancing the communication reliability and reducing the computational complexity of 5G-and-beyond networks. Even though least squares (LS) estimation is popularly used to obtain channel estimates due to its low cost without any prior statistical information regarding the channel, this method has relatively high estimation error. This paper proposes a new channel estimation architecture with the assistance of deep learning in order to improve the channel estimation obtained by the LS approach. Our goal is achieved by utilizing a MIMO (multiple-input multiple-output) system with a multi-path channel profile for simulations in 5G-and-beyond networks under the level of mobility expressed by the Doppler effects. The system model is constructed for an arbitrary number of transceiver antennas, while the machine learning module is generalized in the sense that an arbitrary neural network architecture can be exploited. Numerical results demonstrate the superiority of the proposed deep learning-based channel estimation framework over the other traditional channel estimation methods popularly used in previous works. In addition, bidirectional long short-term memory offers the best channel estimation quality and the lowest bit error ratio among the considered artificial neural network architectures.

Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks

Despite the popularity of Convolutional Neural Networks (CNN), the problem of uncertainty quantification (UQ) of CNN has been largely overlooked. Lack of efficient UQ tools severely limits the application of CNN in certain areas, such as medicine, where prediction uncertainty is critically important. Among the few existing UQ approaches that have been proposed for deep learning, none of them has theoretical consistency that can guarantee the uncertainty quality. To address this issue, we propose a novel bootstrap based framework for the estimation of prediction uncertainty. The inference procedure we use relies on convexified neural networks to establish the theoretical consistency of bootstrap. Our approach has a significantly less computational load than its competitors, as it relies on warm-starts at each bootstrap that avoids refitting the model from scratch. We further explore a novel transfer learning method so our framework can work on arbitrary neural networks. We experimentally demonstrate our approach has a much better performance compared to other baseline CNNs and state-of-the-art methods on various image datasets.

Stability of Neural Ordinary Differential Equations with Power Nonlinearities

The article presents a study of solutions of ODEs system with a special nonlinear part, which is a continuous analogue of an arbitrary recurrent neural network (neural ODEs). As a nonlinear part of the mentioned system of differential equations, we used sums of piecewise continuous functions, where each term is a power function. (These are activation functions.) The use of power activation functions (PAF) in neural networks is a generalization of well-known the rectified linear units (ReLU). In the present time ReLU are commonly used to increase the depth of trained of a neural network. Therefore, the introduction of PAF into neural networks significantly expands the possibilities of ReLU. Note that the purpose of introducing power activation functions is that they allow one to obtain verifiable Lyapunov stability conditions for solutions of the system differential equations simulating the corresponding dynamic processes. In turn, Lyapunov stability is one of the guarantees of the adequacy of the neural network model for the process under study. In addition, from the global stability (or at least the boundedness) of continuous analog solutions it follows that learning process of the corresponding neural network will not diverge for any training sample.

EncoDeep

This article proposes EncoDeep, an end-to-end framework that facilitates encoding, bitwidth customization, fine-tuning, and implementation of neural networks on FPGA platforms. EncoDeep incorporates nonlinear encoding to the computation flow of neural networks to save memory. The encoded features demand significantly lower storage compared to the raw full-precision activation values; therefore, the execution flow of EncoDeep hardware engine is completely performed within the FPGA using on-chip streaming buffers with no access to the off-chip DRAM. We further propose a fully automated optimization algorithm that determines the flexible encoding bitwidths across network layers. EncoDeep full-stack framework comprises a compiler that takes a high-level Python description of an arbitrary neural network. The compiler then instantiates the corresponding elements from EncoDeep Hardware library for FPGA implementation. Our evaluations on MNIST, SVHN, and CIFAR-10 datasets demonstrate an average of 4.65× throughput improvement compared to stand-alone weight encoding. We further compare EncoDeep with six FPGA accelerators on ImageNet, showing an average of 3.6× and 2.54× improvement in throughput and performance-per-watt, respectively.

Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks

Architecture optimization, which is a technique for finding an efficient neural network that meets certain requirements, generally reduces to a set of multiple-choice selection problems among alternative sub-structures or parameters. The discrete nature of the selection problem, however, makes this optimization difficult. To tackle this problem we introduce a novel concept of a trainable gate function. The trainable gate function, which confers a differentiable property to discrete-valued variables, allows us to directly optimize loss functions that include non-differentiable discrete values such as 0-1 selection. The proposed trainable gate can be applied to pruning. Pruning can be carried out simply by appending the proposed trainable gate functions to each intermediate output tensor followed by fine-tuning the overall model, using any gradient-based training methods. So the proposed method can jointly optimize the selection of the pruned channels while fine-tuning the weights of the pruned model at the same time. Our experimental results demonstrate that the proposed method efficiently optimizes arbitrary neural networks in various tasks such as image classification, style transfer, optical flow estimation, and neural machine translation.

Implementacija neuronske mreže u Javi

Artificial neural networks are a powerful tool that engineers can use in variety of purposes. The most common tasks are classification and regression, as well as control, modeling and prediction. For more than three decades, the field of artificial neural networks has been the center of intensive research. A large number of software tools have been developed to train these types of networks, but there is still interest in implementing neural networks in different programming languages. This paper aims to present the implementation of an arbitrary neural network in the Java programming language.

Average number of fixed points and attractors in Hopfield neural networks

Calculating the exact number of fixed points and attractors of an arbitrary Hopfield neural network is a non-deterministic polynomial (NP)-hard problem. In this paper, we first calculate the average number of fixed points in such networks versus their size and threshold of neurons, in terms of a statistical method, which has been applied to the calculation of the average number of metastable states in spin glass systems. Then the same method is expanded to study the average number of attractors in such networks. The results of the calculation qualitatively agree well with the numerical calculation. The discrepancies between them are also well explained.

Triggering of high-speed neurite outgrowth using an optical microheater.

Optical microheating is a powerful non-invasive method for manipulating biological functions such as gene expression, muscle contraction, and cell excitation. Here, we demonstrate its potential usage for regulating neurite outgrowth. We found that optical microheating with a water-absorbable 1,455-nm laser beam triggers directional and explosive neurite outgrowth and branching in rat hippocampal neurons. The focused laser beam under a microscope rapidly increases the local temperature from 36 °C to 41 °C (stabilized within 2 s), resulting in the elongation of neurites by more than 10 μm within 1 min. This high-speed, persistent elongation of neurites was suppressed by inhibitors of both microtubule and actin polymerization, indicating that the thermosensitive dynamics of these cytoskeletons play crucial roles in this heat-induced neurite outgrowth. Furthermore, we showed that microheating induced the regrowth of injured neurites and the interconnection of neurites. These results demonstrate the efficacy of optical microheating methods for the construction of arbitrary neural networks.

On the stability of the Cartesian product of a neural ring and an arbitrary neural network

Abstract The stability of a system of neural networks connected to a ring has been studied extensively throughout recent years. Our main contribution within this work states that the stability region in the parameter space of a discrete-time model can be extended by breaking such a ring provided that there is a sufficiently large number of networks. Also, it has been shown that for a small ring, paradoxical values may appear within its parameter space for which such a ring is stable; meanwhile, corresponding linear configuration is unstable. MSC:37B25.

IDENTIFYING NEURAL NETWORK TOPOLOGIES THAT FOSTER DYNAMICAL COMPLEXITY

We use an ecosystem simulator capable of evolving arbitrary neural network topologies to explore the relationship between an information theoretic measure of the complexity of neural dynamics and several graph theoretical metrics calculated for the underlying network topologies. Evolutionary trends confirm and extend previous results demonstrating an evolutionary selection for complexity and small-world network properties during periods of behavioral adaptation. The resultant mapping of the space of network topologies occupied by the most complex networks yields new insights into the relationship between network structure and function. The highest complexity networks are found within limited numerical ranges of clustering coefficient, characteristic path length, small-world index, and global efficiency. The widths of these ranges vary from quite narrow to modest, and provide a guide to the most productive regions of the space of neural topologies in which to search for complexity. Our demonstration that evolution selects for complex dynamics and small-world networks helps explain biological evidence for these trends and provides evidence for selection of these characteristics based purely on network function—with no physical constraints on network structure—thus suggesting that functional and structural evolutionary pressures cooperate to produce brains optimized for adaptation to a complex, variable world.

Managing Burstiness and Scalability in Event-Driven Models on the SpiNNaker Neuromimetic System

Neural networks present a fundamentally different model of computation from the conventional sequential digital model, for which conventional hardware is typically poorly matched. However, a combination of model and scalability limitations has meant that neither dedicated neural chips nor FPGA’s have offered an entirely satisfactory solution. SpiNNaker introduces a different approach, the “neuromimetic” architecture, that maintains the neural optimisation of dedicated chips while offering FPGA-like universal configurability. This parallel multiprocessor employs an asynchronous event-driven model that uses interrupt-generating dedicated hardware on the chip to support real-time neural simulation. Nonetheless, event handling, particularly packet servicing, requires careful and innovative design in order to avoid local processor congestion and possible deadlock. We explore the impact that spatial locality, temporal causality and burstiness of traffic have on network performance, using tunable, biologically similar synthetic traffic patterns. Having established the viability of the system for real-time operation, we use two exemplar neural models to illustrate how to implement efficient event-handling service routines that mitigate the problem of burstiness in the traffic. Extending work published in ACM Computing Frontiers 2010 with on-chip testing, simulation results indicate the viability of SpiNNaker for large-scale neural modelling, while emphasizing the need for effective burst management and network mapping. Ultimately, the goal is the creation of a library-based development system that can translate a high-level neural model from any description environment into an efficient SpiNNaker instantiation. The complete system represents a general-purpose platform that can generate an arbitrary neural network and run it with hardware speed and scale.

Action control of autonomous agents in continuous valued space using RFCN

AbstractResearchers on action control of autonomous agents and multiple agents have attracted increasing attention in recent years. The general methods using action control of agents are neural network, genetic programming, and reinforcement learning. In this study, we use neural network for action control of autonomous agents. Our method determines the structure and parameter of neural network in evolution. We proposed Flexibly Connected Neural Network (FCN) previously as a method of constructing arbitrary neural networks with optimized structures and parameters to solve unknown problems. FCN was applied to action control of an autonomous agent and showed experimentally that it is effective for perceptual aliasing problems. All of the experiments of FCN, however, are only in grid space. In this paper, we propose a new method based on FCN which can decide correction action in real and continuous valued space. The proposed method, called Real‐valued FCN (RFCN), optimizes input–output functions of each unit, parameters of the input–output functions and speed of each unit. In order to examine its effectiveness, we applied the proposed method to action control of an autonomous agent to solve continuous‐valued maze problems. © 2008 Wiley Periodicals, Inc. Electron Comm Jpn, 91(2): 31–39, 2008; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/eej.10032

RFCNによる連続値空間上での自律エージェントの行動制御

Researchers on action control of autonomous agents and multiple agents have attracted increasing attention in recent years. The general methods using action control of agents are neural network, genetic programming, and reinforcement learning. In this study, we use neural network for action control of autonomous agents. Our method determines the structure and parameter of neural network in evolution. We proposed Flexibly Connected Neural Network (FCN) previously as a method of constructing arbitrary neural networks with optimized structures and parameters to solve unknown problems. FCN was applied to action control of an autonomous agent and showed experimentally that it is effective for perceptual aliasing problems. All of the experiments of FCN, however, are only in grid space. In this paper, we propose a new method based on FCN which can decide correction action in real and continuous valued space. The proposed method, called Real-valued FCN (RFCN), optimizes input–output functions of each unit, parameters of the input–output functions and speed of each unit. In order to examine its effectiveness, we applied the proposed method to action control of an autonomous agent to solve continuous-valued maze problems. © 2008 Wiley Periodicals, Inc. Electron Comm Jpn, 91(2): 31–39, 2008; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/eej.10032