Instant neural graphics primitives with a multiresolution hash encoding

In order to prevent the individual neural networks from becoming similar in the long learning period of negative correlation learning for designing neural network ensembles, two approaches were adopted in this paper. The first approach is to replace large neural networks with small neural networks in neural network ensembles. Samll neural networks would be more practical in the real applications when the capability is limited. The second approach is to introduce random separation learning in negative correlation learning for each small neural network. The idea of random separation learning is to let each individual neural network learn differently on the randomly separated subsets of the given training samples. It has been found that the small neural networks could easily become weak and different each other by negative correlation learning with random separation learning. After applying large number of small neural networks for neural network ensembles, two combination methods were used to generate the output of the neural network ensembles while their performance had been compared.

Knowledge distillation has been proven to be effective in model acceleration and compression. It transfers knowledge from a large neural network to a small one by using the large neural network predictions as targets of the small neural network. But this way ignores the knowledge inside the large neural networks, e.g., parameters. Our preliminary study as well as the recent success in pre-training suggests that transferring parameters are more effective in distilling knowledge. In this paper, we propose Weight Distillation to transfer the knowledge in parameters of a large neural network to a small neural network through a parameter generator. On the WMT16 En-Ro, NIST12 Zh-En, and WMT14 En-De machine translation tasks, our experiments show that weight distillation learns a small network that is 1.88~2.94x faster than the large network but with competitive BLEU performance. When fixing the size of small networks, weight distillation outperforms knowledge distillation by 0.51~1.82 BLEU points.

Rhythmic motor patterns such as walking, flying, and chewing form a large part of the behavioral repertoire of most animals. It is now clear that the fundamental characteristics of such motor patterns (their rhythmicity and the sequence of movements that make up the pattern) result from the activity of relatively small neural networks that produce appropriately ordered motor outputs in the absence of timed input from either the periphery (e.g., proprioceptive inputs) or the rest of the nervous system (e.g., descending pathways) (Delcomyn, 1980). Over the last twenty years the mechanisms underlying the ability of these central pattern generator (CPG) networks to inherently produce ordered rhythmic motor outputs has been intensively studied, and a fairly deep understanding of this process has been achieved for several invertebrate model systems.

Efficient machine learning based surrogate models for surface kinetics by approximating the rates of the rate-determining steps

Optical computing may one day provide fast and efficient calculation, with complex nonlinear interactions occurring at the speed of light. Here we use supercontinuum generation as an optical computing block within a neural network. Supercontinuum generation is an interesting candidate for optical computing because it can be adjusted to operate over a large range of nonlinearity in a single device by adjusting laser pulse parameters. Here, the optical block accepts inputs in the form of actuator settings for a chirped fiber Bragg grating pulse shaper, and outputs a measured supercontinuum spectrum. A small neural network before the optical block translates the calculation data into actuator settings, and another small network after the optical block translates the spectrum into the desired output. In this architecture, the optical block acts like a static, fixed-weight neural network, while the two translation networks are trained to adapt the optical block to the particular computational problem. As first demonstrations, we train the system to perform hand-writing identification, and use the supercontinuum as the decoder in an autoencoder network. As temperature-controlled pulse shaping is slow, we first emulate the supercontinuum with a neural network trained on spectra measured at randomized heater settings. The emulation network is frozen and inserted in place of the optical block for training of the translation layers on computer. The training data is converted to spectra with the optical device for retraining of the output translator, compensating for emulation errors.

Balanced ensemble learning was developed from negative correlation learning by shifting the learning targets. From the different learning behaviors in balance ensemble learning for the two structures of neural network ensembles on both low noisy data and high noisy data, a number of new discoveries are revealed in this paper. The first discovery is that the ensembles with small neural networks by balanced ensemble learning could perform as well as the ensembles with large neural networks by negative correlation learning. The second discovery is that there is seldom overfitting in balanced ensemble learning for the ensembles with small neural networks. In contrast, overfitting had been observed in balanced ensemble learning for the ensembles with large neural networks on both low noisy data and high noisy data. The third discovery is that both the large and the small mean squared errors could lead to overfitting. Overfitting rather than underfitting arising from the larger mean squared error might come out at a surprise. The explanations of such a rare phenomenon are presented in this paper.

Retrieving the layout of different cluttered indoor scenes from monocular images is a challenging task, using a smaller deep neural network than the existing proposals. Previous geometric solutions are prone to failure in the presence of cluttered scenes because they depend strongly on hand-engineering features and the expectation of the possibility of the vanishing points’ calculation. With the growth of neural networks, the geometric methods were either replaced or fused within the emerging area of deep learning. The more recent solutions rely on dense neural networks with additional adjustments, either by the calculation of the vanishing points, position based, or by layout ranking. All these methods presented valid solutions to this challenge with the flaw of being computationally demanding. Here, we present a more lightweight solution, running the segmentation on a smaller neural network and introducing a discriminative classifier for the posterior layout ranking and optimization. Our proposed method is evaluated by two standard dataset benchmarks, achieving near state of the art results even with a fraction of the required parameters than the available state of the art methods.

We introduce a framework for compactly representing light field content with the novel concept of neural subspaces. While the recently proposed neural light field representation achieves great compression results by encoding a light field into a single neural network, the unified design is not optimized for the composite structures exhibited in light fields. Moreover, encoding every part of the light field into one network is not ideal for applications that require rapid transmission and decoding. We recognize this problem's connection to subspace learning. We present a method that uses several small neural networks, specializing in learning the neural subspace for a particular light field segment. Moreover, we propose an adaptive weight sharing strategy among those small networks, improving parameter efficiency. In effect, this strategy enables a concerted way to track the similarity among nearby neural subspaces by leveraging the layered structure of neural networks. Furthermore, we develop a soft-classification technique to enhance the color prediction accuracy of neural representations. Our experimental results show that our method better reconstructs the light field than previous methods on various light field scenes. We further demonstrate its successful deployment on encoding light fields with irregular viewpoint layout and dynamic scene content.

Patch based denoising algorithms seek to approximate the conditional expectation of clean patches given their related noisy observations. In this note, we give a probabilistic account of how various algorithms approach this problem and in particular, we argue that small neural networks can denoise small-scale texture patterns almost as well as their large counterparts. The analysis further indicates that self-similarity and Bayesian approaches such as neural networks are complementary paradigms for patch denoising, which we illustrate with an algorithm that effectively complements BM3D with small neural networks, thereby outperforming BM3D with minor additional cost.

Deep learning for image-based detection of weeds from emergence to maturity in wheat fields

Multilayer neural networks are among the most powerful models in machine learning, yet the fundamental reasons for this success defy mathematical understanding. Learning a neural network requires optimizing a nonconvex high-dimensional objective (risk function), a problem that is usually attacked using stochastic gradient descent (SGD). Does SGD converge to a global optimum of the risk or only to a local optimum? In the former case, does this happen because local minima are absent or because SGD somehow avoids them? In the latter, why do local minima reached by SGD have good generalization properties? In this paper, we consider a simple case, namely two-layer neural networks, and prove that-in a suitable scaling limit-SGD dynamics is captured by a certain nonlinear partial differential equation (PDE) that we call distributional dynamics (DD). We then consider several specific examples and show how DD can be used to prove convergence of SGD to networks with nearly ideal generalization error. This description allows for "averaging out" some of the complexities of the landscape of neural networks and can be used to prove a general convergence result for noisy SGD.

Nowadays, research, modeling, simulation and realization of brain-like systems to reproduce brain behaviors have become urgent requirements. In this paper, neural bursting and synchronization are imitated by modeling two neural network models based on the Hopfield neural network (HNN). The first neural network model consists of four neurons, which correspond to realizing neural bursting firings. Theoretical analysis and numerical simulation show that the simple neural network can generate abundant bursting dynamics including multiple periodic bursting firings with different spikes per burst, multiple coexisting bursting firings, as well as multiple chaotic bursting firings with different amplitudes. The second neural network model simulates neural synchronization using a coupling neural network composed of two above small neural networks. The synchronization dynamics of the coupling neural network is theoretically proved based on the Lyapunov stability theory. Extensive simulation results show that the coupling neural network can produce different types of synchronous behaviors dependent on synaptic coupling strength, such as anti-phase bursting synchronization, anti-phase spiking synchronization, and complete bursting synchronization. Finally, two neural network circuits are designed and implemented to show the effectiveness and potential of the constructed neural networks.

Classification of 2-dimensional array patterns: Assembling many small neural networks is better than using a large one

In this research, the deep-learning optimizers Adagrad, AdaDelta, Adaptive Moment Estimation (Adam), and Stochastic Gradient Descent (SGD) were applied to the deep convolutional neural networks AlexNet, GoogLeNet, VGGNet, and ResNet that were trained to recognize weeds among alfalfa using photographic images taken at 200×200, 400×400, 600×600, and 800×800 pixels. An increase in the image sizes reduced the classification accuracy of all neural networks. The neural networks that were trained with images of 200×200 pixels resulted in better classification accuracy than the other image sizes investigated here. The optimizers AlexNet and GoogLeNet trained with AdaDelta and SGD outperformed the Adagrad and Adam optimizers; VGGNet trained with AdaDelta outperformed Adagrad, Adam, and SGD; and ResNet trained with AdaDelta and Adagrad outperformed the Adam and SGD optimizers. When the neural networks were trained with the best-performing input image size (200×200 pixels) and the best-performing deep learning optimizer, VGGNet was the most effective neural network, with high precision and recall values (≥0.99) when validation and testing datasets were used. Alternatively, ResNet was the least effective neural network in its ability to classify images containing weeds. However, there was no difference among the different neural networks in their ability to differentiate between broadleaf and grass weeds. The neural networks discussed herein may be used for scouting weed infestations in alfalfa and further integrated into the machine vision subsystem of smart sprayers for site-specific weed control.

This work presents a novel framework for understanding learning dynamics and scaling relations in neural networks. We show that certain measures on the spectrum of the empirical neural tangent kernel (NTK), specifically entropy and trace, provide insight into the representations learned by a neural network and how these can be improved through architecture scaling. These results are demonstrated first on test cases before being applied to more complex networks, including transformers, auto-encoders, graph neural networks, and reinforcement learning studies. In testing on a wide range of architectures, we highlight the universal nature of training dynamics and further discuss how it can be used to understand the mechanisms behind learning in neural networks. We identify two such dominant mechanisms present throughout machine learning training. The first, information compression, is seen through a reduction in the entropy of the NTK spectrum during training, and occurs predominantly in small neural networks. The second, coined structure formation, is seen through an increasing entropy and thus, the creation of structure in the neural network representations beyond the prior established by the network at initialization. Due to the ubiquity of the latter in deep neural network architectures and its flexibility in the creation of feature-rich representations, we argue that this network entropy evolution be considered the onset of a deep learning regime.