Neural Network Weight Matrices Research Articles

A method for the dimensionality reduction of neural network weight matrices for natural language processing

The growth in the number of parameters of neural network models for language modeling makes the question of reducing the dimensionality of the matrices storing the models' weights increasingly relevant. The study proposes a novel method to reducing the dimensionality of training weight matrices and an algorithm called Convolutional Weights Compression. The essence of the method is to apply the convolution operation to the analysis of matrices of neural network training weights to reduce their dimensionality. В связи с ростом количества параметров нейросетевых моделей для обработки естественного языка все более актуальным становится вопрос о том, как уменьшить размерность матриц, в которых хранятся веса таких моделей. Исследование предлагает новый метод уменьшения размерности матриц обучающих весов и алгоритм сжатия под названием «сверточная компрессия весов». Суть метода заключается в применении операции свертки к анализу матриц обучающих весов нейронных сетей для уменьшения их размерности.

Boundary between noise and information applied to filtering neural network weight matrices.

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.

Random matrix analysis of deep neural network weight matrices.

Neural networks have been used successfully in a variety of fields, which has led to a great deal of interest in developing a theoretical understanding of how they store the information needed to perform a particular task. We study the weight matrices of trained deep neural networks using methods from random matrix theory (RMT) and show that the statistics of most of the singular values follow universal RMT predictions. This suggests that they are random and do not contain system specific information, which we investigate further by comparing the statistics of eigenvector entries to the universal Porter-Thomas distribution. We find that for most eigenvectors the hypothesis of randomness cannot be rejected, and that only eigenvectors belonging to the largest singular values deviate from the RMT prediction, indicating that they may encode learned information. In addition, a comparison with RMT predictions also allows to distinguish networks trained in different learning regimes-from lazy to rich learning. We analyze the spectral distribution of the large singular values using the Hill estimator and find that the distribution cannot in general be characterized by a tail index, i.e., is not of power-law type.

Adaptive neural network hierarchical sliding mode control for six degrees of freedom overhead crane

AbstractThis paper presents an effective control method for three‐dimensional (3D) overhead cranes with six degrees of freedom (DOF). Two payload swings and an axial payload oscillation should be minimized besides driving the bridge, trolley, and hoisting drum to bring the payload to the desired position in space. First, a novel 3D‐6DOF crane model is developed, where the sixth degree of freedom is axial cargo oscillation that has never been considered in previous studies. A controller is then designed using the hierarchical sliding mode control method. Moreover, a radial basis function neural network (RBFNN) is used to approximate the system's unknown dynamic model accurately. According to the Lyapunov principle, a control law and an updated law for the neural network's weight matrices are designed to ensure the stability of the closed‐loop system. Simulation results on Matlab software show the proposed approach's effectiveness, such as smaller swing, minor axial oscillation, and precise position as desired.

Compact and Computationally Efficient Representation of Deep Neural Networks.

At the core of any inference procedure, deep neural networks are dot product operations, which are the component that requires the highest computational resources. For instance, deep neural networks, such as VGG-16, require up to 15-G operations in order to perform the dot products present in a single forward pass, which results in significant energy consumption and thus limits their use in resource-limited environments, e.g., on embedded devices or smartphones. One common approach to reduce the complexity of the inference is to prune and quantize the weight matrices of the neural network. Usually, this results in matrices whose entropy values are low, as measured relative to the empirical probability mass distribution of its elements. In order to efficiently exploit such matrices, one usually relies on, inter alia, sparse matrix representations. However, most of these common matrix storage formats make strong statistical assumptions about the distribution of the elements; therefore, cannot efficiently represent the entire set of matrices that exhibit low-entropy statistics (thus, the entire set of compressed neural network weight matrices). In this paper, we address this issue and present new efficient representations for matrices with low-entropy statistics. Alike sparse matrix data structures, these formats exploit the statistical properties of the data in order to reduce the size and execution complexity. Moreover, we show that the proposed data structures can not only be regarded as a generalization of sparse formats but are also more energy and time efficient under practically relevant assumptions. Finally, we test the storage requirements and execution performance of the proposed formats on compressed neural networks and compare them to dense and sparse representations. We experimentally show that we are able to attain up to ×42 compression ratios, ×5 speed ups, and ×90 energy savings when we lossless convert the state-of-the-art networks, such as AlexNet, VGG-16, ResNet152, and DenseNet, into the new data structures and benchmark their respective dot product.

An FPGA-Based Hardware Emulator for Neuromorphic Chip With RRAM

Neuromorphic chip with RRAM devices has been demonstrated as a promising computing platform for neural network-based applications. By directly mapping the weight matrices of neural networks onto RRAM-based crossbar arrays, high energy, and area efficiency can be achieved. However, the design of an RRAM-based neuromorphic chip faces many constraints due to the variability and limitations of RRAM. Simulation and emulation can help in the design of a neuromorphic chip prior to fabrication. However, software-based chip simulation on CPU is slow, especially for large-scale network-on-chip (NoC)-based chip design. In this paper, we present a hardware emulator on field-programmable gate array (FPGA) for an RRAM-based neuromorphic chip. Our emulator supports the emulation of static and dynamic variation of the RRAM-based crossbars used in the neural cores of a neuromorphic chip. Furthermore, an NoC is also implemented on FPGA to emulate the communication between the neural cores. Using the emulator, we show that effects, such as RRAM write and read noise and stuck-at faults affect the accuracy of an application on a neuromorphic chip. We also demonstrate the utility of the emulator in investigating NoC topologies, routing buffer depths, and neural core mappings.

Sliding mode adaptive neural network control for hybrid visual servoing of underwater vehicles

In this paper, a hybrid visual servo (HVS) controller is proposed for underwater vehicles, in which a combination of the vehicle's 3-D Cartesian pose and the 2-D image coordinates of a single feature is exploited. A dynamic inversion-based sliding mode adaptive neural network control (DI-SMANNC) method is developed for tracking the HVS reference trajectory generated from a constant target pose. A single hidden-layer (SHL) feedforward neural network, in conjunction with an adaptive sliding mode controller, is utilized to compensate for dynamic uncertainties. The adaptation laws of neural network weight matrices and control gains are designed to ensure the asymptotical stability of tracking errors and the ultimate uniform boundedness (UUB) of neural network weight matrices. The main advantage of the proposed DI-SMANNC over conventional sliding model neural network controllers lies in the fact that the knowledge of the bounds on system uncertainties and neural approximation errors is not required to be previously known. Simulation results are presented to validate the effectiveness of the developed controller, especially the robustness with respect to dynamic modeling uncertainties and camera calibration errors.

Adaptive neural network visual servo control for dynamic positioning of underwater vehicles

Dynamic positioning is an important technique to ensure the capability of a next generation underwater vehicle to fulfill complex intervention and observation tasks. This work proposes an adaptive neural network image-based visual servo (IBVS) controller for positioning an underwater vehicle with respect to a fixed target in camera-in-hand configuration. The visual servoing error is defined by projecting the image error to the vehicle space using a pseudo inverse of the estimated Jacobian matrix. In the proposed dual-loop system architecture, the velocity command is designed to stabilize the visual servoing error in the kinematic loop, and the desired acceleration is derived to track the velocity command in the dynamic loop. A single hidden layer (SHL) feedforward neural network, in conjunction with a conventional proportional-integral (PI) controller, is applied to the cascaded systems to compensate for the uncertainties in both the Jacobian matrix and the vehicle dynamics. The thruster control signals are obtained by using the approximate dynamic inversion (ADI) of the vehicle. The uniform ultimate boundedness of the visual servoing error and the neural network weight matrices are proved by Lyapunov theory. Simulation results are provided to verify the effectiveness of the controller with two candidate methods of estimating the image Jacobian matrix.

Sliding mode control for a class of nonlinear systems based on robust adaptive neural network estimation

PurposeThe purpose of this paper is to present a sliding mode controller design method for a class of uncertain nonlinear systems with uncertainties and to demonstrate a recursive derivative estimation procedure for the derivatives of system outputs.Design/methodology/approachA recursive derivative estimation procedure for the derivatives of system outputs is demonstrated. Radial basis function (RBF) neural networks are used to approximate the uncertainties and filters are introduced to estimate the derivatives of system outputs step‐by‐step. The adaptive tuning rules of RBF neural network weight matrices are derived by the Lyapunov stability theorem, which guarantees filter errors and network weight errors are bounded and exponentially converge to a neighborhood of the origin globally. The sliding mode controller is designed based on the estimation for the derivatives of system outputs such that the sliding surface converges to zero and the system control input is bounded.FindingsThe sliding mode controller can make the system output track the desired output with arbitrarily small tracking error. The filter errors and network weight estimation errors can be made arbitrarily small, and all the system signals are bounded. The proposed method does not need the supper bounds of the unmatched uncertainties and their any order derivatives.Research limitations/implicationsThe system output and uncertainties are required to be sufficiently smooth in the proposed method. In practice, this condition is always satisfied generally.Practical implicationsThis paper contains very useful advice for researchers on the sliding mode control and the use of neural networks.Originality/valueThe paper presents a new sliding mode controller design method based on recursive derivative estimation of system outputs using neural networks. The paper is aimed at theoretical researchers, especially those who have interest in sliding mode control, neural networks, adaptive techniques, and recursive estimation.

Intelligent decision making in multi-agent robot soccer system through compounded artificial neural networks

This paper proposes a two-stage approach using artificial neural networks for the intelligent decision-making by the robots in a MiroSot small league. The first stage involves the use of an evolutionary algorithm for getting a rough estimate of the neural network weight matrices. The proposed approach is then generalized to the case of quick, intelligent and accurate decision-making in the case of a robot soccer system with robots utilizing the concept of compounded artificial neural networks. In the proposed approach a soccer field is divided into three zones so that the decision of the robots depends on the zone of the ball at any instant. The concept of a forward robot is also introduced in this paper to enhance the accuracy of the decision-making with the global strategy of advancing towards the goal area of the opponent for scoring a goal. Simulation results indicate that the proposed techniques are very effective in taking intelligent decision-making in a multi-agent robot soccer system in MiroSot small league as well as middle league.

Absolute exponential stability of a class of recurrent neural networks with multiple and variable delays

In this paper, we derive some new conditions for absolute exponential stability (AEST) of a class of recurrent neural networks with multiple and variable delays. By using the Holder's inequality and the Young's inequality to estimate the derivatives of the Lyapunov functionals, we are able to establish more general results than some existing ones. The first type of conditions established involves the convex combinations of column-sum and row-sum dominance of the neural network weight matrices, while the second type involves the p-norm of the weight matrices with p∈[1,+∞].

Transformations of sigma–pi nets: obtaining reflected functions by reflecting weight matrices

This paper presents a methodology that reflected functions by reflecting the weight matrices of an artificial neural network. One of the major problems with the connectionist approach is that trained neural networks can only associate fixed sets of input–output mappings. We provide a methodology which allows the post-trained net to associate different input–output mappings. The different mappings are reflected in a horizontal axis, reflected in a vertical axis and scaling of the initial mapping. The methodology does not train the net on the different mappings but it transforms the weight matrix of the neural network. This paper describes a novel way of utilising sigma–pi neural networks. Our new methodology manipulates sigma–pi unit's weight matrices which transform the unit's output. The weights are cast in a matrix formulation, and then transformations can be performed on the weight matrix of the sigma–pi net. To test the new methodology, the following three steps were carried out on a neural network: (1) the network was trained to perform a mapping function, f; (2) the weights of the network were transformed; and (3) the network was tested to evaluate whether it performs the reflection in the vertical axis, f ref−vert( x)= a− f( x). This reflects the function in one dimension. A reflection transformation was used to manipulate the network's weight matrices to obtain a reflection in the vertical axis. Note that the network was not trained to perform the reflection in the vertical axis. The transformation of the weight matrix transformed the function the output performs. This article explains the theory which enables us to perform transformations of sigma–pi networks and obtain reflections of the output by reflecting the weight matrices. These transforms empower the network to perform related mapping tasks once one mapping task has been learnt. This article explains how each transformation is performed and it considers whether a set of ‘standard’ transformations can indeed be derived.