Matrix Vector Research Articles

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today's prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation though time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient, non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.

Lighter and faster simulations on domains with symmetries

A strategy to improve the performance and reduce the memory footprint of simulations on meshes with spatial reflection symmetries is presented in this work. By using an appropriate mirrored ordering of the unknowns, discrete partial differential operators are represented by matrices with a regular block structure that allows replacing the standard sparse matrix–vector product with a specialised version of the sparse matrix-matrix product, which has a significantly higher arithmetic intensity. Consequently, matrix multiplications are accelerated, whereas their memory footprint is reduced, making massive simulations more affordable. As an example of practical application, we consider the numerical simulation of turbulent incompressible flows using a low-dissipation discretisation on unstructured collocated grids. All the required matrices are classified into three sparsity patterns that correspond to the discrete Laplacian, gradient, and divergence operators. Therefore, the above-mentioned benefits of exploiting spatial reflection symmetries are tested for these three matrices on both CPU and GPU, showing up to 5.0x speed-ups and 8.0x memory savings. Finally, a roofline performance analysis of the symmetry-aware sparse matrix–vector product is presented.

Tail mean-variance portfolio selection with estimation risk

Tail Mean-Variance (TMV) has emerged from the actuarial community as a criterion for risk management and portfolio selection, with a focus on extreme losses. The existing literature on portfolio optimization under the TMV criterion relies on the plug-in approach that substitutes the unknown mean vector and covariance matrix of asset returns in the optimal portfolio weights with their sample counterparts. However, the plug-in method inevitably introduces estimation risk and usually leads to poor out-of-sample portfolio performance. To address this issue, we propose a combination of the plug-in and 1/N rules and optimize its expected out-of-sample performance. Our study is based on the Mean-Variance-Standard-deviation (MVS) performance measure, which encompasses the TMV, classical Mean-Variance, and Mean-Standard-Deviation (MStD) as special cases. The MStD criterion is particularly relevant to mean-risk portfolio selection when risk is measured by quantile-based risk measures. Our proposed combined portfolio consistently outperforms both the plug-in MVS and 1/N portfolios in simulated and real-world datasets.

Binarized neural network of diode array with high concordance to vector–matrix multiplication

In this study, a binarized neural network (BNN) of silicon diode arrays achieved vector–matrix multiplication (VMM) between the binarized weights and inputs in these arrays. The diodes that operate in a positive-feedback loop in their p+-n-p-n+ device structure possess steep switching and bistable characteristics with an extremely low subthreshold swing (below 1 mV) and a high current ratio (approximately 108). Moreover, the arrays show a self-rectifying functionality and an outstanding linearity by an R-squared value of 0.99986, which allows to compose a synaptic cell with a single diode. A 2 × 2 diode array can perform matrix multiply-accumulate operations for various binarized weight matrix cases with some input vectors, which is in high concordance with the VMM, owing to the high reliability and uniformity of the diodes. Moreover, the disturbance-free, nondestructive readout, and semi-permanent holding characteristics of the diode arrays support the feasibility of implementing the BNN.

An Efficient Limited Memory Multi-Step Quasi-Newton Method

This paper is dedicated to the development of a novel class of quasi-Newton techniques tailored to address computational challenges posed by memory constraints. Such methodologies are commonly referred to as “limited” memory methods. The method proposed herein showcases adaptability by introducing a customizable memory parameter governing the retention of historical data in constructing the Hessian estimate matrix at each iterative stage. The search directions generated through this novel approach are derived from a modified version closely resembling the full memory multi-step BFGS update, incorporating limited memory computation for a singular term to approximate matrix–vector multiplication. Results from numerical experiments, exploring various parameter configurations, substantiate the enhanced efficiency of the proposed algorithm within the realm of limited memory quasi-Newton methodologies category.

Incorporating STATCOM into a Hydro-Thermal Optimal Power Flow Algorithm

This paper presents the results of the inclusion of Synchronous Static Compensator (STATCOM) power flow models into a hydro-thermal optimal power flow (HTOPF) algorithm. STATCOM basically introduces the voltage magnitude and phase angle of the source converter into the algorithm. For each incorporated STATCOM, an augmented Lagrangian function was formed. The first and second derivatives of these functions were added to the gradient vector and Hessian matrix of an existing algorithm. The modified algorithm was implemented using MATLAB R2018a and tested on 30 and 57 bus systems. Two and three STATCOMs were, respectively, tested on 30 and 57 bus systems. The results obtained showed that one of the STATCOMs used on 30-bus system injected reactive power that ranges from 8.99 MVAR to 28.22 MVAR while the other one injected reactive power in the range of 8.20 MVAR to 33.20 MVAR. For the STATCOMs placed on the 57-bus system, the range of reactive power absorption by the one placed at bus 5 is 7.83 MVAR to 22.86 MVAR while the one at bus 55 absorbed from 5.06 MVAR to 12.92 MVAR. The STATCOM’s reactive power injection at bus 31 ranges from 4.63 MVAR to 10.35 MVAR. All the lower and higher values were obtained at hours 3 and 7, respectively. While the STATCOMs significantly improved the systems’ voltage profile, the impacts of STATCOM on the total systems daily energy loss, daily energy generations (from both plants), daily fuel cost and hydro plant water worth are insignificant.

Optimization design of configurable standard-type coils based on singular value decomposition for transcranial magnetic stimulation

Objective: This study aims to introduce a configurable standard-type (CST) coil design methodology that can accommodate different stimulation needs through the overlapping of standard coil components, while ensuring the performance of CST coil approaches the existing optimal physical limit as closely as possible. Approach: The study utilizes planar, spherical, and hemispherical stream function to represent the spatial geometry of the coils. Singular value decomposition is employed to design the standard coil components while minimizing the number of coil components. Initially, this study delineates the relationship between singular vectors and coil geometry, defining a multi-objective optimization problem for the research. By contrasting the decay characteristics of singular values in the inductance and electric field matrices of the stream functions, the singular vectors of the electric field matrix are utilized as the foundation for CST coil optimization. Subsequently, the relationships between the number of singular vectors and the coil’s Pareto fronts are determined through the optimization algorithms, and the geometric patterns of the singular vectors, i.e., the standard coil components, are analyzed. Main results: By constructing the CST coil components using seven singular vectors and comparing with the Pareto front of the performance of over 50 existing coil types, the stimulation depth range of the CST coil based on the planar stream function decreased by only 7.7%; the focality performance of the CST coil based on the spherical stream function improved by 22.4%; the focality performance of the CST coil based on the hemispherical stream function decreased by only 1.6%. Significance: The construction of CST coils with a minimal number of components is explored to approximate the performance at any discrete point on the physical limit curve. This approach overcomes the limitation of one coil associating with one performance point in traditional designs, presenting a novel perspective on coil design.

DESIGN NEURAL NETWORK-PID CONTROLLER FOR TRAJECTORY TRACKING OF DIFFERENTIAL DRIVE MOBILE ROBOT

This paper proposes the design of a neural network controller based on a sample controller for controlling the trajectory-tracking motion of a differential drive mobile robot (DDMR). Firstly, the trajectory tracking model for DDMR is established based on position error. Next, a perceptron neural network is designed with three hidden layers to control the trajectory tracking of DDMR. The backpropagation algorithm is used to train the neural network with training data obtained from the PID controller with time-varying parameters. The authors have developed this approach and experimentally verified it with minor tracking errors. The neural network's weight matrix (W) and bias vector (b) are updated in real-time, providing an advantage over other methods. The effectiveness of the proposed controller is demonstrated by the DDMR's NURBS trajectory tracking error, which does not exceed 2.17 cm, and the DDMR's motion error, with linear and angular velocities not exceeding 0.004 m/s and 0.0007 rad/s, respectively. The proposed controller can supplement traditional controllers in controlling the trajectory of autonomous mobile robots, thereby improving the ability to generate local trajectories to avoid dynamic obstacles by the neural network

High-order linearly implicit exponential integrators conserving quadratic invariants with application to scalar auxiliary variable approach

This paper proposes a framework for constructing high-order linearly implicit exponential integrators that conserve a quadratic invariant. This is then applied to the scalar auxiliary variable (SAV) approach. Quadratic invariants are significant objects that are present in various physical equations and also in computationally efficient conservative schemes for general invariants. For instance, the SAV approach converts the invariant into a quadratic form by introducing scalar auxiliary variables, which have been intensively studied in recent years. In this vein, Sato et al. (Appl. Numer. Math. 187, 71-88 2023) proposed high-order linearly implicit schemes that conserve a quadratic invariant. In this study, it is shown that their method can be effectively merged with the Lawson transformation, a technique commonly utilized in the construction of exponential integrators. It is also demonstrated that combining the constructed exponential integrators and the SAV approach yields schemes that are computationally less expensive. Specifically, the main part of the computational cost is the product of several matrix exponentials and vectors, which are parallelizable. Moreover, we conduct some mathematical analyses on the proposed schemes.

Enhancing security for physical layer communication in RIS-aided MIMO-NOMA systems in the presence of an eavesdropper

This paper investigates the potential of leveraging a reconfigurable intelligent surface (RIS) in the multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) wireless communication systems. In fact, NOMA improves spectral efficiency (SE) through power domain utilization, and RIS enhances the system performance by manipulating the propagation environment. In this paper, we investigate the physical layer security (PLS) of a RIS-aided MIMO-NOMA system serving two users in the presence of an eavesdropper (EV), where the channel state information (CSI) of the EV is unknown for the BS. To address the security challenge, we utilize secrecy outage probability (SOP) and minimize it by optimizing the RIS’s phase shift matrix, the beamforming vector, and the power allocation coefficients. Due to the non-convexity of the main optimization problem, we decompose the initial problem into three subproblems, and then, we propose an iterative approach to determine the unknown parameters alternately. The first subproblem involves obtaining the optimal beamforming vector and phase shift matrix using the semidefinite relaxation (SDR) method, which introduces high complexity. As an alternative, another algorithm is proposed based on the duality of the optimization problem, offering lower complexity and better accuracy for high-dimensional optimization problems. In this algorithm, the successive convex approximation (SCA) technique is employed to achieve the optimum phase shift matrix, and the stochastic gradient (SG) approach is used to determine the beamforming vector. Furthermore, the paper establishes the optimal values of power allocation coefficients by minimizing the difference between the SOPs of the two users. Finally, the simulation results demonstrate that the proposed power allocation coefficients scheme significantly enhances system performance compared to the existing fixed power allocation coefficients scheme.

A Spiking LSTM Accelerator for Automatic Speech Recognition Application Based on FPGA

Long Short-Term Memory (LSTM) finds extensive application in sequential learning tasks, notably in speech recognition. However, existing accelerators tailored for traditional LSTM networks grapple with high power consumption, primarily due to the intensive matrix–vector multiplication operations inherent to LSTM networks. In contrast, the spiking LSTM network has been designed to avoid these multiplication operations by replacing multiplication and nonlinear functions with addition and comparison. In this paper, we present an FPGA-based accelerator specifically designed for spiking LSTM networks. Firstly, we employ a low-cost circuit in the LSTM gate to significantly reduce power consumption and hardware cost. Secondly, we propose a serial–parallel processing architecture along with hardware implementation to reduce inference latency. Thirdly, we quantize and efficiently deploy the synapses of the spiking LSTM network. The power consumption of the accelerator implemented on Artix-7 and Zynq-7000 is only about 1.1 W and 0.84 W, respectively, when performing the inference for speech recognition with the Free Spoken Digit Dataset (FSDD). Additionally, the energy consumed per inference is remarkably efficient, with values of 87 µJ and 66 µJ, respectively. In comparison with dedicated accelerators designed for traditional LSTM networks, our spiking LSTM accelerator achieves a remarkable reduction in power consumption, amounting to orders of magnitude.

An Algorithm Based on Non-Negative Matrix Factorization for Detecting Communities in Networks

Community structure is a significant characteristic of complex networks, and community detection has valuable applications in network structure analysis. Non-negative matrix factorization (NMF) is a key set of algorithms used to solve the community detection issue. Nevertheless, the localization of feature vectors in the adjacency matrix, which represents the characteristics of complex network structures, frequently leads to the failure of NMF-based approaches when the data matrix has a low density. This paper presents a novel algorithm for detecting sparse network communities using non-negative matrix factorization (NMF). The algorithm utilizes local feature vectors to represent the original network topological features and learns regularization matrices. The resulting feature matrices effectively reveal the global structure of the data matrix, demonstrating enhanced feature expression capabilities. The regularized data matrix resolves the issue of localized feature vectors caused by sparsity or noise, in contrast to the adjacency matrix. The approach has superior accuracy in detecting community structures compared to standard NMF-based community detection algorithms, as evidenced by experimental findings on both simulated and real-world networks.

A Novel Higher-Order Numerical Scheme for System of Nonlinear Load Flow Equations

Power flow problems can be solved in a variety of ways by using the Newton–Raphson approach. The nonlinear power flow equations depend upon voltages Vi and phase angle δ. An electrical power system is obtained by taking the partial derivatives of load flow equations which contain active and reactive powers. In this paper, we present an efficient seventh-order iterative scheme to obtain the solutions of nonlinear system of equations, with only three steps in its formulation. Then, we illustrate the computational cost for different operations such as matrix–matrix multiplication, matrix–vector multiplication, and LU-decomposition, which is then used to calculate the cost of our proposed method and is compared with the cost of already seventh-order methods. Furthermore, we elucidate the applicability of our newly developed scheme in an electrical power system. The two-bus, three-bus, and four-bus power flow problems are then solved by using load flow equations that describe the applicability of the new schemes.

Inverse-designed low-index-contrast structures on a silicon photonics platform for vector–matrix multiplication

Inverse-designed low-index-contrast structures on a silicon photonics platform for vector–matrix multiplication

Small Sample Fiber Full State Diagnosis Based on Fuzzy Clustering and Improved ResNet Network

The optical time domain reflectometer (OTDR) curve features of communication fibers exhibit subtle differences among their normal, subhealthy, and faulty operating states, making it challenging for existing machine learning-based fault diagnosis algorithms to extract these minute features. In addition, the OTDR curve field fault data are scarce, and data-driven deep neural network that needs a lot of data training cannot meet the requirements. In response to this issue, this paper proposes a communication fiber state diagnosis model based on fuzzy clustering and an improved ResNet. First, the pretrained residual network (ResNet) is modified by removing the classification layer and retaining the feature extraction layers. A global average pooling (GAP) layer is designed as a replacement for the fully connected layer. Second, fuzzy clustering, instead of the softmax classification layer, is employed in ResNet for its characteristic of requiring no subsequent data training. The improved model requires only a small amount of sample training to optimize the parameters of the GAP layer, thereby accommodating state diagnosis in scenarios with limited data availability. During the diagnosis process, the OTDR curves are input into the network, resulting in 512 features outputted in the GAP layer. These features are used to construct a feature vector matrix, and a dynamic clustering graph is formed using fuzzy clustering to realize the fiber state diagnosis. Through on-site data detection and validation, it has been demonstrated that the improved ResNet can effectively identify the full cycle of fiber states.

Frictional contact multi-point constraint in two dimensions

Node-to-Segment slideline frictional contact formulation as one of the simplest penalty-based contact algorithms could be further simplified to a general Multi-Point Constraint formulation. This effective MPC formulation is proposed in this paper.Two case specific and a general MPC formulations are proposed which uses the distance vector of the master boundaries with respect to a unique point for the calculation of the contact interaction. Accordingly, the general master surface is reduced to a control node and its MPC distance vector and MPC normal matrix. As a result, the numerous expressions for the contact stiffness matrix will be reduced to its first simple expression, exclusively.Static and dynamic examples are solved to illustrate that the MPC algorithm proposed maintains the same accuracy of Node-to-Segment algorithms while presents the efficiency and user-friendly properties of the multiple-point constraint algorithms used in advanced finite element programs.

Extending matrix–vector framework on multiple relaxation time lattice Boltzmann method

This paper introduces an extension of a fully discrete matrix–vector form (MVF) for the lattice Boltzmann method (LBM) to handle the multiple relaxation time parameter (MRT) LBM. The proposed approach offers a more efficient and practical framework for simulating fluid flows, with the added benefit of being able to handle complex geometries using the image-based ghost (IBG) method. The advantages and limitations of this approach are discussed, including its simplicity in linear algebraic extension, parallel computing capability, computational speed, and memory usage. The results of numerical experiments demonstrate the improved computational efficiency of the proposed method, highlighting its potential for future applications in various fields of computational fluid dynamics.

High-precision Solution of Monostatic Radar Cross Section based on Compressive Sensing and QR Decomposition Techniques

In solving the monostatic electromagnetic scattering problem, the traditional improved primary characteristic basis function method (IPCBFM) often encounters difficulties in constructing the reduced matrix due to the long computation time and low accuracy. Therefore, a new method combining the compressed sensing (CS) technique with IPCBFM is proposed and applied to solve the monostatic electromagnetic scattering problem. The proposed method utilizes the characteristic basis functions (CBFs) generated by the IPCBFM to achieve a sparse transformation of the surface-induced currents. Several rows in the impedance matrix and excitation vector are selected as the observation matrix and observation vector. The QR decomposition is adopted as the recovery algorithm to realize the recovery of surface-induced currents. Numerical simulations are performed for cylinder, cube, and almond models, and the results show that the new method has higher solution accuracy, shorter computation time, and stronger solution stability than the traditional IPCBFM. It is worth mentioning that the new method reduces the recovery matrix size and the number of CBFs quantitatively, and provides a novel solution for solving monostatic RCS of complex targets.

Compact, efficient, and scalable nanobeam core for photonic matrix-vector multiplication

Optical neural networks have emerged as a promising avenue for implementing artificial intelligence applications, with matrix computations being a crucial component. However, the existing implementations based on microring resonators (MRRs) face bottlenecks in integration, power efficiency, and scalability, hindering the practical applications of wavelength division multiplexing (WDM)-based matrix-vector multiplications at the hardware level. Here we present a photonic crystal nanobeam cavity (PCNC) matrix core. Remarkably compact with dimensions reduced to 20µm×0.5µm, the PCNC unit exhibits a thermal tuning efficiency more than three times that of MRRs. Crucially, it is immune to the free spectral range constraint, thus able to harness the wealth of independent wavelength channels provided by WDM. A 3×3 PCNC core chip is demonstrated for animal face recognition and a six-channel chip is employed for handwritten digit classification to demonstrate the scalability. The PCNC solution holds immense promise, offering a versatile platform for next-generation photonic artificial intelligence chips.

Efficient FPGA Implementation of Convolutional Neural Networks and Long Short-Term Memory for Radar Emitter Signal Recognition

In recent years, radar emitter signal recognition has enjoyed a wide range of applications in electronic support measure systems and communication security. More and more deep learning algorithms have been used to improve the recognition accuracy of radar emitter signals. However, complex deep learning algorithms and data preprocessing operations have a huge demand for computing power, which cannot meet the requirements of low power consumption and high real-time processing scenarios. Therefore, many research works have remained in the experimental stage and cannot be actually implemented. To tackle this problem, this paper proposes a resource reuse computing acceleration platform based on field programmable gate arrays (FPGA), and implements a one-dimensional (1D) convolutional neural network (CNN) and long short-term memory (LSTM) neural network (NN) model for radar emitter signal recognition, directly targeting the intermediate frequency (IF) data of radar emitter signal for classification and recognition. The implementation of the 1D-CNN-LSTM neural network on FPGA is realized by multiplexing the same systolic array to accomplish the parallel acceleration of 1D convolution and matrix vector multiplication operations. We implemented our network on Xilinx XCKU040 to evaluate the effectiveness of our proposed solution. Our experiments show that the system can achieve 7.34 giga operations per second (GOPS) data throughput with only 5.022 W power consumption when the radar emitter signal recognition rate is 96.53%, which greatly improves the energy efficiency ratio and real-time performance of the radar emitter recognition system.