Matrix Vector Research Articles

A joint-norm distance metric 2DPCA for robust dimensionality reduction

Two-dimensional principal component analysis (2DPCA) is one of the most representative dimensionality reduction methods and the robustness degrades in the presence of noise. To improve the robustness, several 2DPCA methods using similar distance metrics were proposed to achieve the objective of the maximum projection distance or the minimum reconstruction error. However, these two objectives cannot be accomplished simultaneously or are constrained by the same projection matrix. To handle these problems, we propose a generalized robust 2DPCA method called 2DPCA-2-Lp, which joins 2-norm and lp-norm metrics in the objective function. It aims to maximize the ratio of projected vectors to image row vectors and to fulfill the dual objectives of directly maximizing projection distances and indirectly minimizing reconstruction errors. 2DPCA-2-Lp measures the distances between row vectors of matrices rather than the distances between matrices, thus remarkably enhancing the robustness. Furthermore, the relationship between reconstruction errors and projection distances for different parameter values is analyzed theoretically under the joint-norm metric. Then, the closed greedy iterative algorithm is developed for obtaining optimal solutions. Finally, extensive experimental results on four datasets show that 2DPCA-2-Lp outperforms most existing 2DPCA methods in terms of reconstruction and classification performances and has better robustness against noise.

Automatic stabilization of finite-element simulations using neural networks and hierarchical matrices

Petrov–Galerkin formulations with optimal test functions allow for the stabilization of finite element simulations. In particular, given a discrete trial space, the optimal test space induces a numerical scheme delivering the best approximation in terms of a problem-dependent energy norm. This ideal approach has two shortcomings: first, we need to explicitly know the set of optimal test functions; and second, the optimal test functions may have large supports inducing expensive dense linear systems. A concise proposal on how to overcome these shortcomings has been raised during the last decade by the Discontinuous Petrov–Galerkin (DPG) methodology. However, DPG has also some limitations and difficulties: the method requires ultraweak variational formulations, obtained through a hybridization process, which is not trivial to implement at the discrete level.Our motivation is to offer a simpler alternative for the case of parametric PDEs, which can be used with any variational formulation. Indeed, parametric families of PDEs are an example where it is worth investing some (offline) computational effort to obtain stabilized linear systems that can be solved efficiently in an online stage, for a given range of parameters. Therefore, as a remedy for the first shortcoming, we explicitly compute (offline) a function mapping any PDE parameter, to the matrix of coefficients of optimal test functions (in some basis expansion) associated with that PDE parameter. Next, as a remedy for the second shortcoming, we use the low-rank approximation to hierarchically compress the (non-square) matrix of coefficients of optimal test functions. In order to accelerate this process, we train a neural network to learn a critical bottleneck of the compression algorithm (for a given set of PDE parameters). When solving online the resulting (compressed) Petrov–Galerkin formulation, we employ a GMRES iterative solver with inexpensive matrix–vector multiplications thanks to the low-rank features of the compressed matrix. We perform experiments showing that the full online procedure is as fast as an (unstable) Galerkin approach. We illustrate our findings by means of 2D–3D Eriksson–Johnson problems, together with 2D Helmholtz equation.

Orthogonal Inductive Matrix Completion.

We propose orthogonal inductive matrix completion (OMIC), an interpretable approach to matrix completion based on a sum of multiple orthonormal side information terms, together with nuclear-norm regularization. The approach allows us to inject prior knowledge about the singular vectors of the ground-truth matrix. We optimize the approach by a provably converging algorithm, which optimizes all components of the model simultaneously. We study the generalization capabilities of our method in both the distribution-free setting and in the case where the sampling distribution admits uniform marginals, yielding learning guarantees that improve with the quality of the injected knowledge in both cases. As particular cases of our framework, we present models that can incorporate user and item biases or community information in a joint and additive fashion. We analyze the performance of OMIC on several synthetic and real datasets. On synthetic datasets with a sliding scale of user bias relevance, we show that OMIC better adapts to different regimes than other methods. On real-life datasets containing user/items recommendations and relevant side information, we find that OMIC surpasses the state of the art, with the added benefit of greater interpretability.

Performance Portable Batched Sparse Linear Solvers

Solving large number of small linear systems is increasingly becoming a bottleneck in computational science applications. While dense linear solvers for such systems have been studied before, batched sparse linear solvers are just starting to emerge. In this paper, we discuss algorithms for solving batched sparse linear systems and their implementation in the Kokkos Kernels library. The new algorithms are performance portable and map well to the hierarchical parallelism available in modern accelerator architectures. The sparse matrix vector product (SPMV) kernel is the main performance bottleneck of the Krylov solvers we implement in this work. The implementation of the batched SPMV and its performance are therefore discussed thoroughly in this paper. The implemented kernels are tested on different Central Processing Unit (CPU) and Graphic Processing Unit (GPU) architectures. We also develop batched Conjugate Gradient (CG) and batched Generalized Minimum Residual (GMRES) solvers using the batched SPMV. Our proposed solver was able to solve 20,000 sparse linear systems on V100 GPUs with a mean speedup of 76x and 924x compared to using a parallel sparse solver with a block diagonal system with all the small linear systems, and compared to solving the small systems one at a time, respectively. We see mean speedup of 0.51 compared to dense batched solver of cuSOLVER on V100, while using lot less memory. Thorough performance evaluation on three different architectures and analysis of the performance are presented.

Low-Rank Gradient Descent for Memory-Efficient Training of Deep In-Memory Arrays

The movement of large quantities of data during the training of a deep neural network presents immense challenges for machine learning workloads, especially those based on future functional memories deployed to store network models. As the size of network models begins to vastly outstrip traditional silicon computing resources, functional memories based on flash, resistive switches, magnetic tunnel junctions, and other technologies can store these new ultra-large models. However, new approaches are then needed to minimize hardware overhead, especially on the movement and calculation of gradient information that cannot be efficiently contained in these new memory resources. To do this, we introduce streaming batch principal component analysis (SBPCA) as an update algorithm. Streaming batch principal component analysis uses stochastic power iterations to generate a stochastic rank- k approximation of the network gradient. We demonstrate that the low-rank updates produced by streaming batch principal component analysis can effectively train convolutional neural networks on a variety of common datasets, with performance comparable to standard mini-batch gradient descent. Our approximation is made in an expanded vector form that can efficiently be applied to the rows and columns of crossbars for array-level updates. These results promise improvements in the design of application-specific integrated circuits based around large vector-matrix multiplier memories.

Secrecy Capacity Enhancement in Active IRS-Assisted UAV Communication System.

As a new technology for reconstructing communication environments, intelligent reflecting surfaces (IRSs) can be applied to UAV communication systems. However, some challenges exist in IRS-assisted UAV communication system design, such as physical layer security issues, IRS design, and power consumption issues owing to the limitation of the hardware. Therefore, a secrecy capacity optimization scheme for an active IRS-assisted unmanned aerial vehicle (UAV) communication system is proposed to solve multi-user security issues. In particular, controllable power amplifiers are integrated into reflecting units to solve the problem of blocked links, and the UAV can dynamically select the served user according to the channel quality. In order to maximize the system average achievable secrecy capacity and ensure the power constraints of the UAV and active IRS, user scheduling, UAV trajectory, beamforming vector, and reflection matrix are jointly optimized, and the block coordinate descent (BCD) algorithm is applied to solve this non-convex problem. Simulation results show that the active IRS-assisted UAV communication scheme can significantly weaken the "multiplicative fading" effect and enhance the system secrecy capacity by 55.4% and 11.9% compared with the schemes with passive IRS and without optimal trajectory, respectively.

From technology opportunities to ideas generation via cross-cutting patent analysis: Application of generative topographic mapping and link prediction

Technology opportunity analysis (TOA) with ideas generation has been recognized an important activity to remain competitive and lead the industry in the future. However, there are several issues with existing TOA, such as an unclear path from technology opportunities to ideas generation, a fuzzy integration between automated TOA techniques and expert-based methods, and a lack of detailed schemes for technology opportunities. This study proposes a new systematic approach to show the way from technology opportunities to ideas generation via cross-cutting patent analysis. The proposed approach is comprised of three stages: 1) establishing a cross-cutting relationship between the target and reference technologies through the results of F-terms; 2) collecting and processing patents to construct patent-keyword vector matrices of the target and reference technologies, respectively; and 3) migrating corresponding ideas via cosine similarity and link prediction for the target technology opportunities that are discovered based on generative topographic mapping (GTM). The feasibility and effectiveness of the proposed approach is demonstrated by empirical research on the exploitation technology in both the natural gas hydrate (NGH) and the coal bed methane (CBM) fields. This study represents a contribution to expand the existing TOA research into generating creative ideas by providing more detailed schemes for technology opportunities.

Editorial for the special issue “Control‐theoretic approaches for systems in the life sciences”

Editorial for the special issue “Control<scp>‐theoretic</scp> approaches for systems in the life sciences”

APPLICATION OF THE QUASI-NEWTON IN INTERIOR POINT METHODS FOR SOLVING THE PREDISPATCH PROBLEM

Brazilian energy matrix is essentially based on hydroelectricity with long transmission lines, allowing the exchange of energy produced in all regions of the country, the increased demand for energy and the search for lower costs, the application of more efficient and robust methods to minimize generation and transmission losses is necessary, since these are functions of generated and transmitted power, respectively. The purpose of this work is to implement primal dual interior point method for the predispatchof a hydroelectric system, with partial replacement of the Newton method with the quasi-Newton method in order to compute the system Jacobian matrix and reduce the computational costs of the iterations arising from approximations of the inverses of the Hessian matrix. This means that, in order to obtain a search direction, only a matrix vector product is necessary, which is much more efficient, for example, than the Newton method, in which a linear system has to be solved at each iteration. Computational results prove the efficiency of the approach used.

Closed-form derivation of aerodynamic damping matrix and pitch vector of an aero-servo-elastic wind turbine system

This paper develops closed-form expressions of aerodynamic damping matrix and pitch controller of an aero-servo-elastic wind turbine (WT) system. This facilitates establishing a linearized WT model completely analytically at an arbitrary operational point, without resorting to extensive numerical linearizations or fittings. The rotor blade flexibility and pitch controller are included in the model development, thus unfolding a full understanding of the aerodynamic coupling effects. The linearized WT model is verified by good agreement with results from the original nonlinear WT model. Based on the verified linearized model, it is seen that aerodynamic coupling between tower fore–aft and side–side vibrations is solely due to the tower top rotations, and the top rotations also significantly enlarge the diagonal aerodynamic damping coefficients. Both tower top rotations and flexible blade–tower interaction play important roles in tower aerodynamic damping ratios. The developed formulas of load coefficients are also readily applicable to a FE modeling framework of the tower–foundation system.

Analytical solution for laterally loaded non-uniform circular piles in multi-layered inhomogeneous soil

ABSTRACT Non-prismatic piles are typically used in cases where large lateral loads must be resisted. In many applications, piles are partially or fully embedded in multi-layered non-homogeneous soil, with each layer having its own set of properties. Analytical, simple solutions to study this problem are more limited and complex than that of prismatic ones. The analysis becomes even more complicated when both the variation of the cross-sectional area of the element and the soil inhomogeneity are included in the formulation. This work presents the derivation of the stiffness matrix and load vector of a non-uniform section of pile partially or fully embedded in non-homogeneous soil. The analysis of non-uniform piles in multi-layered soil is carried out by dividing the pile into multiple sub-elements and then assembling them using conventional matrix methods. Four examples, encompassing partially and fully embedded piles, are presented to validate the simplicity and accuracy of the proposed solution.

A linear finite element for timber–concrete layered beams with interlayer slip

In this paper, a beam theory for linear two-layer beams is presented in the framework of the Finite Element (FE) method. In particular, the beam theory aims to represent the static and dynamic response of timber–concrete layered structures, in which a concrete slab is supported by timber girders. Since the timber layer has a rather low shear stiffness, a shear-flexible formulation of the displacement field is adopted. In general, it cannot be assumed that the layers of such members are perfectly bonded. Therefore, the relative interlayer displacement in the longitudinal direction is considered, which is determined by a linear slip modulus in a smeared approach. Based on these assumptions, the stiffness matrix, mass matrix and load vector of a finite beam element are derived. Modal analyses on a single-span beam with clamped, free and hinged supports are performed and compared to the outcomes of 3D FE simulations to verify the applicability of the proposed beam theory over a wide frequency range. Static analyses on two structural systems illustrate the accuracy of the beam deflection and the interlayer slip, as well as the realistic assessment of normal and shear stresses in the two layers.

Dammann gratings-based truly parallel optical matrix multiplication accelerator.

Matrix multiplication (MM) is a fundamental operation in various scientific and engineering computations, as well as in artificial intelligence algorithms. Efficient implementation of MM is crucial for speeding up numerous applications. Photonics presents an opportunity for efficient acceleration of dense matrix computation, owing to its intrinsic advantages, such as huge parallelism, low latency, and low power consumption. However, most optical matrix computing architectures have been limited to realizing single-channel vector-matrix multiplication or using complex configurations to expand the number of channels, which does not fully exploit the parallelism of optics. In this study, we propose a novel, to the best of our knowledge, scheme for the implementation of large-scale two-dimensional optical MM with truly massive parallelism based on a specially designed Dammann grating. We demonstrate a sequence of MMs of 50 pairs of randomly generated 4 × 8 and 8 × 4 matrices in our proof-of-principle experiment. The results indicate that the mean relative error is approximately 0.048, thereby demonstrating optical robustness and high accuracy.

Characterizing the performance of node-aware strategies for irregular point-to-point communication on heterogeneous architectures

Supercomputer architectures are trending toward higher computational throughput due to the inclusion of heterogeneous compute nodes. These multi-GPU nodes increase on-node computational efficiency, while also increasing the amount of data to be communicated and the number of potential data flow paths. In this work, we characterize the performance of irregular point-to-point communication with MPI on heterogeneous compute environments through performance modeling, demonstrating the limitations of standard communication strategies for both device-aware and staging-through-host communication techniques. Presented models suggest staging communicated data through host processes then using node-aware communication strategies for high inter-node message counts. Notably, the models also predict that node-aware communication utilizing all available CPU cores to communicate inter-node data leads to the most performant strategy when communicating with a high number of nodes. Model validation is provided via a case study of irregular point-to-point communication patterns in distributed sparse matrix–vector products. Importantly, we include a discussion on the implications model predictions have on communication strategy design for emerging supercomputer architectures.

Parameterized complexity of categorical clustering with size constraints

In the Categorical Clustering problem, we are given a set of vectors (matrix) A={a1,…,an} over Σm, where Σ is a finite alphabet, and integers k and B. The task is to partition A into k clusters such that the median objective of the clustering in the Hamming norm is at most B. Fomin, Golovach, and Panolan [ICALP 2018] proved that the problem is fixed-parameter tractable for the binary case Σ={0,1}. We extend this algorithmic result to a popular capacitated clustering model, where in addition the sizes of the clusters are lower and upper bounded by integer parameters p and q, respectively. Our main theorem is that the problem is solvable in time 2O(Blog⁡B)|Σ|B⋅(mn)O(1).

A new iterative initialization of EM algorithm for Gaussian mixture models.

The expectation maximization (EM) algorithm is a common tool for estimating the parameters of Gaussian mixture models (GMM). However, it is highly sensitive to initial value and easily gets trapped in a local optimum. To address these problems, a new iterative method of EM initialization (MRIPEM) is proposed in this paper. It incorporates the ideas of multiple restarts, iterations and clustering. In particular, the mean vector and covariance matrix of sample are calculated as the initial values of the iteration. Then, the optimal feature vector is selected from the candidate feature vectors by the maximum Mahalanobis distance as a new partition vector for clustering. The parameter values are renewed continuously according to the clustering results. To verify the applicability of the MRIPEM, we compared it with other two popular initialization methods on simulated and real datasets, respectively. The comparison results of the three stochastic algorithms indicate that MRIPEM algorithm is comparable in relatively high dimensions and high overlaps and significantly better in low dimensions and low overlaps.

CUDA-accelerated protein electrostatics in linear space

Protein interactions are a key for understanding biological function and processes. Among interaction terms, molecular electrostatics is the dominant influencing factor on the long-range visibility of interaction partners. However, accurate electrostatic models are often computationally expensive or require unsatisfiable amounts of memory. Here we present implicit, yet exact representations for the notoriously dense and asymmetric system matrices of boundary element methods (BEM), exemplified on the case of nonlocal protein electrostatics. Our representations only require a small and constant number of floating-point values per surface element and, thus, yield BEM systems for protein surface meshes with hundreds of millions of elements even in strictly memory-limited environments such as hardware accelerators. We provide reference implementations for the Julia and Impala programming languages, including CUDA-accelerated matrix–vector products for our implicit systems. Additionally, we provide CUDA-accelerated BEM solvers for local and nonlocal protein electrostatics that can process real-world biomolecular systems previously inaccessible with exact BEM systems in a matter of minutes.

Efficient grouping approach for fault tolerant weight mapping in memristive crossbar array

The ability of resistive memory (ReRAM) to naturally conduct vector–matrix multiplication (VMM), which is the primary operation carried out during the training and inference of neural networks, has caught the interest of researchers. The memristor crossbar is one of the desirable architectures to perform VMM because it offers various benefits over other memory technologies, including in-memory computing, low power, and high density. Direct downloading and chip-on-the-loop approaches are typically used to train ReRAM-based neural networks. In these methods, all weight computations are carried out by a host machine, and the computed weights are downloaded in the crossbar. It has been seen that the network does not deliver the same precision as promised by the host system once the weights have been downloaded. This is because crossbars contain a significant number of faulty memristors and suffer from cell resistance variations because of immature manufacturing technologies. As a result, a cell may not be able to take the exact weight values that the host system generates, and may lead to incorrect inferences. Existing techniques for fault-tolerant mapping either involve network retraining or employ a graph-matching strategy that comes with hardware, power, and latency overheads. In this paper, we propose a mapping method to tolerate the effect of defective memristors. In order to lessen the impact of faulty memristors, the mapping is done in a way that allows network weights to cover up faulty memristors. Further, this work prioritizes the different faults based on the frequency of occurrence. The mapping efficiency is found to increase significantly with low power, area and latency overheads in the proposed approach. Experimental analyses show considerable improvement as compared to state-of-the-art works.

Derivative-free separable quadratic modeling and cubic regularization for unconstrained optimization

We present a derivative-free separable quadratic modeling and cubic regularization technique for solving smooth unconstrained minimization problems. The derivative-free approach is mainly concerned with building a quadratic model that could be generated by numerical interpolation or using a minimum Frobenius norm approach, when the number of points available does not allow to build a complete quadratic model. This model plays a key role to generate an approximated gradient vector and Hessian matrix of the objective function at every iteration. We add a specialized cubic regularization strategy to minimize the quadratic model at each iteration, that makes use of separability. We discuss convergence results, including worst case complexity, of the proposed schemes to first-order stationary points. Some preliminary numerical results are presented to illustrate the robustness of the specialized separable cubic algorithm.

Recursive computation of the multipole expansions of layer potential integrals over simplices for efficient fast multipole accelerated boundary elements

In boundary element methods (BEM) in R3, matrix elements and right hand sides are typically computed via analytical or numerical quadrature of the layer potential multiplied by some function over line, triangle and tetrahedral volume elements. When the problem size gets large, the resulting linear systems are often solved iteratively via Krylov subspace methods, with fast multipole methods (FMM) used to accelerate the matrix vector products needed. When FMM acceleration is used, most entries of the matrix never need be computed explicitly - they are only needed in terms of their contribution to the multipole expansion coefficients. We propose a new fast method - Quadrature to Expansion (Q2X) - for the analytical generation of the multipole expansion coefficients produced by the integral expressions for single and double layers on surface triangles; charge distributions over line segments and over tetrahedra in the volume; so that the overall method is well integrated into the FMM, with controlled error. The method is based on the O(1) per moment cost recursive computation of the moments. The method is developed for boundary element methods involving the Laplace Green's function in R3. The derived recursions are first compared against classical quadrature algorithms, and then integrated into FMM accelerated boundary element and vortex element methods. Numerical tests are presented and discussed.