Block Circulant Research Articles

Exploring GPU acceleration of Deep Neural Networks using Block Circulant Matrices

Training a Deep Neural Network (DNN) is a significant computing task since it places high demands on computing resources and memory bandwidth. Many approaches have been proposed to compress the network, while maintaining high model accuracy, reducing the computational demands associated with large-scale DNN training. One attractive approach is to leverage Block Circulant Matrices (BCM), compressing the linear transformation layers, e.g., convolutional and fully-connected layers, that heavily rely on performing General Matrix Multiplications (GEMM). By using BCMs, we can reduce the weight storage for a linear transformation layer from O(N2) to O(N). BCMs are also more efficient in terms of computational complexity, improving algorithmic complexity from O(N2) to O(Nlog(N)).Previous work has only evaluated DNNs using BCMs targeting FPGAs for inference. There has been little prior work that considers the potential benefits of using BCMs for accelerating DNN training on GPUs. In this paper, we explore acceleration of DNNs using BCM on a state-of-the-art GPU. First, we identify the challenges posed by using BCMs. Next, we perform both general and GPU-specific optimizations that impact: (i) the decomposition and interaction of individual operations, and (ii) the overall GPU kernel design. We modify the algorithmic steps to remove redundant computations, while maintaining mathematical integrity. We also leverage multiple GPU kernel optimizations, considering performance factors, such as occupancy, data sharing/reuse patterns, and memory coalescing. We evaluate the performance of DNN training on an NVIDIA Tesla V100, providing insights into the benefits of our proposed kernel optimizations on a state-of-the-art GPU. Based on our results, we can achieve average speedups of 1.31× and 2.79× for the convolutional layers and fully-connected layers, respectively for AlexNet. We can also achieve average speedups of 1.33× and 3.66× for the convolutional layers and fully-connected layers, respectively for VGGNet-16.

Changes in General Atmospheric Circulation in the Northern Hemisphere in 1998–2018

The analysis of systematic changes in the Northern Hemisphere 500 hPa geopotential height field is performed. The dataset of the 5-day mean values of 500 hPa geopotential height during 1998–2018 is used for this purpose and is represented by the coefficients of its expansions in spherical functions. It is found that the hemispheric mean value of 500 hPa geopotential height increased by 1.0–2.0 dam during the analyzed 20-year period depending on a season. The west-to-east propagation of air particles in the mid-latitudes systematically weakened in the cold season (January, October) but increased in the warm season (April, July). A small negative trend in the amplitudes of some ultralong waves is detected. This can facilitate a weakening of the planetary blocking of zonal circulation. Besides, a weak positive trend in the amplitudes of some synoptic waves is found, which can favor an increase in the cyclone activity.

Detection Algorithm for Single-Carrier Spatial Modulation Signals in Frequency-Selective Fading Channels

In this paper, we address the problem of detector design in the frequency-selective fading channels for unique-word-based single-carrier spatial modulation (UWSC-SM) systems. We propose to precede the conventional M-algorithm with an appropriately designed frequency-domain prefilter to condition the equivalent channel so that the number of the survivor paths needed for a satisfactory performance can be significantly reduced. By exploiting the block circulant channel induced by the UWSC-SM block format, the prefilter can be computed, and carried out efficiently using the fast Fourier transforms. Simulation results demonstrate that, compared to the conventional M-algorithm, the proposed algorithm achieves essentially the same error-rate performance with a considerably lower overall complexity in systems where the number of receive antennas is no smaller than that of transmit antennas.

VoxCap: FFT-Accelerated and Tucker-Enhanced Capacitance Extraction Simulator for Voxelized Structures

VoxCap, a fast Fourier transform (FFT)-accelerated and Tucker-enhanced integral equation simulator for capacitance extraction of voxelized structures, is proposed. The VoxCap solves the surface integral equations (SIEs) for conductor and dielectric surfaces with three key attributes that make the VoxCap highly CPU and memory efficient for the capacitance extraction of the voxelized structures: (i) VoxCap exploits the FFTs for accelerating the matrix-vector multiplications during the iterative solution of linear system of equations arising due to the discretization of SIEs. (ii) During the iterative solution, VoxCap uses a highly effective and memory-efficient preconditioner that reduces the number of iterations significantly. (iii) VoxCap employs Tucker decompositions to compress the block Toeplitz and circulant tensors, requiring the largest memory in the simulator. By doing so, it reduces the memory requirement of these tensors from hundreds of gigabytes to a few megabytes and the CPU time required to obtain Toeplitz tensors from tens of minutes (even hours) to a few seconds for very large scale problems. VoxCap is capable of accurately computing capacitance of arbitrarily shaped and large-scale voxelized structures on a desktop computer.

Linking crop yields in Tuscany, Italy, to large-scale atmospheric variability, circulation regimes and weather types

AbstractThe paper presents results from a study examining the relationship between large-scale modes of climate variability with the fluctuations in the yield of barley, durum wheat, olives and sunflower crops in Tuscany, Italy. In particular, the blocking circulation over the growing season, with associated hot and dry conditions, decreased yield for olive crops, barley and durum wheat. The teleconnections analysed in this study are the winter North Atlantic Oscillation (NAO) and the Summer North Atlantic Oscillation (SNAO); the West African Monsoon (WAM) and the Intertropical Front (ITF); and although NAO, SNAO, ITF and WAM are not strictly related to each other, the values of these indices are strongly related to the atmospheric circulation regimes and related weather types. Thus, they have an impact on precipitation and temperature patterns in Italy and on yields of important crops in Tuscany. Results show that the large-scale temperate and tropical variability directly influences the crop yield through three main circulation regimes. These patterns illustrate the importance of the large-scale modes, which, together with the associated weather types, have an impact directly on Tuscan crop yields; both barley and olive yields decline significantly when the ITF is further north with warmer and drier conditions in Italy.

Decadal predictability of North Atlantic blocking and the NAO

Can multi-annual variations in the frequency of North Atlantic atmospheric blocking and mid-latitude circulation regimes be skilfully predicted? Recent advances in seasonal forecasting have shown that mid-latitude climate variability does exhibit significant predictability. However, atmospheric predictability has generally been found to be quite limited on multi-annual timescales. New decadal prediction experiments from NCAR are found to exhibit remarkable skill in reproducing the observed multi-annual variations of wintertime blocking frequency over the North Atlantic and of the North Atlantic Oscillation (NAO) itself. This is partly due to the large ensemble size that allows the predictable component of the atmospheric variability to emerge from the background chaotic component. The predictable atmospheric anomalies represent a forced response to oceanic low-frequency variability that strongly resembles the Atlantic Multi-decadal Variability (AMV), correctly reproduced in the decadal hindcasts thanks to realistic ocean initialization and ocean dynamics. The occurrence of blocking in certain areas of the Euro-Atlantic domain determines the concurrent circulation regime and the phase of known teleconnections, such as the NAO, consequently affecting the stormtrack and the frequency and intensity of extreme weather events. Therefore, skilfully predicting the decadal fluctuations of blocking frequency and the NAO may be used in statistical predictions of near-term climate anomalies, and it provides a strong indication that impactful climate anomalies may also be predictable with improved dynamical models.

Some Results on the Field of Values of Matrix Polynomials

In this paper, the notions of pseudofield of values and joint pseudofield of values of matrix polynomials are introduced and some of their algebraic and geometrical properties are studied. Moreover, the relationship between the pseudofield of values of a matrix polynomial and the pseudofield of values of its companion linearization is stated, and then some properties of the augmented field of values of basic A-factor block circulant matrices are investigated.

A preconditioned fast collocation method for a linear bond-based peridynamic model

We develop a fast collocation method for a static bond-based peridynamic model. Based on the analysis of the structure of the stiffness matrix, a fast matrix-vector multiplication technique was found, which can be used in the Krylov subspace iteration method. In this paper, we also present an effective preconditioner to accelerate the convergence of the Krylov subspace iteration method. Using the block-Toeplitz–Toeplitz-block (BTTB)-type structure of the stiffness matrix, we give a block-circulant-circulant-block (BCCB)-type preconditioner. The numerical experiments show the utility of the preconditioned fast collocation method.

Generalized Structure Preserving Preconditioners for Frame-Based Image Deblurring

We are interested in fast and stable iterative regularization methods for image deblurring problems with space invariant blur. The associated coefficient matrix has a Block Toeplitz Toeplitz Blocks (BTTB) like structure plus a small rank correction depending on the boundary conditions imposed on the imaging model. In the literature, several strategies have been proposed in the attempt to define proper preconditioner for iterative regularization methods that involve such linear systems. Usually, the preconditioner is chosen to be a Block Circulant with Circulant Blocks (BCCB) matrix because it can efficiently exploit Fast Fourier Transform (FFT) for any computation, including the (pseudo-)inversion. Nevertheless, for ill-conditioned problems, it is well known that BCCB preconditioners cannot provide a strong clustering of the eigenvalues. Moreover, in order to get an effective preconditioner, it is crucial to preserve the structure of the coefficient matrix. On the other hand, thresholding iterative methods have been recently successfully applied to image deblurring problems, exploiting the sparsity of the image in a proper wavelet domain. Motivated by the results of recent papers, the main novelty of this work is combining nonstationary structure preserving preconditioners with general regularizing operators which hold in their kernel the key features of the true solution that we wish to preserve. Several numerical experiments shows the performances of our methods in terms of quality of the restorations.

The tensor t‐function: A definition for functions of third‐order tensors

SummaryA definition for functions of multidimensional arrays is presented. The definition is valid for third‐order tensors in the tensor t‐product formalism, which regards third‐order tensors as block circulant matrices. The tensor function definition is shown to have similar properties as standard matrix function definitions in fundamental scenarios. To demonstrate the definition's potential in applications, the notion of network communicability is generalized to third‐order tensors and computed for a small‐scale example via block Krylov subspace methods for matrix functions. A complexity analysis for these methods in the context of tensors is also provided.

Generalized tensor function via the tensor singular value decomposition based on the T-product

In this paper, we present the definition of generalized tensor function according to the tensor singular value decomposition (T-SVD) based on the tensor T-product. Also, we introduce the compact singular value decomposition (T-CSVD) of tensors, from which the projection operators and Moore-Penrose inverse of tensors are obtained. We establish the Cauchy integral formula for tensors by using the partial isometry tensors and apply it into the solution of tensor equations. Then we establish the generalized tensor power and the Taylor expansion of tensors. Explicit generalized tensor functions are listed. We define the tensor bilinear and sesquilinear forms and propose theorems on structures preserved by generalized tensor functions. For complex tensors, we established an isomorphism between complex tensors and real tensors. In the last part of our paper, we find that the block circulant operator establishes an isomorphism between tensors and matrices. This isomorphism is used to prove the F-stochastic structure is invariant under generalized tensor functions. The concept of invariant tensor cones is raised.

A Fast Block $\alpha$-Circulant Preconditoner for All-at-Once Systems From Wave Equations

In this paper, we propose a fast block $\alpha$-circulant preconditioner for solving the nonsymmetric linear system arising from an all-at-once implicit discretization scheme in time for the wave equation. As a generalization of the well-known block circulant preconditioning technique, the proposed block $\alpha$-circulant preconditioner can also be efficiently inverted in a parallel-in-time manner. The complex eigenvalues of the preconditioned matrix are fully derived in explicit expression and its diagonalizability is also shown. Furthermore, a mesh-independent convergence rate of the preconditioned GMRES method is derived under certain conditions. Building upon the proposed preconditioner, two stationary iterative methods with uniform asymptotic convergence rates were also presented. The extension of our preconditioner within a simplified Newton iteration to the nonlinear wave equation is also discussed. Both linear and nonlinear numerical examples are given to illustrate the promising performance of our proposed block $\alpha$-circulant preconditioner and also validate our theoretical results on convergence analysis.

Spectral distributions of periodic random matrix ensembles

Koloğlu, Kopp and Miller compute the limiting spectral distribution of a certain class of real random matrix ensembles, known as [Formula: see text]-block circulant ensembles, and discover that it is exactly equal to the eigenvalue distribution of an [Formula: see text] Gaussian unitary ensemble. We give a simpler proof that under very general conditions which subsume the cases studied by Koloğlu–Kopp–Miller, real-symmetric ensembles with periodic diagonals always have limiting spectral distribution equal to the eigenvalue distribution of a finite Hermitian ensemble with Gaussian entries which is a ‘complex version’ of a [Formula: see text] submatrix of the ensemble. We also prove an essentially algebraic relation between certain periodic finite Hermitian ensembles with Gaussian entries, and the previous result may be seen as an asymptotic version of this for real-symmetric ensembles. The proofs show that this general correspondence between periodic random matrix ensembles and finite complex Hermitian ensembles is elementary and combinatorial in nature.

Block circulant graphs and the graphs of critical pairs of crowns

In this paper, we provide a natural bijection between a special family of block circulant graphs and the graphs of critical pairs of the posets known as generalized crowns. In particular, every graph in this family of block circulant graphs we investigate has a generating block row that follows a symmetric growth pattern of the all ones matrix. The natural bijection provides an upper bound on the chromatic number for this infinite family of graphs.

A Stationary Spatio‐Temporal GARCH Model

We introduce a lagged nearest‐neighbour, stationary spatio‐temporal generalized autoregressive conditional heteroskedasticity (GARCH) model on an infinite spatial grid that opens for GARCH innovations in a space‐time ARMA model. This is illustrated by a real data application to a classical dataset of sea surface temperature anomalies in the Pacific Ocean. The model and its translation invariant neighbourhood system are wrapped around a torus forming a model with finite spatial domain, which we call circular spatio‐temporal GARCH. Such a model could be seen as an approximation of the infinite one and simulation experiments show that the circular estimator with a straightforward bias correction performs well on such non‐circular data. Since the spatial boundaries are tied together, the well‐known boundary issue in spatial statistical modelling is effectively avoided. We derive stationarity conditions for these circular processes and study the spatio‐temporal correlation structure through an ARMA representation. We also show that the matrices defined by a vectorized version of the model are block circulants. The maximum quasi‐likelihood estimator is presented and we prove its strong consistency and asymptotic normality by generalizing results from univariate GARCH theory.

Closed orbit correction at synchrotrons for symmetric and near-symmetric lattices

This contribution compiles the benefits of lattice symmetry in the context of closed orbit correction. A symmetric arrangement of BPMs and correctors results in structured orbit response matrices of Circulant or block Circulant type. These forms of matrices provide favorable properties in terms of computational complexity, information compression and interpretation of mathematical vector spaces of BPMs and correctors. For broken symmetries, a nearest-Circulant approximation is introduced and the practical advantages of symmetry exploitation are demonstrated with the help of simulations and experiments in the context of FAIR synchrotrons.

Matrix Analysis of Repetitive Circulant Structures: New-block and Near Block Matrices

In many scientific fields and several problems, Block Circulant Matrices (BCM) have been used for a long period of time. Each row of the BCM is a cyclic shift of its upper row to the right. BCM has been studied widely and there are closed-form solutions for problems of BCM. In these problems, the properties of near-BCM and BCM lead to a significant decrease in computational cost and efforts. In other words, these matrices are useful to perform some computational operations at the low cost. This study introduces a method for transforming a structure into a new type of Block Circulant Structure (BCS) by applying minor modifications. Furthermore, transformation of structural matrices into Block Circulant Matrices is discussed, and the properties of these matrices are then described in details. The methods introduce calculating eigenvalues and eigenvectors of these matrices instead of calculating the inverse of their matrices. To achieve this goal, the properties of near-Block and Block Circulant Matrices are used to analyze the structural stiffness matrices. In addition, the inverse of stiffness matrices for structures are calculated and utilized in structural mechanics. For clarification of efficiency and accuracy of the method, some examples are presented.

Fast and exact simulation of Gaussian random fields defined on the sphere cross time

We provide a method for fast and exact simulation of Gaussian random fields on the sphere having isotropic covariance functions. The method proposed is then extended to Gaussian random fields defined over the sphere cross time and having covariance functions that depend on geodesic distance in space and on temporal separation. The crux of the method is in the use of block circulant matrices obtained working on regular grids defined over longitude and latitude.

Line attractors of coupled ring neural networks with block circulant weight matrix

ABSTRACTThe continuous attractors of neural networks with symmetric connection weights have been studied widely. However, most of the matrices are asymmetric. In this paper, the model is coupled by some subnetworks. The connection weight matrix of each subnetwork is a circulant matrix, so the whole connection matrix is a block circulant matrix which is asymmetric. The conditions that guarantee this model with block circulant connection weight matrix to possess continuous attractor is obtained. Furthermore, the continuous attractor is a one-dimensional line attractor.

Circulant preconditioners for functions of Hermitian Toeplitz matrices

Circulant preconditioners for functions of matrices have been recently of interest. In particular, several authors proposed the use of the optimal circulant preconditioners as well as the superoptimal circulant preconditioners in this context and numerically illustrated that such preconditioners are effective for certain functions of Toeplitz matrices. Motivated by their results, we propose in this work the absolute value superoptimal circulant preconditioners and provide several theorems that analytically show the effectiveness of such circulant preconditioners for systems defined by functions of Toeplitz matrices. Namely, we show that the eigenvalues of the preconditioned matrices are clustered around $\pm 1$ and rapid convergence of Krylov subspace methods can therefore be expected. Moreover, we show that our results can be extended to functions of block Toeplitz matrices with Toeplitz blocks provided that the optimal block circulant matrices with circulant blocks are used as preconditioners. Numerical examples are given to support our theoretical results.