PTTS: Power-aware tensor cores using two-sided sparsity

Abstract

Deep Neural networks (DNNs) have become the compelling solution for a broad range of applications such as automatic translation, advertisement recommendation, and speech recognition. Matrix multiplication is the fundamental operation used in various classes of DNNs. Introduction of tensor cores (TCs) in NVIDIA GPGPUs particularly aims at acceleration of neural networks. A TC is a specialized unit dedicated to compute matrix-multiply-and-accumulate (MMA) operations. State-of-the-art DNNs are known to be power-hungry and compute-intensive due to increasing depth of networks, i.e. multiple layers with massive number of neurons. This causes excessive energy in TCs.Sparse neural networks have emerged as an effective solution to address massive amount of computations in DNNs. While sparsity is widely used for acceleration of DNNs, only a handful of studies focused on energy aspect of sparse DNNs and energy-efficient architectures for TCs are scarce. We exploit sparsity in DNNs and propose power gating multipliers with sparse activations or weights. We show that sparsity occurs in TCs for short intervals and conventional power gating techniques cannot fully exploit these idle intervals mainly due to overhead of power gating. To mitigate the overhead of power gating, we propose power-aware tensor cores using two-sided sparsity (PTTS) which monitors inputs of multipliers and turns them off only if inputs remain sparse for long intervals. We also propose an architectural technique that shuffles multiplications to pack idle intervals and increase opportunities for power gating. We introduce a low-cost and implementation-efficient sparse input operand interconnect to change order of effectual multiplications. Our proposed techniques combined are able to save energy by 68% in Tensor Cores with negligible impact on performance while maintaining accuracy.