A Weight-Reload-Eliminated Compute-in-Memory Accelerator for 60 fps 4K Super-Resolution

Abstract

Convolutional neural networks (CNNs) have significantly enhanced the quality of image super-resolution (SR). However, deploying CNN with deeper layers for real-time 60fps and higher resolution SR, e.g., 4K, brings higher performance requirement, incurring huge area and power overhead as well. In this brief, a high performance compute-in-memory (CIM) accelerator with high energy and area efficiency is proposed for deep-layer CNN-based SR (CNNSR). It adopts alternate time division serial (ATS) dataflow to guarantee the high performance, which eliminates frequent and time-consuming weight reload operation of CIM macros. Moreover, considering the extremely high ratio of computation quantity to weight amount in the deep-layer CNNSR scenario, an energy and area efficiency enhanced digital CIM macro with high computation-to-memory ratio (CMR) is proposed. The proposed accelerator achieves 33.3TOPS peak performance, supporting 4K SR at no less than 60fps. It is evaluated under TSMC 28nm process, achieving 13.96TOPS/W energy efficiency and 1.85TOPS/mm2 area efficiency, which are <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.79\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.2\times $ </tex-math></inline-formula> better than the state-of-the-art CNNSR accelerator, respectively.