Rethinking feature extraction and aggregation for lightweight single-image super-resolution

Abstract

Deep feature extraction and aggregation are critical components for lightweight single-image super-resolution networks. However, advanced feature extraction approaches like the reparameterization used in edge-enhanced feature distillation network increase the reparameterization structure to seven branches, which introduces additional training time and limits devices with strong parallel computing power, such as graphics processing unit. Simultaneously, commonly used feature aggregation methods, such as global and local feature aggregation, do not sufficiently exploit hierarchical features extracted from different deep feature extraction blocks. To address the above limitations, we propose a novel reparameterized branching block structure named spatially and channel diversified branching block. It employs three components with different receptive fields and two depthwise convolutions with different kernel sizes to increase spatial and channel diversity, enrich the reconstruction details, and significantly reduce the training time cost of the branches. Meanwhile, we analyze the memory consumption of generic aggregation methods and propose an adaptive hierarchical aggregation (AHA) method. It adopts a set of trainable parameters and channel attention mechanisms to implement a selective aggregation approach, further enhancing the feature aggregation’s generalization ability. Based on the proposed feature extraction and aggregation methods, we construct a spatial and channel diverse network (SCDNet). Extensive experiments across five benchmark datasets demonstrate the effectiveness of spatial and channel features in feature extraction, with AHA showing perfect aggregation ability. Our method provides superior performance over state-of-the-art single-image super-resolution methods, including reparameterized branch structures and other common approaches, in terms of both standard metrics and visual effects.