Rethinking Adaptive Computing: Building a Unified Model Complexity-Reduction Framework With Adversarial Robustness.

Abstract

Adaptive computing (AC) is a technique to dynamically select the layers to pass in a prespecified deep neural network (DNN) according to the input samples. In previous literature, AC was deemed as a standalone complexity-reduction skill. This brief studies AC through a different lens: we investigate how this strategy interacts with mainstream compression techniques in a unified complexity-reduction framework and whether its "input sample related" feature helps with the improvement of model robustness. Following this direction, we first propose a defensive accelerating branch (DAB) based on the AC strategy that can reduce the average computational cost and inference time of DNNs with higher accuracy compared with its counterparts. Then, the proposed DAB is jointly applied with the mainstream parameterwise compression skills, pruning and quantization, to build a unified complexity-reduction framework. Extensive experiments are conducted, and the results reveal quasi-orthogonality between the input-related and parameterwise complexity-reduction skills, which means that the proposed AC can be integrated into an off-the-shelf compressed model without hurting its accuracy. Besides, the robustness of the proposed compression framework is explored, and the experimental results demonstrate that DAB can be used as both the detector and the defensive tool when the model is under adversarial attacks. All these findings shed light on the great potential of DAB in building a unified complexity-reduction framework with both a high compression ratio and great adversarial robustness.