Toward Adversary-aware Non-iterative Model Pruning through D ynamic N etwork R ewiring of DNNs

Abstract

Listen

Translate article

We present a dynamic network rewiring (DNR) method to generate pruned deep neural network (DNN) models that both are robust against adversarially generated images and maintain high accuracy on clean images. In particular, the disclosed DNR training method is based on a unified constrained optimization formulation using a novel hybrid loss function that merges sparse learning with robust adversarial training. This training strategy dynamically adjusts inter-layer connectivity based on per-layer normalized momentum computed from the hybrid loss function. To further improve the robustness of the pruned models, we propose DNR++, an extension of the DNR method where we introduce the idea of sparse parametric Gaussian noise tensor that is added to the weight tensors to yield robust regularization. In contrast to existing robust pruning frameworks that require multiple training iterations, the proposed DNR and DNR++ achieve an overall target pruning ratio with only a single training iteration and can be tuned to support both irregular and structured channel pruning. To demonstrate the efficacy of the proposed method under the no-increased-training-time “free” adversarial training scenario, we finally present FDNR++, a simple yet effective training modification that can yield robust yet compressed models requiring training time comparable to that of an unpruned non-adversarial training. To evaluate the merits of our disclosed training methods, experiments were performed with two widely accepted models, namely VGG16 and ResNet18, on CIFAR-10 and CIFAR-100 as well as with VGG16 on Tiny-ImageNet. Compared to the baseline uncompressed models, our methods provide over 20× compression on all the datasets without any significant drop of either clean or adversarial classification performance. Moreover, extensive experiments show that our methods consistently find compressed models with better clean and adversarial image classification performance than what is achievable through state-of-the-art alternatives. We provide insightful observations to help make various model, parameter density, and prune-type selection choices and have open-sourced our saved models and test codes to ensure reproducibility of our results.