Hardware-Aware AutoML for Efficient Deep Learning Applications

Abstract

Deep Neural Networks (DNNs) have been traditionally designed by human experts in a painstaking and expensive process, dubbed by many researchers to be more of an art than science. However, the ever increasing demand for state-of-the-art performance and real-world deployment has resulted in larger models, making the manual DNN design a daunting task. AutoML presents a promising path towardsalleviating this engineering burden by automatically identifying the DNN hyperparameters, such as thenumber of layers or the type of layer-wise operations. As modern DNNs grow larger, AutoML methods face two key challenges: first, the increased DNN model sizes result in increased computational complexity during inference, making it difficult to deploy AutoML-designed DNNs to resource-constrained devices. Second, due to the large DNN design space, each AutoML search remains considerably costly, with an overall cost of hundreds of GPU-hours. In this thesis, we propose AutoML methods that are both hardware aware and search-cost efficient. Weintroduce a Bayesian optimization (BO) methodology enhanced with hardware-cost predictive models, allowing the AutoML search to traverse the design space in a constraint “complying” manner, up to 3.5? faster compared to vanilla BO methods. Moreover, we formulate the design of adaptive DNNs as an AutoML task and we jointly solve for the DNN architectures and the adaptive execution scheme, reducing energy consumption by up to 6? compared to hand-tuned designs. Next, in a departure from existing one-shotNeural Architecture Search (NAS) assumptions on how the candidate DNN architectures are evaluated, we introduce a novel view of the one-shot NAS problem as finding the subsets of kernel weights across a single-path one-shot model. Our proposed formulation reduces the NAS search cost by up to 5,000? compared to existing NAS methods. Taking advantage of such efficiency, we investigate how various design space and formulation choices affect the AutoML results, achieving a new state-of-the-art NASperformance for image classification accuracy (75.62%) under runtime constraints on mobile devices.