Differentiable Architecture Search Research Articles

Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search

In the past few years, Differentiable Neural Architecture Search (DNAS) rapidly imposed itself as the trending approach to automate the discovery of deep neural network architectures. This rise is mainly due to the popularity of DARTS (Differentiable ARchitecTure Search), one of the first major DNAS methods. In contrast with previous works based on Reinforcement Learning or Evolutionary Algorithms, DNAS is faster by several orders of magnitude and uses fewer computational resources. In this comprehensive survey, we focused specifically on DNAS and reviewed recent approaches in this field. Furthermore, we proposed a novel challenge-based taxonomy to classify DNAS methods. We also discussed the contributions brought to DNAS in the past few years and its impact on the global NAS field. Finally, we concluded by giving some insights into future research directions for the DNAS field.

DARTS-based morphological neural network design and application in bearing fault diagnosis

Morphological filters are widely used in data pre-processing, whereas the existing morphological filters are not optimal in terms of performance, since they are designed by combining the basic morphological operators together artificially. Hence, we proposed an automatic morphological neural network design method based on differentiable architecture search (DARTS), which is a neural architecture search (NAS) method. Firstly, the mechanism is revealed that the morphological dilation with flat structural elements behaves the same as Maxpooling without down-sampling. Secondly, morphological dilation and erosion neural network layers were defined. Thirdly, using DARTS and the search space with morphological neural network layers, end-to-end morphological neural network was implemented. Finally, the morphological neural network was successfully applied in bearing fault diagnosis. Experiments on the three datasets show that the proposed method improved the feature extraction capabilities of neural networks, then achieved end-to-end bearing fault diagnosis, and the diagnosis accuracy was improved by more than 2.0%.

Wavelet-driven differentiable architecture search for planetary gear fault diagnosis

With the advancement of artificial intelligence and the accumulation of industrial big data, intelligent diagnosis methods based on deep learning have become the mainstream for diagnosing mechanical faults in manufacturing systems. Despite this, developing high-performance neural network models for specific tasks necessitates a substantial amount of expert knowledge. This consumes a considerable amount of time in the trial-and-error process, thereby constraining the progress in neural network development. Neural architecture search (NAS) offers a solution to address this problem. However, the feature extraction operators in the existing NAS search space are primarily imported from other fields, leading to domain bias and a lack of interpretability. To address these challenges, we present a NAS method driven by wavelets. To be specific, the wavelet operators that can extract fault related features from vibration signals is added to the search space, and the gradient optimization strategy is utilized to search the optimal architecture from the hypernet. The effectiveness of the proposed method is validated through a dataset specific to planetary gears. Upon comparison with other models, it is evident that the proposed method exhibits superior performance across all added noise levels.

The limitations of differentiable architecture search

The limitations of differentiable architecture search

EG-NAS: Neural Architecture Search with Fast Evolutionary Exploration

Differentiable Architecture Search (DARTS) has achieved a rapid search for excellent architectures by optimizing architecture parameters through gradient descent. However, this efficiency comes with a significant challenge: the risk of premature convergence to local optima, resulting in subpar performance that falls short of expectations. To address this issue, we propose a novel and effective method called Evolutionary Gradient-Based Neural Architecture Search (EG-NAS). Our approach combines the strengths of both gradient descent and evolutionary strategy, allowing for the exploration of various optimization directions during the architecture search process. To begin with, we continue to employ gradient descent for updating network parameters to ensure efficiency. Subsequently, to mitigate the risk of premature convergence, we introduce an evolutionary strategy with global search capabilities to optimize the architecture parameters. By leveraging the best of both worlds, our method strikes a balance between efficient exploration and exploitation of the search space. Moreover, we have redefined the fitness function to not only consider accuracy but also account for individual similarity. This inclusion enhances the diversity and accuracy of the optimized directions identified by the evolutionary strategy. Extensive experiments on various datasets and search spaces demonstrate that EG-NAS achieves highly competitive performance at significantly low search costs compared to state-of-the-art methods. The code is available at https://github.com/caicaicheng/EG-NAS.

Neural architecture search via progressive partial connection with attention mechanism

Differentiable architecture search requires a larger computational consumption during architecture search, and there exists the depth gap problem under deeper network architecture. In this paper, we propose an attention-based progressive partially connected neural architecture search method (PPCAtt-NAS) to address these two issues. First, we introduce a progressive search strategy in the architecture search phase, build up the sophistication of the architecture gradually and perform path-level pruning in stages to bridge the depth gap. Second, we adopt a partial search scheme that performs channel-level partial sampling of the network architecture to further reduce the computational complexity of the architecture search. In addition, an attention mechanism is devised to improve the architecture search capability by enhancing the relevance between the feature channels. Finally, we conduct extensive comparison experiments with state-of-the-art methods on several public datasets, and our method is able to present higher architecture performance.

Evolution and Efficiency in Neural Architecture Search: Bridging the Gap between Expert Design and Automated Optimization

The paper provides a comprehensive overview of Neural Architecture Search (NAS), emphasizing its evolution from manual design to automated, computationally driven approaches. It covers the inception and growth of NAS, highlighting its application across various domains, including medical imaging and natural language processing. The document details the shift from expert-driven design to algorithm-driven processes, exploring initial methodologies like reinforcement learning and evolutionary algorithms. It also discusses the challenges of computational demands and the emergence of efficient NAS methodologies, such as Differentiable Architecture Search and hardware-aware NAS. The paper further elaborates on NAS's application in computer vision, NLP, and beyond, demonstrating its versatility and potential for optimizing neural network architectures across different tasks. Future directions and challenges, including computational efficiency and the integration with emerging AI domains, are addressed, showcasing NAS's dynamic nature and its continued evolution towards more sophisticated and efficient architecture search methods.

Improved Differentiable Architecture Search With Multi-Stage Progressive Partial Channel Connections

Improved Differentiable Architecture Search With Multi-Stage Progressive Partial Channel Connections

Differentiable Architecture Search With Attention Mechanisms for Generative Adversarial Networks

Differentiable Architecture Search With Attention Mechanisms for Generative Adversarial Networks

Inner Loop-Based Modified Differentiable Architecture Search

Inner Loop-Based Modified Differentiable Architecture Search

NDARTS: A Differentiable Architecture Search Based on the Neumann Series

Neural architecture search (NAS) has shown great potential in discovering powerful and flexible network models, becoming an important branch of automatic machine learning (AutoML). Although search methods based on reinforcement learning and evolutionary algorithms can find high-performance architectures, these search methods typically require hundreds of GPU days. Unlike searching in a discrete search space based on reinforcement learning and evolutionary algorithms, the differentiable neural architecture search (DARTS) continuously relaxes the search space, allowing for optimization using gradient-based methods. Based on DARTS, we propose NDARTS in this article. The new algorithm uses the Implicit Function Theorem and the Neumann series to approximate the hyper-gradient, which obtains better results than DARTS. In the simulation experiment, an ablation experiment was carried out to study the influence of the different parameters on the NDARTS algorithm and to determine the optimal weight, then the best performance of the NDARTS algorithm was searched for in the DARTS search space and the NAS-BENCH-201 search space. Compared with other NAS algorithms, the results showed that NDARTS achieved excellent results on the CIFAR-10, CIFAR-100, and ImageNet datasets, and was an effective neural architecture search algorithm.

PWSNAS: Powering Weight Sharing NAS With General Search Space Shrinking Framework.

Neural architecture search (NAS) depends heavily on an efficient and accurate performance estimator. To speed up the evaluation process, recent advances, like differentiable architecture search (DARTS) and One-Shot approaches, instead of training every model from scratch, train a weight-sharing super-network to reuse parameters among different candidates, in which all child models can be efficiently evaluated. Though these methods significantly boost search efficiency, they inherently suffer from inaccurate and unstable performance estimation. To this end, we propose a general and effective framework for powering weight-sharing NAS, namely, PWSNAS, by shrinking search space automatically, i.e., candidate operators will be discarded if they are less important. With the strategy, our approach can provide a promising search space of a smaller size by progressively simplifying the original search space, which can reduce difficulties for existing NAS methods to find superior architectures. In particular, we present two strategies to guide the shrinking process: detect redundant operators with a new angle-based metric and decrease the degree of weight sharing of a super-network by increasing parameters, which differentiates PWSNAS from existing shrinking methods. Comprehensive analysis experiments on NASBench-201 verify the superiority of our proposed metric over existing accuracy-based and magnitude-based metrics. PWSNAS can easily apply to the state-of-the-art NAS methods, e.g., single path one-shot neural architecture search (SPOS), FairNAS, ProxylessNAS, DARTS, and progressive DARTS (PDARTS). We evaluate PWSNAS and demonstrate consistent performance gains over baseline methods.

Personalized Federated DARTS for Electricity Load Forecasting of Individual Buildings

Building-level load forecasting is becoming increasingly crucial since it forms the foundation for better building energy management, which will lower energy consumption and reduce CO2 emissions. However, building-level load forecasting faces the challenges of high load volatility and heterogeneous consumption behaviors. Simple regression models may fail to fit the complex load curves, whereas sophisticated models are prone to overfitting due to the limited data of an individual building. To this end, we develop a novel forecasting model that integrates federated learning (FL), the differentiable architecture search (DARTS) technique, and a two-stage personalization approach. Specifically, buildings are first grouped according to the model architectures, and for each building cluster, a global model is designed and trained in a federated manner. Then, a local fine-tuning approach is used to adapt the cluster global model to each individual building. In this way, data resources from multiple buildings can be utilized to construct high-performance forecasting models while protecting each building’s data privacy. Furthermore, personalized models with specific architectures can be trained for heterogeneous buildings. Extensive experiments on a publicly available dataset are conducted to validate the superiority of the proposed method.

DIY Your EasyNAS for Vision: Convolution Operation Merging, Map Channel Reducing, and Search Space to Supernet Conversion Tooling.

Despite its popularity as a one-shot Neural Architecture Search (NAS) approach, the applicability of differentiable architecture search (DARTS) on complex vision tasks is still limited by the high computation and memory costs incurred by the over-parameterized supernet. We propose a new architecture search method called EasyNAS, whose memory and computational efficiency is achieved via our devised operator merging technique which shares and merges the weights of candidate convolution operations into a single convolution, and a dynamic channel refinement strategy. We also introduce a configurable search space-to-supernet conversion tool, leveraging the concept of atomic search components, to enable its application from classification to more complex vision tasks: detection and semantic segmentation. In classification, EasyNAS achieves state-of-the-art performance on the NAS-Bench-201 benchmark, attaining an impressive 76.2% accuracy on ImageNet. For detection, it achieves a mean average precision (mAP) of 40.1 with 120 frames per second (FPS) on MS-COCO test-dev. Additionally, we transfer the discovered architecture to the rotation detection task, where EasyNAS achieves a remarkable 77.05 mAP 50 on the DOTA-v1.0 test set, using only 21.1 M parameters. In semantic segmentation, it achieves a competitive mean intersection over union (mIoU) of 72.6% at 173 FPS on Cityscape, after searching for only 0.7 GPU-day.

F-DARTS: Foveated Differentiable Architecture Search Based Multimodal Medical Image Fusion.

Multimodal medical image fusion (MMIF) is highly significant in such fields as disease diagnosis and treatment. The traditional MMIF methods are difficult to provide satisfactory fusion accuracy and robustness due to the influence of such possible human-crafted components as image transform and fusion strategies. Existing deep learning based fusion methods are generally difficult to ensure image fusion effect due to the adoption of a human-designed network structure and a relatively simple loss function and the ignorance of human visual characteristics during weight learning. To address these issues, we have presented the foveated differentiable architecture search (F-DARTS) based unsupervised MMIF method. In this method, the foveation operator is introduced into the weight learning process to fully explore human visual characteristics for the effective image fusion. Meanwhile, a distinctive unsupervised loss function is designed for network training by integrating mutual information, sum of the correlations of differences, structural similarity and edge preservation value. Based on the presented foveation operator and loss function, an end-to-end encoder-decoder network architecture will be searched using the F-DARTS to produce the fused image. Experimental results on three multimodal medical image datasets demonstrate that the F-DARTS performs better than several traditional and deep learning based fusion methods by providing visually superior fused results and better objective evaluation metrics.

D-DARTS: Distributed Differentiable Architecture Search

Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. It drastically reduces search cost by resorting to weight-sharing. However, this approach also dramatically reduces the search space, thus excluding potential promising architectures. In this article, we propose D-DARTS, a solution that addresses this problem by nesting neural networks at the cell level instead of using weight-sharing to produce more diversified and specialized architectures. Moreover, we introduce a novel algorithm that can derive deeper architectures from a few trained cells, increasing performance and saving computation time. In addition, we introduce DARTOpti, an alternative search space in which we optimize existing handcrafted architectures instead of searching from scratch. Our solution reaches competitive performance on image classification tasks.

ATL-DAS: Automatic Topology Learning for Differentiable Architecture Search

Differentiable architecture search (DARTS) provides a fast solution to Neural Architecture Search (NAS) by using gradient descent over a continuous search space. Despite its high search speed, DARTS suffers from several problems such as ambiguous topology selection, operation co-adaptation, incomplete NAS pipeline, and large memory consumption. To address these problems, we first introduce topology parameters into the search space to explicitly model the network topology, which ensures the searched network architecture is well-defined. Next, we propose two sampling strategies to sample independent child networks for training and evaluation, which solve the co-adaptation problem while making the NAS pipeline complete. Finally, we use hard pruning to avoid invalid computations, which greatly reduces memory consumption. The proposed Automatic Topology Learning for Differentiable Architecture Search (ATL-DAS) algorithm performs favorably against the state-of-the-art approaches on CIFAR10 and CIFAR100 with error rates of 2.49% and 16.8%, respectively. Moreover, the architectures searched perform well across various visual tasks, including ImageNet classification, COCO object detection, and Composition-1K image matting, highlighting the potential of ATL-DAS for real-world applications.

Improving Differentiable Architecture Search via self-distillation

Differentiable Architecture Search (DARTS) is a simple yet efficient Neural Architecture Search (NAS) method. During the search stage, DARTS trains a supernet by jointly optimizing architecture parameters and network parameters. During the evaluation stage, DARTS discretizes the supernet to derive the optimal architecture based on architecture parameters. However, recent research has shown that during the training process, the supernet tends to converge towards sharp minima rather than flat minima. This is evidenced by the higher sharpness of the loss landscape of the supernet, which ultimately leads to a performance gap between the supernet and the optimal architecture. In this paper, we propose Self-Distillation Differentiable Neural Architecture Search (SD-DARTS) to alleviate the discretization gap. We utilize self-distillation to distill knowledge from previous steps of the supernet to guide its training in the current step, effectively reducing the sharpness of the supernet’s loss and bridging the performance gap between the supernet and the optimal architecture. Furthermore, we introduce the concept of voting teachers, where multiple previous supernets are selected as teachers, and their output probabilities are aggregated through voting to obtain the final teacher prediction. Experimental results on real datasets demonstrate the advantages of our novel self-distillation-based NAS method compared to state-of-the-art alternatives.

DASS: Differentiable Architecture Search for Sparse Neural Networks

The deployment of Deep Neural Networks (DNNs) on edge devices is hindered by the substantial gap between performance requirements and available computational power. While recent research has made significant strides in developing pruning methods to build a sparse network for reducing the computing overhead of DNNs, there remains considerable accuracy loss, especially at high pruning ratios. We find that the architectures designed for dense networks by differentiable architecture search methods are ineffective when pruning mechanisms are applied to them. The main reason is that the current methods do not support sparse architectures in their search space and use a search objective that is made for dense networks and does not focus on sparsity. This paper proposes a new method to search for sparsity-friendly neural architectures. It is done by adding two new sparse operations to the search space and modifying the search objective. We propose two novel parametric SparseConv and SparseLinear operations in order to expand the search space to include sparse operations. In particular, these operations make a flexible search space due to using sparse parametric versions of linear and convolution operations. The proposed search objective lets us train the architecture based on the sparsity of the search space operations. Quantitative analyses demonstrate that architectures found through DASS outperform those used in the state-of-the-art sparse networks on the CIFAR-10 and ImageNet datasets. In terms of performance and hardware effectiveness, DASS increases the accuracy of the sparse version of MobileNet-v2 from 73.44% to 81.35% (+7.91% improvement) with a 3.87× faster inference time.

TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search

In the realm of machine learning, the optimization of hyperparameters and the design of neural architectures entail laborious and time-intensive endeavors. To address these challenges, considerable research effort has been directed towards Automated Machine Learning (AutoML), with a focus on enhancing these inherent inefficiencies. A pivotal facet of this pursuit is Neural Architecture Search (NAS), a domain dedicated to the automated formulation of neural network architectures. Given the pronounced impact of network architecture on neural network performance, NAS techniques strive to identify architectures that can manifest optimal performance outcomes. A prominent algorithm in this area is Differentiable Architecture Search (DARTS), which transforms discrete search spaces into continuous counterparts using gradient-based methodologies, thereby surpassing prior NAS methodologies. Notwithstanding DARTS’ achievements, a discrepancy between discrete and continuously encoded architectures persists. To ameliorate this disparity, we propose TA-DARTS in this study—a temperature annealing technique applied to the Softmax function, utilized for encoding the continuous search space. By leveraging temperature values, architectural weights are judiciously adjusted to alleviate biases in the search process or to align resulting architectures more closely with discrete values. Our findings exhibit advancements over the original DARTS methodology, evidenced by a 0.07%p enhancement in validation accuracy and a 0.16%p improvement in test accuracy on the CIFAR-100 dataset. Through systematic experimentation on benchmark datasets, we establish the superiority of TA-DARTS over the original mixed operator, thereby underscoring its efficacy in automating neural architecture design.