Searching on Massive Graphs and Regularizing Deep Learning

Abstract

We have designed di erent heuristics for both searching on Massive graphs and regularizing Deep Neural Networks in this work. Both the problem of nding a minimum vertex cover (MinVC) and the maximum edge weight clique (MEWC) in a graph are prominent NP-hard problems of great importance in both theory and application. During recent decades, there has been much interest in nding optimal or near-optimal solutions to these two problems. Many existing heuristic algorithms for MinVC are based on local search strategies. An algorithm called FastVC takes a rst step towards solving the MinVC problem for large real-world graphs. However, FastVC may be trapped at local minima during the local search stage due to the lack of suitable diversi cation mechanisms. Besides, since the traditional best-picking heuristic was believed to be of high complexity, FastVC replaces it with an approximate best-picking strategy. However, best-picking has been proved to be robust for a wide range of problems, so abandoning it may be a great sacri ce. Therefore, we rstly design a diversi cation heuristic to help FastVC escape from local minima, and the proposed solver is named WalkVC. Secondly, we develop a local search MinVC solver, named NoiseVC, which utilizes best-picking (low complexity) with noise to remove vertices during the local search stage in massive graphs. On the other hand, most of existing heuristics for the MEWC problem focus on academic benchmarks with relatively size. However, very little attention was paid to solving the MEWC problem in large sparse graphs. In this thesis, we exploit the so-called deterministic tournament selection (DTS) heuristic for selecting edges to improve the local search based MEWC algorithms. Deep Neural Networks (DNN), have an extremely large number of parameters comparing with traditional machine earning methods, su er from the the problem of over tting. Dropout [Hinton et al., 2012, Srivastava et al., 2014] has been proposed to address this problem. Dropout is an useful technique for regularizing and preventing the co-adaptation of neurons in DNN. It randomly drops units with a probability p during the training stage of DNN to avoid over tting. The working mechanism of dropout can be interpreted as approximately and exponentially combining many di erent neural network architectures e ciently, leading to a powerful ensemble. We propose a novel diversi cation strategy for dropout named Tabu Dropout, which aims at generating more di erent neural network architectures in fewer numbers of iterations. Besides, a recent work named Curriculum Dropout achieves the state-of-the-art performance among the dropout variants by using a scheduled p instead of a xed one. It gradually increases the dropping probability from 0 to 1 ���� p according to a time scheduling from curriculum learning. The primary intuition is that dropout seems unnecessary at the beginning of training and Curriculum Dropout starts training the whole neural networks without dropping, which is called \starting easy. In this thesis, we design a new scheduled dropout strategy using \starting small instead of \starting easy, which gradually decreases the dropping probability from 1 to p. We call this strategy Annealed Curriculum Dropout. Experiments conducted on related public standard datasets show that our proposed heuristics for both searching on massive graphs and regularizing deep learning have achieved better performance than the comparison methods.