Introduction to Neural Network Verification

Comprehensive Study for Breast Cancer Using Deep Learning and Traditional Machine Learning

GenSyth: a new way to understand deep learning

Deep learning based on structured deep neural networks has provided powerful applications in various fields. The structures imposed on the deep neural networks are crucial, which makes deep learning essentially different from classical schemes based on fully connected neural networks. One of the commonly used deep neural network structures is generated by convolutions. The produced deep learning algorithms form the family of deep convolutional neural networks. Despite of their power in some practical domains, little is known about the mathematical foundation of deep convolutional neural networks such as universality of approximation. In this paper, we propose a family of new structured deep neural networks: deep distributed convolutional neural networks. We show that these deep neural networks have the same order of computational complexity as the deep convolutional neural networks, and we prove their universality of approximation. Some ideas of our analysis are from ridge approximation, wavelets, and learning theory.

Artificial intelligence in interdisciplinary life science and drug discovery research.

In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotics for deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and classes of Gaussian stochastic processes (SPs). Such an interplay has proved to be critical in several contexts of practical interest, e.g. Bayesian inference under Gaussian SP priors, kernel regression for infinite-wide deep NNs trained via gradient descent, and information propagation within infinite-wide NNs. Motivated by empirical analysis, showing the potential of replacing Gaussian distributions with Stable distributions for the NN's weights, in this paper we investigate large-width asymptotics for (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stable-distributed weights. First, we show that as the width goes to infinity jointly over the NN's layers, a suitable rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively through the NN's layers. Because of the non-triangular NN's structure, this is a non-standard asymptotic problem, to which we propose a novel and self-contained inductive approach, which may be of independent interest. Then, we establish sup-norm convergence rates of a deep Stable NN to a Stable SP, quantifying the critical difference between the settings of ``joint growth and ``sequential growth of the width over the NN's layers. Our work extends recent results on infinite-wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates for infinite-wide deep NNs.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Artificial Intelligence and the Common Sense of Animals.

What can classic Atari video games tell us about the human brain?

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

Abstract: The advancement of network security faces growing challenges as cyberattacks become more sophisticated. Traditional rule-based systems struggle with zero-day attacks and obfuscation techniques. This study examines the development trends of AI-based algo-rithms, particularly machine learning and deep learning, in threat detection. A literature review evaluates AI-driven approaches, including support vector machines, random for-est, deep neural networks, convolutional neural networks, and reinforcement learning. Findings show that AI enhances detection accuracy, adaptability, and reduces false posi-tives. Machine learning efficiently classifies known attacks, while deep learning excels in identifying complex patterns such as distributed denial-of-service and advanced persis-tent threats. Unsupervised learning improves anomaly detection without labeled data. However, AI models require high-quality data, substantial computational resources, and remain vulnerable to adversarial attacks. Despite these challenges, AI provides a dynam-ic and adaptive security solution, surpassing traditional systems. Future research should enhance AI scalability and resilience for evolving cybersecurity threats. Keywords: anomaly detection; artificial intelligence; deep learning; machine learning; network security Abstrak: Perkembangan keamanan jaringan menghadapi tantangan yang semakin besar seiring meningkatnya kompleksitas serangan siber. Sistem berbasis aturan tradisional kesulitan mendeteksi zero-day attack dan teknik penyamaran. Penelitian ini mengkaji tren pengembangan algoritma berbasis AI, khususnya machine learning dan deep learning, dalam deteksi ancaman. Literature review mengevaluasi pendekatan berbasis AI, termasuk support vector machines, random forest, deep neural networks, convolutional neural networks, dan reinforcement learning. Hasil penelitian menunjukkan bahwa AI meningkatkan akurasi deteksi, adaptabilitas terhadap ancaman baru, serta mengurangi false positive. Machine learning efektif mengklasifikasikan serangan yang telah diketahui, sementara deep learning unggul dalam mengenali pola kompleks seperti distributed denial-of-service dan advanced persistent threats. Unsupervised learning meningkatkan deteksi anomali tanpa memerlukan data berlabel. Namun, AI masih bergantung pada data berkualitas tinggi, sumber daya komputasi besar, dan rentan terhadap adversarial attack. Meskipun demikian, AI menawarkan solusi keamanan yang lebih dinamis dan adaptif dibandingkan sistem tradisional. Penelitian selanjutnya perlu difokuskan pada peningkatan skalabilitas dan ketahanan AI dalam menghadapi ancaman siber yang terus berkembang. Kata kunci: deteksi anomali; jaringan keamanan; kecerdasan buatan; pembelajaran dalam; pembelajaran mesin

Machine Learning is a sub-category of Artificial Intelligence enabling computers with the ability of pattern recognition, or to continuously learn from, making predictions based on data, and carry out decisions without being specifically programmed for doing so. In this context, Machine Learning is a broader category of algorithms being able to use datasets to identify patterns, discover insights, and enhance understanding and make decisions or predictions. Compared with Machine Learning, Deep Learning is a particular branch of Machine Learning that makes use of Machine Learning functionality, and moves beyond its capabilities. Deep Learning Algorithm is interpreted as a layered structure that tries to replicate the structure of the human brain. These capabilities enable Machine Learning and Deep Learning Algorithms usage in applications to identify and respond to cybercriminals manifold cyberattacks. This is achieved by analyzing Big Datasets of cybersecurity incidents to identify patterns of malicious activities. For this purpose, Machine Learning and Deep Learning compare known threat event attacks with detected threat event attacks to identify similarities they automatically dealt with trained Machine Learning or Deep Learning model for response. Against this background, this chapter seeks to offer a clear explanation of the classification of Machine Learning and Deep Learning and comparing them with regard to effectivity and efficiency in their specific application domains. This requires (i) discussing the methodological background of Machine Learning and Deep Learning; (ii) introducing relevant application areas of Machine Learning and Deep Learning like Intrusion Detection Systems; and (iii) use cases showing how to combat against threat event attacks based cybersecurity risks. In this context, this chapter provides, in Sect. 8.1, a brief introduction in classical Machine Learning, which consists of Supervised, Unsupervised, and Reinforcement Machine Learning. In this regard, Sect. 8.1.1.1 introduces Supervised Machine Learning, while Sect. 8.1.1.2 refers to Unsupervised Machine Learning, and Sect. 8.1.1.3 focuses on Reinforcement Machine Learning. Sect. 8.1.1.4 finally compares the different Machine Learning methods with regard to advantages and disadvantages. Based on this methodological introduction of classical Machine Learning, Sect. 8.2.1 introduces in Machine Learning and cybersecurity issues. Machine Learning-based intrusion detection in industrial application is therefore the topic of Sect. 8.2.1.1. Section 8.2.1.2 introduces Machine Learning-based intrusion detection based on feature learning, and Machine Learning-based intrusion detection of unknown cyberattacks is the topic of Sect. 8.2.1.3. In Section 8.3, the classification of Deep Learning methods is given which contains in Sect. 8.3.1 the topics Feedforward Deep Neural Networks, Convolutional Feedforward Deep Neural Networks, Recurrent Deep Neural Networks, Deep Beliefs Networks, and the Deep Bayesian Neural Network. Based on this methodological background of Deep Learning methods, Sect. 8.3.2 introduces Deep Bayesian Neural Networks, while Sect. 8.3.3 refers to Deep Learning-based intrusion detection. Finally, Sect. 8.4 refers to Deep Learning methods in cybersecurity applications. Section 8.5 contains comprehensive questions from the topics Machine Learning and Deep Learning, followed by “References” with references for further reading.

Deep learning and neural networks are powerful computational methods that have been widely applied in various fields, such as healthcare and robotics. In this paper, we review some of the recent research studies that use deep learning and neural networks in healthcare and robotics, particularly focusing on their application in prosthetics and exoskeletons. The main source of data for this review is Scopus, which is a large and multidisciplinary database of peer-reviewed literature. The search criteria for this review are exoskeleton AND prosthetic AND deep AND learning. The search is limited to documents published from 2014 to 2023, as this period covers the recent developments and trends in the field4. The search results in 488 documents that match the criteria. We selected 20 papers that represent the state-of-the-art methods and applications of deep learning and neural networks in prosthetics and exoskeletons. We categorized these papers by various attributes, such as document type, subject area, sensor type, respondent, condition, etc. The main finding of this paper was that deep learning techniques and neural networks have diverse and transformative potential in healthcare and robotics, especially in the development and improvement of prosthetics and exoskeletons. The paper highlighted how these advanced computational methods can be harnessed to interpret complex biological signals, improve device functionality, enhance user safety, and ultimately improve quality of life for individuals using these devices. The paper also identified some possible future directions for this topic, such as exploring the impact of deep learning techniques and neural networks on the performance, usability, and user satisfaction of prosthetics and exoskeletons. This paper provided a valuable insight into the current state-of-the-art and future prospects of deep learning techniques and neural networks in healthcare and robotics

Classifying the molecular functions of Rab GTPases in membrane trafficking using deep convolutional neural networks

Deep Neural Networks (DNNs) have greatly advanced the state-of-the-art in a wide range of machine learning tasks involving image, video, speech and text analytics, and are deployed in numerous widely-used products and services. Improvements in the capabilities of hardware platforms such as Graphics Processing Units (GPUs) and specialized accelerators have been instrumental in enabling these advances as they have allowed more complex and accurate networks to be trained and deployed. However, the enormous computational and memory demands of DNNs continue to increase with growing data size and network complexity, posing a continuing challenge to computing system designers. For instance, state-of-the-art image recognition DNNs require hundreds of millions of parameters and hundreds of billions of multiply-accumulate operations while state-of-the-art language models require hundreds of billions of parameters and several trillion operations to process a single input instance. Another major obstacle in the adoption of DNNs, despite their impressive accuracies on a range of datasets, has been their lack of robustness. Specifically, recent efforts have demonstrated that small, carefully-introduced input perturbations can force a DNN to behave in unexpected and erroneous ways, which can have to severe consequences in several safety-critical DNN applications like healthcare and autonomous vehicles. In this dissertation, we explore approximate computing as an avenue to improve the speed and energy efficiency of DNNs, as well as their robustness to input perturbations. Approximate computing involves executing selected computations of an application in an approximate manner, while generating favorable trade-offs between computational efficiency and output quality. The intrinsic error resilience of machine learning applications makes them excellent candidates for approximate computing, allowing us to achieve execution time and energy reductions with minimal effect on the quality of outputs. This dissertation performs a comprehensive analysis of different approximate computing techniques for improving the execution efficiency of DNNs. Complementary to generic approximation techniques like quantization, it identifies approximation opportunities based on the specific characteristics of three popular classes of networks - Feed-forward Neural Networks (FFNNs), Recurrent Neural Networks (RNNs) and Spiking Neural Networks (SNNs), which vary considerably in their network structure and computational patterns. First, in the context of feed-forward neural networks, we identify sparsity, or the presence of zero values in the data structures (activations, weights, gradients and errors), to be a major source of redundancy and therefore, an easy target for approximations. We develop lightweight micro-architectural and instruction set extensions to a general-purpose processor core that enable it to dynamically detect zero values when they are loaded and skip future instructions that are rendered redundant by them. Next, we explore LSTMs (the most widely used class of RNNs), which map sequences from an input space to an output space. We propose hardware-agnostic approximations that dynamically skip redundant symbols in the input sequence and discard redundant elements in the state vector to achieve execution time benefits. Following that, we consider SNNs, which are an emerging class of neural networks that represent and process information in the form of sequences of binary spikes. Observing that spike-triggered updates along synaptic connections are the dominant operation in SNNs, we propose hardware and software techniques to identify connections that can be minimally impact the output quality and deactivate them dynamically, skipping any associated updates. The dissertation also delves into the efficacy of combining multiple approximate computing techniques to improve the execution efficiency of DNNs. In particular, we focus on the combination of quantization, which reduces the precision of DNN data-structures, and pruning, which introduces sparsity in them. We observe that the ability of pruning to reduce the memory demands of quantized DNNs decreases with precision as the overhead of storing non-zero locations alongside the values starts to dominate in different sparse encoding schemes. We analyze this overhead and the overall compression of three different sparse formats across a range of sparsity and precision values and propose a hybrid compression scheme that identifies that optimal sparse format for a pruned low-precision DNN. Along with improved execution efficiency of DNNs, the dissertation explores an additional advantage of approximate computing in the form of improved robustness. We propose ensembles of quantized DNN models with different numerical precisions as a new approach to increase robustness against adversarial attacks. It is based on the observation that quantized neural networks often demonstrate much higher robustness to adversarial attacks than full precision networks, but at the cost of a substantial loss in accuracy on the original (unperturbed) inputs. We overcome this limitation to achieve the best of both worlds, i.e., the higher unperturbed accuracies of the full precision models combined with the higher robustness of the low precision models, by composing them in an ensemble. In summary, this dissertation establishes approximate computing as a promising direction to improve the performance, energy efficiency and robustness of neural networks.

In recent years, deep learning has revolutionized the field of computer vision and has achieved state-of-the-art performance in a variety of applications. However, training a robust deep neural network necessitates a large amount of hand-labeled training data, which is time-consuming and labor-intensive to acquire. Active learning and transfer learning are two popular methodologies to address the problem of learning with limited labeled data. Active learning attempts to select the salient and exemplar instances from large amounts of unlabeled data; transfer learning leverages knowledge from a labeled source domain to develop a model for a (related) target domain, where labeled data is scarce. In this paper, we propose a novel active transfer learning algorithm with the objective of learning informative feature representations from a given dataset using a deep convolutional neural network, under the constraint of weak supervision. We formulate a loss function relevant to the research task and exploit the gradient descent algorithm to optimize the loss and train the deep network. To the best of our knowledge, this is the first research effort to propose a task-specific loss function integrating active and transfer learning, with the goal of learning informative feature representations using a deep neural network, under weak human supervision. Our extensive empirical studies on a variety of challenging, real-world applications depict the merit of our framework over competing baselines.