Evolution Of Deep Neural Networks Research Articles

3D reconstruction of coronary arteries using deep networks from synthetic X-ray angiogram data

Cardiovascular disease (CVD) is one of the most common health problems that are responsible for one-third of all deaths around the globe. Although X-Ray angiography has deficiencies such as two-dimensional (2D) representation of three dimensional (3D) structures, vessel overlapping, noisy background, the existence of other tissues/organs in images, etc., it is used as the gold standard technique for the diagnosis and in some cases treatment of CVDs. To overcome the deficiencies, great efforts have been drawn on retrieval of actual 3D representation of coronary arterial tree from 2D X-ray angiograms. However, the proposed algorithms are based on analytical methods and enforce some constraints. With the evolution of deep neural networks, 3D reconstruction from images can be achieved effectively. In this study, we propose a new data structure for the representation of objects in a tubular shape for 3D reconstruction of arteries using deep learning. Moreover, we propose a method to generate synthetic coronaries from data of real subjects. Then, we validate tubular shape representation using 3 typical deep learning architectures with synthetic X-ray data we produced. The input to deep learning architectures is multi-view segmented X-Ray images and the output is the structured tubular representation. We compare results qualitatively in terms of visual appearance and quantitatively in terms of Chamfer Distance and Mean Squared Error. The results demonstrate that tubular representation has promising performance in 3D reconstruction of coronaries. We observe that convolutional neural network (CNN) based architectures yield better 3D reconstruction performance with 9.9e-3 on Chamfer Distance. On the other hand, LSTM-based network fails to learn the coronary tree structure and we conclude that LSTMs are not appropriate for auto-regression problems as depicted in this study.

DeepAuth: A DNN Authentication Framework by Model-Unique and Fragile Signature Embedding

Along with the evolution of deep neural networks (DNNs) in many real-world applications, the complexity of model building has also dramatically increased. Therefore, it is vital to protect the intellectual property (IP) of the model builder and ensure the trustworthiness of the deployed models. Meanwhile, adversarial attacks on DNNs (e.g., backdoor and poisoning attacks) that seek to inject malicious behaviors have been investigated recently, demanding a means for verifying the integrity of the deployed model to protect the users. This paper presents a novel DNN authentication framework DeepAuth that embeds a unique and fragile signature to each protected DNN model. Our approach exploits sensitive key samples that are well crafted from the input space to latent space and then to logit space for producing signatures. After embedding, each model will respond distinctively to these key samples, which creates a model-unique signature as a strong tool for authentication and user identity. The signature embedding process is also designed to ensure the fragility of the signature, which can be used to detect malicious modifications such that an illegitimate user or an altered model should not have the intact signature. Extensive evaluations on various models over a wide range of datasets demonstrate the effectiveness and efficiency of the proposed DeepAuth.

A Formalism of DNN Accelerator Flexibility

The high efficiency of domain-specific hardware accelerators for machine learning (ML) has come fromspecialization, with the trade-off of less configurability/ flexibility. There is growing interest in developingflexible ML accelerators to make them future-proof to the rapid evolution of Deep Neural Networks (DNNs). However, the notion of accelerator flexibility has always been used in an informal manner, restricting computer architects from conducting systematic apples-to-apples design-space exploration (DSE) across trillions of choices. In this work, we formally define accelerator flexibility and show how it can be integrated for DSE. % flows. Specifically, we capture DNN accelerator flexibility across four axes: %the map-space of DNN accelerator along four flexibility axes: tiling, ordering, parallelization, and array shape. We categorize existing accelerators into 16 classes based on their axes of flexibility support, and define a precise quantification of the degree of flexibility of an accelerator across each axis. We leverage these to develop a novel flexibility-aware DSE framework. %It respects the difference of accelerator flexibility classes and degree of flexibility support in different accelerators, creating unique map-spaces. %and forms a unique map space for exploration. % We demonstrate how this can be used to perform first-of-their-kind evaluations, including an isolation study to identify the individual impact of the flexibility axes. We demonstrate that adding flexibility features to a hypothetical DNN accelerator designed in 2014 improves runtime on future (i.e., present-day) DNNs by 11.8x geomean.

Ambient Air Pollution Forecasting System using Deep Neural Networks

Air pollution is a major problem that has been recognized throughout the world. Harmful impacts of air contamination include hypersensitive reactions such as throat irritation, itchy eyes, nose, and some other serious problems. In recent years, the number of fatalities occurred due to air pollution has been increasing dramatically. In this paper, various air pollutants such as Carbon Monoxide, Methane or natural gas, LPG, and air quality at different places of city are measured using sensors. Further, the detected values are then used in the prediction of future values. The evolution of deep neural networks and Internet of Things made this possible to detect and forecast the concentration of pollutants underlying in the air. We use a special module called pyFirmata firmware which is used to connect the Arduino with python and upload the data into csv file on Jupyter Notebook. Here, the data collected is univariate i.e. it varies with only time. Though there are many statistical models to predict time series datasets such as ARIMA, their efficiency is low. Deep Neural Networks works well for predicting univariate as well as time series datasets. Hence, the Keras sequential model is employed to predict the hourly future values of air pollutants based on previous readings. The final results of prediction are compared with the actual values and error is calculated. As a result, the level of air pollutants at a particular hour can be predicted. The concentration of air pollutants in coming years, month or week helps us to reduce its concentration to lesser than the harmful or toxic range.

Speculative Symbolic Graph Execution of Imperative Deep Learning Programs

The rapid evolution of deep neural networks is demanding deep learning (DL) frameworks not only to satisfy the requirement of quickly executing large computations, but also to support straightforward programming models for quickly implementing and experimenting with complex network structures. However, existing frameworks fail to excel in both departments simultaneously, leading to diverged efforts for optimizing performance and improving usability. This paper presents JANUS, a system that combines the advantages from both sides by transparently converting an imperative DL program written in Python, a de-facto scripting language for DL, into an efficiently executable symbolic dataflow graph. JANUS can convert various dynamic features of Python, including dynamic control flow, dynamic types, and impure functions, into elements of a symbolic dataflow graph. Our experiments show that JANUS can achieve fast DL training by exploiting the techniques imposed by symbolic graph-based DL frameworks, while maintaining the simple and flexible programmability of imperative DL frameworks at the same time.

What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models

Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. Previous work largely focused on what these models learn at the representation level. We break this analysis down further and study individual dimensions (neurons) in the vector representation learned by end-to-end neural models in NLP tasks. We propose two methods: Linguistic Correlation Analysis, based on a supervised method to extract the most relevant neurons with respect to an extrinsic task, and Cross-model Correlation Analysis, an unsupervised method to extract salient neurons w.r.t. the model itself. We evaluate the effectiveness of our techniques by ablating the identified neurons and reevaluating the network’s performance for two tasks: neural machine translation (NMT) and neural language modeling (NLM). We further present a comprehensive analysis of neurons with the aim to address the following questions: i) how localized or distributed are different linguistic properties in the models? ii) are certain neurons exclusive to some properties and not others? iii) is the information more or less distributed in NMT vs. NLM? and iv) how important are the neurons identified through the linguistic correlation method to the overall task? Our code is publicly available as part of the NeuroX toolkit (Dalvi et al. 2019a). This paper is a non-archived version of the paper published at AAAI (Dalvi et al. 2019b).

Visual Genealogy of Deep Neural Networks.

A comprehensive and comprehensible summary of existing deep neural networks (DNNs) helps practitioners understand the behaviour and evolution of DNNs, offers insights for architecture optimization, and sheds light on the working mechanisms of DNNs. However, this summary is hard to obtain because of the complexity and diversity of DNN architectures. To address this issue, we develop DNN Genealogy, an interactive visualization tool, to offer a visual summary of representative DNNs and their evolutionary relationships. DNN Genealogy enables users to learn DNNs from multiple aspects, including architecture, performance, and evolutionary relationships. Central to this tool is a systematic analysis and visualization of 66 representative DNNs based on our analysis of 140 papers. A directed acyclic graph is used to illustrate the evolutionary relationships among these DNNs and highlight the representative DNNs. A focus + context visualization is developed to orient users during their exploration. A set of network glyphs is used in the graph to facilitate the understanding and comparing of DNNs in the context of the evolution. Case studies demonstrate that DNN Genealogy provides helpful guidance in understanding, applying, and optimizing DNNs. DNN Genealogy is extensible and will continue to be updated to reflect future advances in DNNs.