Can DNN models simulate appearance variations of #TheDress?

In the recent years the applications of deep neural networks are increasing rapidly. There are two important factors determining the efficiency of training a computer vision system using deep neural networks. The first factor is the difficulty of training a very deep neural network with large number of parameters. The second factor is the efficiency of the trained network for decreasing the computational cost. In this paper an efficient deep neural network which uses the grid size reduction, factorization and hyper parameter tuning is proposed. In order to deal with large number of layers the residual units are used. A series of experimental simulations are performed on the application of the proposed deep neural network for classification of aerial images. The experimental results show that the proposed architecture has acceptable accuracy for aerial scene classification.

Memory resource is a critical bottleneck for large-scale Deep Neural Network (DNN) applications. Hybrid Memory System (HMS) provides a promising solution to increase memory capacity in an affordable way. However, to release the powerful performance of HMS, data migration plays an important role. A typical DNN application has a couple of execution layers, and each requires distinct data objects. Deploying DNN on HMS imposes enormous challenges on data migration strategy and inspires us to pursue smart solutions. To tackle the data migration problem on HMS for DNN applications, we propose a runtime system for HMS that automatically optimizes scalable data migration and exploits domain knowledge on DNN to decide data migrations between the fast and slow memories in HMS. To achieve a better performance in data migrations for DNN training, we introduce a reference distance and location based data management strategy (ReDL) that treats short-lived and long-lived data objects with Idle and Dynamic migration methods, respectively. Using ReDL, DNN training on HMS with a smaller fast memory size can achieve similar performance to the fast memory-only system. The experimental results demonstrate that with configured the size of fast memory to be 20% of each workload’s peak memory consumption, our work achieves a similar performance (at most 9.6% performance difference) to the fast memory-only system. It further achieves an average of 19% and 11% improvement in data locality against the state-of-the-art solutions.

Online dynamic stability analysis plays a key role in modern power systems and is becoming even more vital with the penetration of Distributed Energy Resources (DERs). Identification of an accurate lower-order model for the ‘external system” makes it possible to carry out the stability analysis of a study system faster than the analysis for the original detailed system. However, the identified lower-order model becomes invalid when the operating state changes in the power system. ‘Black box’ representation of the ‘external system’ with the application of Deep Neural Networks (DNN) is widely discussed in the literature. In such equivalent models, the DNN response changes based on two aspects i.e. the operating point of the power system and the location/type of the fault. In this paper, a clustering algorithm is proposed to identify the clusters of the ‘external system’ models for different states of the power system, and a DNN is trained for each cluster. The clustering algorithm is based on the pre-fault steady-state power flow from the ‘external system’ to the ‘study system’, which is a reflection of changes in the system state. Mean Absolute Error (MAE) between responses of the actual fault and the predicted fault is used to evaluate the accuracy of the identified system. The proposed technique is easy to implement and requires a lesser number of DNN models compared to identifying an individual DNN model for each operating point to represent the ‘external system’.

Automatic speech recognition has experienced a breathtaking progress in the last few years, partially thanks to the introduction of deep neural networks into their approaches. This evolution in speech recognition systems has spread across related areas such as language and speaker recognition, where deep neural networks have noticeably improved their performance. In this PhD thesis, we have explored different approaches to the tasks of speaker and language recognition, focusing on systems where deep neural networks become part of traditional pipelines, replacing some stages or the whole system itself. Specifically, in the first experimental block, end-to-end language recognition systems based on deep neural networks are analyzed, where the neural network is used as classifier directly, without the use of any other backend but performing the language recognition task from the scores (posterior probabilities) provided by the network. Besides, these research works are focused on two architectures, convolutional neural networks and long short-term memory (LSTM) recurrent neural networks, which are less demanding in terms of computational resources due to the reduced amount of free parameters in comparison with other deep neural networks. Thus, these systems constitute an alternative to classical i-vectors, and achieve comparable results to them, especially when dealing with short utterances. In particular, we conducted experiments comparing a system based on convolutional neural networks with classical Factor Analysis GMM and i-vector reference systems, and evaluate them on two different tasks from the National Institute of Standards and Technology (NIST) Language Recognition Evaluation (LRE) 2009: one focused on language-pairs and the other, on multi-class language identification. Results shown comparable performance of the convolutional neural network based approaches and some improvements are achieved when fusing both classical and neural network approaches. We also present the experiments performed with LSTM recurrent neural networks, which have proven their ability to model time depending sequences. We evaluate our LSTM-based language recognition systems on different subsets of the NIST LRE 2009 and 2015, where LSTM systems are able to outperform the reference i-vector system, providing a model with less parameters, although more prone to overfitting and not able to generalize as well as i-vector in mismatched datasets. In the second experimental block of this Dissertation, we explore one of the most prominent applications of deep neural networks in speech processing, which is their use as feature extractors. In this kind of systems, deep neural networks are used to obtain a frame-by-frame representation of the speech signal, the so-called bottleneck feature vector, which is learned directly by the network and is then used instead of traditional acoustic features as input in language and speaker recognition systems based on i-vectors. This approach revolutionized these two fields, since they highly outperformed classical systems which had been state-of-the-art for many year (i-vector based on acoustic features). Our analysis focuses on how different configurations of the neural network used as bottleneck feature extractor, and which is trained for automatic speech recognition, influences performance of resulting features for language and speaker recognition. For the case of language recognition, we compare bottleneck features from networks that vary their depth in terms of number of hidden layers, the position of the bottleneck layer where it comprises the information and the number of units (size) of this layer, which would influence the representation obtained by the network. With the set of experiments performed on bottleneck features for speaker recognition, we analyzed the influence of the type of features used to feed the network, their pre-processing and, in general, the optimization of the network for the task of feature extraction for speaker recognition, which might not mean the optimal configuration for ASR. Finally, the third experimental block of this Thesis proposes a novel approach for language recognition, in which the neural network is used to extract a fixed-length utterance-level representation of speech segments known as embedding, able to replace the classical i-vector, and overcoming the variable length sequence of feature provided by the bottleneck features. This embedding based approach has recently shown promising results for speaker verification tasks, and our proposed system was able to outperform a strong state-of-the-art reference i-vector system on the last challenging language recognition evaluations organized by NIST in 2015 and 2017. Thus, we analyze language recognition systems based on embeddings, and explore different deep neural network architectures and data augmentation techniques to improve results of our system. In general, these embeddings are a fair competitor to the well-established i-vector pipeline which allows replacing the whole i-vector model by a deep neural network. Furthermore, the network is able to extract complementary information to the one contained in the i-vectors, even from the same input features. All this makes us consider that this contribution is an interesting research line to explore in other fields.

This paper explores the application of a deep neural network (DNN) framework to human gait analysis for injury classification. The paper aims to identify whether a subject is healthy or has an injury of the ankle, knee, hip, or heel solely based on ground reaction force plate measurements. We consider how three DNNs-the multi-layer perceptron (MLP), fully convolutional network (FCN), and residual network (ResNet)-can be applied to gait analysis when the number of trainable network parameters far exceeds the number of training samples, and benchmark their performance in this context against that of shallow neural networks. The DNN architectures outperformed unsupervised clustering models?self-organizing map and k-means clustering?by a large margin. When tested against support vector machines, which is considered the state-of-the-art approach for supervised gait classification, DNNs performed equal or better despite the propensity for overfitting. We did not find evidence that applying data augmentation to the overfitting problem via timeGAN, a generative adversarial network for time-series data generation, leads to improved classification accuracy. While DNNs are hypothesized to have intrinsic feature extraction capability, the results suggest an advantage to implementing an explicit feature extraction process as a frontend for future applications of deep neural networks: specifically, principal component analysis (PCA) preprocessing improved classification accuracy robustness in all models. In the absence of an explicit feature extraction layer, the use of dropouts in convolutional neural networks caused significant performance degradation; by contrast, the use of a PCA frontend and dropouts synergistically achieved the highest test accuracy among the DNNs studied.

The purpose of this paper is to examine the useful application of deep neural networks in stock price prediction in efficient markets and under Volatile, uncertain, complex and ambiguous (VUCA) environments, especially in the covid-induced USA financial of 2021 crisis. VUCA environments such as stock markets have made it difficult to predict stock prices. This study investigates the usefulness of deep learning architectures in stock price prediction for S&P 500’s top 3 stocks namely Apple, Microsoft and Amazon. The Bidirectional Long Short Term Memory (BLSTM) and Bidirectional Gated Recurrent Unit (BGRU) were implemented in this study and provided excellent accuracy results, the highest been 95.04% using the BGRU for Microsoft stock. The novelty of this study is the successful application of bidirectional deep neural networks to financial time series and forecasting of stock prices under financial crisis.KeywordsVolatileUncertainComplex and ambiguous environmentsStock market predictionsFinancial crisisCovid-19Deep learning technologies

Macromolecules often exchange between functional states on timescales that can be accessed with NMR spectroscopy and many NMR tools have been developed to characterise the kinetics and thermodynamics of the exchange processes, as well as the structure of the conformers that are involved. However, analysis of the NMR data that report on exchanging macromolecules often hinges on complex least-squares fitting procedures as well as human experience and intuition, which, in some cases, limits the widespread use of the methods. The applications of deep neural networks (DNNs) and artificial intelligence have increased significantly in the sciences, and recently, specifically, within the field of biomolecular NMR, where DNNs are now available for tasks such as the reconstruction of sparsely sampled spectra, peak picking, and virtual decoupling. Here we present a DNN for the analysis of chemical exchange saturation transfer (CEST) data reporting on two- or three-site chemical exchange involving sparse state lifetimes of between approximately 3–60 ms, the range most frequently observed via experiment. The work presented here focuses on the 1H CEST class of methods that are further complicated, in relation to applications to other nuclei, by anti-phase features. The developed DNNs accurately predict the chemical shifts of nuclei in the exchanging species directly from anti-phase 1HN CEST profiles, along with an uncertainty associated with the predictions. The performance of the DNN was quantitatively assessed using both synthetic and experimental anti-phase CEST profiles. The assessments show that the DNN accurately determines chemical shifts and their associated uncertainties. The DNNs developed here do not contain any parameters for the end-user to adjust and the method therefore allows for autonomous analysis of complex NMR data that report on conformational exchange.

Deep neural networks (DNNs) have evolved as a beneficial machine learning method that has been successfully used in various applications. Currently, DNN is a superior technique of extracting information from massive sets of data in a self-organized method. DNNs have different structures and parameters, which are usually produced for particular applications. Nevertheless, the training procedures of DNNs can be protracted depending on the given application and the size of the training set. Further, determining the most precise and practical structure of a deep learning method in a reasonable time is a possible problem related to this procedure. Meta-heuristics techniques, such as swarm intelligence (SI) and evolutionary computing (EC), represent optimization frames with specific theories and objective functions. These methods are adjustable and have been demonstrated their effectiveness in various applications; hence, they can optimize the DNNs models. This paper presents a comprehensive survey of the recent optimization methods (i.e., SI and EC) employed to enhance DNNs performance on various tasks. This paper also analyzes the importance of optimization methods in generating the optimal hyper-parameters and structures of DNNs in taking into consideration massive-scale data. Finally, several potential directions that still need improvements and open problems in evolutionary DNNs are identified.

IoT devices provide a rich data source that is not available in the past, which is valuable for a wide range of intelligence applications, especially deep neural network (DNN) applications that are data-thirsty. An established DNN model provides useful analysis results that can improve the operation of IoT systems in turn. The progress in distributed/federated DNN training further unleashes the potential of integration of IoT and intelligence applications. When a large number of IoT devices are deployed in different physical locations, distributed training allows training modules to be deployed to multiple edge data centers that are close to the IoT devices to reduce the latency and movement of large amounts of data. In practice, these IoT devices and edge data centers are usually owned and managed by different parties, who do not fully trust each other or have conflicting interests. It is hard to coordinate them to provide end-to-end integrity protection of the DNN construction and application with classical security enhancement tools. For example, one party may share an incomplete data set with others, or contribute a modified sub DNN model to manipulate the aggregated model and affect the decision-making process. To mitigate this risk, we propose a novel blockchain based end-to-end integrity protection scheme for DNN applications integrated with an IoT system in the edge computing environment. The protection system leverages a set of cryptography primitives to build a blockchain adapted for edge computing that is scalable to handle a large number of IoT devices. The customized blockchain is integrated with a distributed/federated DNN to offer integrity and authenticity protection services.

With the rapid development of the Internet of Things (IoT) and communication technology, deep neural network (DNN) applications, such as medical imaging, speech transcription, handwritten text recognition, have been widely used in IoT devices. However, due to resource constraints on these devices, e.g., limited memory capacity, weak computing capacity, and low battery capacity, IoT devices cannot support complicated DNN operation effectively and, thus, fail to fulfill the requirements of Quality of Service of mobile users. One promising approach is to migrate the DNN model to a remote cloud server to reduce the computing burden on IoT devices. Unfortunately, it still suffers from high delay and low bandwidth when communicating with cloud servers. Although the transmission delay of the edge server is low, its computing capacity lacks scalability and elasticity.To make matters worse, in the real world, the wireless connection between IoT devices and the cloud is intermittent, which can cause offloading failures during large-scale DNN data transmission. In this article, we describe a DNN model migration framework to overcome the abovementioned challenges, which consists of three parts: DNN model preprocessing, partition-offloading plan, and partition-uploading plan. Accordingly, we introduce the operation of the DNN migration and the available methods for each part. In addition, we improve the DNN partition-uploading plan in a multiuser edge-cloud collaborative computing environment. Finally, we highlight the important challenges of achieving more efficient DNN migration and point out the unresolved issues of DNN migration, which may shed light on future research directions.

A deep reinforcement transfer convolutional neural network for rolling bearing fault diagnosis

The paper proposes and demonstrates a Deep Convolutional Neural Network (DCNN) architecture to identify users with disguised face attempting a fraudulent Automated Teller Machine (ATM) transaction. The recent introduction of Disguised Face Identification (DFI) framework [1] proves the applicability of deep neural networks for this very problem. All the ATMs nowadays incorporate a hidden camera in them and capture the footage of their users. However, it is impossible for the police to track down the impersonators with disguised faces from the ATM footage. The proposed DCNN model is trained to identify, in real time, whether the user in the captured image is trying to cloak his identity or not. The output of the DCNN is then reported to the ATM to take appropriate steps and prevent the swindler from completing the transaction. The network is trained using a dataset of images captured in similar situations as of an ATM. The comparatively low background clutter and less viewpoint variation in the images enable the network to demonstrate high accuracy in feature extraction and classification for all the different disguises.

GONE: A generic [formula omitted](1) NoisE layer for protecting privacy of deep neural networks

With the vigorous development of MOOC open online courses, low completion rate of courses has become an important side effect factor which is affecting the promotion and application of MOOC courses in worldwide. Through analysis of edX open data sets, it is found that loss of learners is one of the factors which are leading to low completion rate of MOOC courses, and also there is a close correlation between learning behavior and final scores with loss of learners. Based on this challenge, some research results show that low completion rate of MOOC courses can be greatly improved by predicting the relationship between learning behavior and scores. This article focuses on application of Deep Neural Network (DNN) in prediction and analysis of MOOC learners' learning behavior. First, data analysis of course performance and learning behavior are discussed, then prediction model based on DNN is given for learning behavior and final scores. Second, experiment is done for DNN model, the results show that DNN can be used for analysis and prediction of complex correlation between learning behavior and final scores, thus which can realize more accurate prediction, reduce loss of learner, and carry out individualized teaching intervention to potential learners.

With the growing performance and wide application of deep neural networks (DNNs), recent years have seen enormous efforts on DNN accelerator hardware design for platforms from mobile devices to data centers. The systolic array has been a popular architectural choice for many proposed DNN accelerators with hundreds to thousands of processing elements (PEs) for parallel computing. Systolic array-based DNN accelerators for datacenter applications have high power consumption and nonuniform workload distribution, which makes power delivery network (PDN) design challenging. Server-class multicore processors have benefited from distributed on-chip voltage regulation and heterogeneous voltage regulation (HVR) for improving energy efficiency while guaranteeing power delivery integrity. This paper presents the first work on HVR-based PDN architecture and control for systolic array-based DNN accelerators. We propose to employ a PDN architecture comprising heterogeneous on-chip and off-chip voltage regulators and multiple power domains. By analyzing patterns of typical DNN workloads via a modeling framework, we propose a DNN workload-aware dynamic PDN control policy to maximize system energy efficiency while ensuring power integrity. We demonstrate significant energy efficiency improvements brought by the proposed PDN architecture, dynamic control, and power gating, which lead to a more than five-fold reduction of leakage energy and PDN energy overhead for systolic array DNN accelerators.