Deep Neural Network Representations Research Articles

A Framework for Adapting DNN Speaker Embedding Across Languages

Language mismatch remains a major hindrance to the extensive deployment of speaker verification (SV) systems. Current language adaptation methods in SV mainly rely on linear projection in embedding space; i.e., adaptation is carried out after the speaker embeddings have been created, which underutilizes the powerful representation of deep neural networks. This article proposes a maximum mean discrepancy (MMD) based framework for adapting deep neural network (DNN) speaker embedding across languages, featuring multi-level domain loss, separate batch normalization, and consistency regularization. We refer to the framework as MSC. We show that (1) minimizing domain discrepancy at both frame- and utterance-levels performs significantly better than at utterance-level alone; (2) separating the source-domain data from the target-domain in batch normalization improves adaptation performance; and (3) data augmentation can be utilized in the unlabelled target-domain through consistency regularization. By combining these findings, we achieve an EER of 8.69% and 7.95% in NIST SRE 2016 and 2018, respectively, which are significantly better than the previously proposed DNN adaptation methods. Our framework also works well with backend adaptation. By combining the proposed framework with backend adaptation, we achieve an 11.8% improvement over the backend adaptation in SRE18. When applying our framework to a 121-layer Densenet, we achieved an EER of 7.81% and 7.02% in NIST SRE 2016 and 2018, respectively.

Feature Consistency Training With JPEG Compressed Images

Deep neural networks (DNNs) are recently found to be vulnerable to JPEG compression artifacts, which distort the feature representations of DNNs leading to serious accuracy degradation. Most existing training methods which aim to address this problem add compressed images to the training data to enhance the robustness of DNNs. However, their improvements are limited since these methods usually regard the compressed images as new training samples instead of distorted samples. The feature distortions between the raw images and the compressed images are not investigated. In this work, we propose a new training method, called Feature Consistency Training, that is designed to minimize the feature distortions caused by JPEG artifacts. At each training iteration, we simultaneously input a raw image and its compressed version with a randomly sampled quality into a DNN model and extract the features from the internal layers. By adding feature consistency constraint to the objective function, the feature distortions in the representation space are minimized in order to learn robust filters. Besides, we present a residual mapping block which takes the quality factor of the compressed image as an additional information to further reduce the feature distortion. Extensive experiments demonstrate that our method outperforms several existed training methods on JPEG compressed images. Furthermore, DNN models trained by our method are found to be more robust to unseen distortions.

A Standardized PMML Format for Representing Convolutional Neural Networks with Application to Defect Detection

Convolutional neural networks are becoming a popular tool for image processing in the engineering and manufacturing sectors. However, managing the storage and distribution of trained models is still a difficult task, partially due to the lack of standardized methods for deep neural network representation. Additionally, the interoperability between different machine learning frameworks remains poor. This paper seeks to address this issue by proposing a standardized format for convolutional neural networks, based on the Predictive Model Markup Language (PMML). A new standardized schema is proposed to represent a range of convolutional neural networks, including classification, regression and semantic segmentation systems. To demonstrate the practical application of this standard, a semantic segmentation model, which is trained to detect casting defects in Xray images, is represented in the proposed PMML format. A high-performance scoring engine is developed to evaluate images and videos against the PMML model. The utility of the proposed format and the scoring engine is evaluated by benchmarking the performance of the defect detection models on a range of different computational platforms.

Hidden representations in deep neural networks: Part 1. Classification problems

Deep neural networks have evolved into a powerful tool applicable for a wide range of problems. However, a clear understanding of their internal mechanism has not been developed satisfactorily yet. Factors such as the architecture, number of hidden layers and neurons, and activation function are largely determined in a guess-and-test manner that is reminiscent of alchemy more than of chemistry. In this paper, we attempt to address these concerns systematically using carefully chosen model systems to gain insights for classification problems. We show how wider networks result in several simple patterns identified on the input space, while deeper networks result in more complex patterns. We show also the transformation of input space by each layer and identify the origin of techniques such as transfer learning, weight normalization and early stopping. This paper is an initial step towards a systematic approach to uncover key hidden properties that can be exploited to improve the performance and understanding of deep neural networks.

Transfer learning of deep neural network representations for fMRI decoding

BackgroundDeep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data. New methodIn this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images. ResultsThe fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance. Comparison with existing methodsThe decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone. ConclusionIn this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.

Front-End Smart Visual Sensing and Back-End Intelligent Analysis: A Unified Infrastructure for Economizing the Visual System of City Brain

The visual data, which are acquired from the ubiquitous visual sensors deployed in metropolitans, are of great value and paramount significance to enhance the effectiveness and pursue the future development of smart cities. In this paper, the essential building blocks of the unified visual data management and analysis infrastructure that serve as the foundation for the economical visual system in the city brain, are introduced to facilitate the utilization of the visual signal in the artificial intelligence era. In particular, we start by the discussion of the front-end smart visual sensing in the context of economical communication and service with the heterogeneous network, and the functionalities and necessities of compact visual feature and deep learning model representations are detailed. Subsequently, the utilities of the infrastructure are demonstrated through two intelligent applications at the back-end, including vehicle re-identification and person re-identification. The standardizations regarding compact feature and deep neural network representations, which are regarded as the key ingredients in this infrastructure and greatly facilitate the construction of the visual system in the city brain, are also discussed. Finally, we envision how the potential issues regarding the economical visual communications for future smart cities might be pragmatically approached within this unified infrastructure.

Compact and Computationally Efficient Representation of Deep Neural Networks.

At the core of any inference procedure, deep neural networks are dot product operations, which are the component that requires the highest computational resources. For instance, deep neural networks, such as VGG-16, require up to 15-G operations in order to perform the dot products present in a single forward pass, which results in significant energy consumption and thus limits their use in resource-limited environments, e.g., on embedded devices or smartphones. One common approach to reduce the complexity of the inference is to prune and quantize the weight matrices of the neural network. Usually, this results in matrices whose entropy values are low, as measured relative to the empirical probability mass distribution of its elements. In order to efficiently exploit such matrices, one usually relies on, inter alia, sparse matrix representations. However, most of these common matrix storage formats make strong statistical assumptions about the distribution of the elements; therefore, cannot efficiently represent the entire set of matrices that exhibit low-entropy statistics (thus, the entire set of compressed neural network weight matrices). In this paper, we address this issue and present new efficient representations for matrices with low-entropy statistics. Alike sparse matrix data structures, these formats exploit the statistical properties of the data in order to reduce the size and execution complexity. Moreover, we show that the proposed data structures can not only be regarded as a generalization of sparse formats but are also more energy and time efficient under practically relevant assumptions. Finally, we test the storage requirements and execution performance of the proposed formats on compressed neural networks and compare them to dense and sparse representations. We experimentally show that we are able to attain up to ×42 compression ratios, ×5 speed ups, and ×90 energy savings when we lossless convert the state-of-the-art networks, such as AlexNet, VGG-16, ResNet152, and DenseNet, into the new data structures and benchmark their respective dot product.

State-Space Representations of Deep Neural Networks.

This letter deals with neural networks as dynamical systems governed by finite difference equations. It shows that the introduction of -many skip connections into network architectures, such as residual networks and additive dense networks, defines th order dynamical equations on the layer-wise transformations. Closed-form solutions for the state-space representations of general th order additive dense networks, where the concatenation operation is replaced by addition, as well as th order smooth networks, are found. The developed provision endows deep neural networks with an algebraic structure. Furthermore, it is shown that imposing th order smoothness on network architectures with -many nodes per layer increases the state-space dimension by a multiple of , and so the effective embedding dimension of the data manifold by the neural network is -many dimensions. It follows that network architectures of these types reduce the number of parameters needed to maintain the same embedding dimension by a factor of when compared to an equivalent first-order, residual network. Numerical simulations and experiments on CIFAR10, SVHN, and MNIST have been conducted to help understand the developed theory and efficacy of the proposed concepts.

Multimodal Deep Learning for Music Genre Classification

Music genre labels are useful to organize songs, albums, and artists into broader groups that share similar musical characteristics. In this work, an approach to learn and combine multimodal data representations for music genre classification is proposed. Intermediate representations of deep neural networks are learned from audio tracks, text reviews, and cover art images, and further combined for classification. Experiments on single and multi-label genre classification are then carried out, evaluating the effect of the different learned representations and their combinations. Results on both experiments show how the aggregation of learned representations from different modalities improves the accuracy of the classification, suggesting that different modalities embed complementary information. In addition, the learning of a multimodal feature space increases the performance of pure audio representations, which may be specially relevant when the other modalities are available for training, but not at prediction time. Moreover, a proposed approach for dimensionality reduction of target labels yields major improvements in multi-label classification not only in terms of accuracy, but also in terms of the diversity of the predicted genres, which implies a more fine-grained categorization. Finally, a qualitative analysis of the results sheds some light on the behavior of the different modalities on the classification task.

Deep learning and the AdS/CFT correspondence

We present a deep neural network representation of the AdS/CFT correspondence, and demonstrate the emergence of the bulk metric function via the learning process for given data sets of response in boundary quantum field theories. The emergent radial direction of the bulk is identified with the depth of the layers, and the network itself is interpreted as a bulk geometry. Our network provides a data-driven holographic modeling of strongly coupled systems. With a scalar $\phi^4$ theory with unknown mass and coupling, in unknown curved spacetime with a black hole horizon, we demonstrate our deep learning (DL) framework can determine them which fit given response data. First, we show that, from boundary data generated by the AdS Schwarzschild spacetime, our network can reproduce the metric. Second, we demonstrate that our network with experimental data as an input can determine the bulk metric, the mass and the quadratic coupling of the holographic model. As an example we use the experimental data of magnetic response of a strongly correlated material Sm$_{0.6}$Sr$_{0.4}$MnO$_3$. This AdS/DL correspondence not only enables gravity modeling of strongly correlated systems, but also sheds light on a hidden mechanism of the emerging space in both AdS and DL.

Deep Learning for Tuning Optical Beamforming Networks

In communication between planes and satellites, Optical Beamforming Networks (OBFNs), which rely on many small and flat Phased Array Antennas (PAAs), need to be tuned in order to receive signals from specific angles. In this paper, we develop a deep neural network representation of tuning OBFNs. The problem of tuning an OBFN is in many aspects similar to training a deep neural network. We present a way to exploit the special structure of OBFNs into deep neural network and an algorithm for tuning OBFNs based on feedback that can be easily measured in real system. Training data, which consists of full signals, can be measured, and therefore is used in this paper. For pilot signals, the desired signal is known explicitly. Given the configuration of OBFNs and all nominal parameters required, it was verified in simulation that the deep neural network can be used to tune large scale OBFNs for any desired delays.

Shared spatiotemporal category representations in biological and artificial deep neural networks.

Visual scene category representations emerge very rapidly, yet the computational transformations that enable such invariant categorizations remain elusive. Deep convolutional neural networks (CNNs) perform visual categorization at near human-level accuracy using a feedforward architecture, providing neuroscientists with the opportunity to assess one successful series of representational transformations that enable categorization in silico. The goal of the current study is to assess the extent to which sequential scene category representations built by a CNN map onto those built in the human brain as assessed by high-density, time-resolved event-related potentials (ERPs). We found correspondence both over time and across the scalp: earlier (0–200 ms) ERP activity was best explained by early CNN layers at all electrodes. Although later activity at most electrode sites corresponded to earlier CNN layers, activity in right occipito-temporal electrodes was best explained by the later, fully-connected layers of the CNN around 225 ms post-stimulus, along with similar patterns in frontal electrodes. Taken together, these results suggest that the emergence of scene category representations develop through a dynamic interplay between early activity over occipital electrodes as well as later activity over temporal and frontal electrodes.

Sharpening of Hierarchical Visual Feature Representations of Blurred Images.

The robustness of the visual system lies in its ability to perceive degraded images. This is achieved through interacting bottom-up, recurrent, and top-down pathways that process the visual input in concordance with stored prior information. The interaction mechanism by which they integrate visual input and prior information is still enigmatic. We present a new approach using deep neural network (DNN) representation to reveal the effects of such integration on degraded visual inputs. We transformed measured human brain activity resulting from viewing blurred images to the hierarchical representation space derived from a feedforward DNN. Transformed representations were found to veer toward the original nonblurred image and away from the blurred stimulus image. This indicated deblurring or sharpening in the neural representation, and possibly in our perception. We anticipate these results will help unravel the interplay mechanism between bottom-up, recurrent, and top-down pathways, leading to more comprehensive models of vision.

Deep Potential: A General Representation of a Many-Body Potential Energy Surface

We present a simple, yet general, end-to-end deep neural network representation of the potential energy surface for atomic and molecular systems. This methodology, which we call Deep Potential, is "first-principle" based, in the sense that no ad hoc approximations or empirical fitting functions are required. The neural network structure naturally respects the underlying symmetries of the systems. When tested on a wide variety of examples, Deep Potential is able to reproduce the original model, whether empirical or quantum mechanics based, within chemical accuracy. The computational cost of this new model is not substantially larger than that of empirical force fields. In addition, the method has promising scalability properties. This brings us one step closer to being able to carry out molecular simulations with accuracy comparable to that of quantum mechanics models and computational cost comparable to that of empirical potentials.

Low-resource Multi-task Audio Sensing for Mobile and Embedded Devices via Shared Deep Neural Network Representations

Continuous audio analysis from embedded and mobile devices is an increasingly important application domain. More and more, appliances like the Amazon Echo, along with smartphones and watches, and even research prototypes seek to perform multiple discriminative tasks simultaneously from ambient audio; for example, monitoring background sound classes (e.g., music or conversation), recognizing certain keywords (‘Hey Siri' or ‘Alexa'), or identifying the user and her emotion from speech. The use of deep learning algorithms typically provides state-of-the-art model performances for such general audio tasks. However, the large computational demands of deep learning models are at odds with the limited processing, energy and memory resources of mobile, embedded and IoT devices. In this paper, we propose and evaluate a novel deep learning modeling and optimization framework that specifically targets this category of embedded audio sensing tasks. Although the supported tasks are simpler than the task of speech recognition, this framework aims at maintaining accuracies in predictions while minimizing the overall processor resource footprint. The proposed model is grounded in multi-task learning principles to train shared deep layers and exploits, as input layer, only statistical summaries of audio filter banks to further lower computations. We find that for embedded audio sensing tasks our framework is able to maintain similar accuracies, which are observed in comparable deep architectures that use single-task learning and typically more complex input layers. Most importantly, on an average, this approach provides almost a 2.1× reduction in runtime, energy, and memory for four separate audio sensing tasks, assuming a variety of task combinations.

Top-down neural processing that supplements missing image features revealed by brain decoding with deep neural network representation

Top-down neural processing that supplements missing image features revealed by brain decoding with deep neural network representation

Face Representations in Deep Convolutional Neural Networks

Face Representations in Deep Convolutional Neural Networks

A Comparison Among Different Numeric Representations in Deep Convolution Neural Networks

A Comparison Among Different Numeric Representations in Deep Convolution Neural Networks

Visual features versus categories: Explaining object representations in primate IT and deep neural networks with weighted representational modeling

Visual features versus categories: Explaining object representations in primate IT and deep neural networks with weighted representational modeling

Articulated Pose Estimation Using Hierarchical Exemplar-Based Models

Exemplar-based models have achieved great success on localizing the parts of semi-rigid objects. However, their efficacy on highly articulated objects such as humans is yet to be explored. Inspired by hierarchical object representation and recent application of Deep Convolutional Neural Networks (DCNNs) on human pose estimation, we propose a novel formulation that incorporates both hierarchical exemplar-based models and DCNNs in the spatial terms. Specifically, we obtain more expressive spatial models by assuming independence between exemplars at different levels in the hierarchy; we also obtain stronger spatial constraints by inferring the spatial relations between parts at the same level. As our method strikes a good balance between expressiveness and strength of spatial models, it is both effective and generalizable, achieving state-of-the-art results on different benchmarks: Leeds Sports Dataset and CUB-200-2011.