Deep Neural Network Representations Research Articles

FBSDE based neural network algorithms for high-dimensional quasilinear parabolic PDEs

In this paper, we propose forward and backward stochastic differential equations (FBSDEs) based deep neural network (DNN) learning algorithms for the solution of high dimensional quasi-linear parabolic partial differential equations (PDEs), which is related to the FBSDEs from the Pardoux-Peng theory. The algorithms rely on a learning process by minimizing the path-wise difference between two discrete stochastic processes, which are defined by the time discretization of the FBSDEs and the DNN representation of the PDE solution, respectively. The proposed algorithms are shown to generate DNN solution for a 100-dimensional Black–Scholes–Barenblatt equation, which is accurate in a finite region in the solution space, and has a convergence rate close to that of the Euler–Maruyama scheme used for discretizing the FBSDEs.

Beyond linear regression: mapping models in cognitive neuroscience should align with research goals

Many cognitive neuroscience studies use large feature sets to predict and interpret brain activity patterns. Feature sets take many forms, from human stimulus annotations to representations in deep neural networks. Of crucial importance in all these studies is the mapping model, which defines the space of possible relationships between features and neural data. Until recently, most encoding and decoding studies have used linear mapping models. Increasing availability of large datasets and computing resources has recently allowed some researchers to employ more flexible nonlinear mapping models; however, the question of whether nonlinear mapping models can yield meaningful scientific insights remains debated. Here, we discuss the choice of a mapping model in the context of three overarching desiderata: predictive accuracy, interpretability, and biological plausibility. We show that these desiderata do not map cleanly onto the linear/nonlinear divide; instead, each desideratum can refer to multiple research goals, each of which imposes its own constraints on the mapping model. Moreover, we argue that, instead of categorically treating the mapping models as linear or nonlinear, researchers should report the complexity of these models. We show that, in many cases, complexity provides a more accurate reflection of restrictions imposed by various research goals and outline several complexity metrics that can be used to effectively evaluate mapping models.

Deep MIMO radar target detector in Gaussian clutter

We focus on the target detection problem of the multiple-input multiple-output (MIMO) radar in clutter. Most of the existing MIMO radar detection works rely on the clutter models, which may limit their scopes of application. To address this issue, a deep learning based MIMO radar target detection framework is proposed in this paper, which utilises the powerful representation and discrimination capabilities of deep neural networks (DNNs) to improve the detection performance in a data-driven manner and does not require any prior information about the clutter distribution. In most classification problems, DNNs are trained with the cross entropy loss, which promotes DNNs to achieve higher accuracy. However, the main concern in the radar target detection problem is the probability of detection (PD) under a given probability of false alarm. In this framework, a novel loss is proposed to directly maximise the average PD of the DNN, or equivalently maximise the area under curve of the DNN. Under the DL-based MIMO radar target detection framework, a deep convolutional neural network (CNN) architecture is introduced, and an input feature of the received signal for the CNN based on the conventional generalised likelihood ratio test statistics is designed to further improve the detection performance. Finally, extensive simulation results are presented to validate the detection performance and the robustness of the proposed methods.

Urban-GAN: An artificial intelligence-aided computation system for plural urban design

The current urban design computation is mostly centered on the professional designer while ignoring the plural dimension of urban design. In addition, available public participation computational tools focus mainly on information and idea sharing, leaving the public excluded in design generation because of their lack of design expertise. To address such an issue, this study develops Urban-GAN, a plural urban design computation system, to provide new technical support for design empowerment, allowing the public to generate their own designs. The sub-symbolic representation and artificial intelligence techniques of deep convolutional neural networks, case-based reasoning, and generative adversarial networks are used to acquire and embody design knowledge as the density function, and generate design schemes with this knowledge. The system consists of an urban form database and five process models through which the user with little design expertise can select urban form cases, generate designs similar to those cases, and make design decisions. The Urban-GAN is applied to hypothetical design experiments, which show that the user is able to apply the system to successfully generate distinctive designs following the urban form “styles” in Manhattan, Portland, and Shanghai. This study further extends the discussion about the plural urban design computation to general reflections on the goals and values in AI technique application in planning and design.

Medical Professional Enhancement Using Explainable Artificial Intelligence in Fetal Cardiac Ultrasound Screening.

Diagnostic support tools based on artificial intelligence (AI) have exhibited high performance in various medical fields. However, their clinical application remains challenging because of the lack of explanatory power in AI decisions (black box problem), making it difficult to build trust with medical professionals. Nevertheless, visualizing the internal representation of deep neural networks will increase explanatory power and improve the confidence of medical professionals in AI decisions. We propose a novel deep learning-based explainable representation “graph chart diagram” to support fetal cardiac ultrasound screening, which has low detection rates of congenital heart diseases due to the difficulty in mastering the technique. Screening performance improves using this representation from 0.966 to 0.975 for experts, 0.829 to 0.890 for fellows, and 0.616 to 0.748 for residents in the arithmetic mean of area under the curve of a receiver operating characteristic curve. This is the first demonstration wherein examiners used deep learning-based explainable representation to improve the performance of fetal cardiac ultrasound screening, highlighting the potential of explainable AI to augment examiner capabilities.

DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation

We propose DeepMiner, a framework to discover interpretable representations in deep neural networks and to build explanations for medical predictions. By probing convolutional neural networks (CNNs) trained to classify cancer in mammograms, we show that many individual units in the final convolutional layer of a CNN respond strongly to diseased tissue concepts specified by the Breast Imaging-Reporting and Data System (BI-RADS) lexicon. After expert annotation of the interpretable units, our proposed method is able to generate explanations for CNN mammogram classification that are consistent with ground truth radiology reports on the Digital Database for Screening Mammography. We show that DeepMiner not only enables better understanding of the nuances of CNN classification decisions but also possibly discovers new visual knowledge relevant to medical diagnosis.

ImageNet-trained deep neural networks exhibit illusion-like response to the Scintillating grid.

Deep neural network (DNN) models for computer vision are capable of human-level object recognition. Consequently, similarities between DNN and human vision are of interest. Here, we characterize DNN representations of Scintillating grid visual illusion images in which white disks are perceived to be partially black. Specifically, we use VGG-19 and ResNet-101 DNN models that were trained for image classification and consider the representational dissimilarity ( distance in the penultimate layer) between pairs of images: one with white Scintillating grid disks and the other with disks of decreasing luminance levels. Results showed a nonmonotonic relation, such that decreasing disk luminance led to an increase and subsequently a decrease in representational dissimilarity. That is, the Scintillating grid image with white disks was closer, in terms of the representation, to images with black disks than images with gray disks. In control nonillusion images, such nonmonotonicity was rare. These results suggest that nonmonotonicity in a deep computational representation is a potential test for illusion-like response geometry in DNN models.

Deep reinforcement learning in computer vision: a comprehensive survey

Deep reinforcement learning augments the reinforcement learning framework and utilizes the powerful representation of deep neural networks. Recent works have demonstrated the remarkable successes of deep reinforcement learning in various domains including finance, medicine, healthcare, video games, robotics, and computer vision. In this work, we provide a detailed review of recent and state-of-the-art research advances of deep reinforcement learning in computer vision. We start with comprehending the theories of deep learning, reinforcement learning, and deep reinforcement learning. We then propose a categorization of deep reinforcement learning methodologies and discuss their advantages and limitations. In particular, we divide deep reinforcement learning into seven main categories according to their applications in computer vision, i.e. (i) landmark localization (ii) object detection; (iii) object tracking; (iv) registration on both 2D image and 3D image volumetric data (v) image segmentation; (vi) videos analysis; and (vii) other applications. Each of these categories is further analyzed with reinforcement learning techniques, network design, and performance. Moreover, we provide a comprehensive analysis of the existing publicly available datasets and examine source code availability. Finally, we present some open issues and discuss future research directions on deep reinforcement learning in computer vision.

Visual ensemble representations in Deep Neural Networks trained for natural object recognition

Visual ensemble representations in Deep Neural Networks trained for natural object recognition

Organizational motifs of cortical responses to objects emerge in topographic projections of deep neural networks

Visual system responses to object images have a systematic topography along the two-dimensional cortex, with selective regions for faces and places located meaningfully within a larger weaker organization of animacy and object size. Interestingly, standard deep convolutional neural networks do not have explicit topography or category-specialized mechanisms, but nonetheless learn feature tuning that has a significant correspondence with neural responses. Here we developed a method to explore the implicit topography within these deep neural network representations, by smoothly mapping the learned feature spaces with a simulated two-dimensional cortex trained using self-organizing principles. We projected a 20x20 grid of map units into the 4096-dimensional feature space of the FC6 layer of object-trained AlexNet, where nearby units on the map have similar tuning. This simulated cortex reflects a smooth spatialized representation of the data manifold in this layer, while still capturing much of the representational geometry of the original deep net layer (r=0.67). Next, we calculated the simulated cortical responses to several localizer image sets. We found a large-scale organization of animate vs inanimate response preferences. Further, clusters of face-selective and place-selective units were evident, even though the object-trained Alexnet wasn’t optimized with specialized mechanisms for these categories. As in the human brain, these emergent face regions were within animate preferring zones, while place regions were generally within inanimate zones. Finally, these results were not obtained in randomly initialized projections. Overall, these topographic projections reveal that some of the known large-scale organizational motifs of tuning properties across the human occipitotemporal cortex are implicit in the representational structure learned by deep convolutional neural networks. Broadly, this work provides evidence that the entire object-selective cortex may reflect a smoothly mapped, integrated feature space (Prince & Konkle, 2020), and introduces a new method to link hypothesized representational spaces and spatialized cortical responses.

Manufacturing process monitoring using time-frequency representation and transfer learning of deep neural networks

On-line process monitoring increases product quality, improves process stability, and lowers costs in manufacturing. This paper presents a study of using time-frequency representation and deep neural networks to enable real-time, intelligent manufacturing process monitoring. Acoustic emission (AE) signals are obtained during machine turning operations, and transformed into a time-frequency (TF) representation (image) format. Deep neural networks are then applied to the images for process classification according to the operation's spindle speed, feed rate, and depth of cut. The signals are nonstationary with frequency content varying in time. Four commonly used time-frequency analysis methods are discussed and compared, such as the short-time Fourier transform (STFT), continuous wavelet transform (CWT), Hilbert-Huang transform (HHT), and Wigner–Ville distribution (WVD). The study finds that the multi-resolution capability of the CWT technique allows it to render more accurate and richer details of the signals. Classification of machine-turning-operation processes through conventional monitoring techniques is challenging. First, there are not theories directly relating the signal characteristics to the physical process. Also, important signal components associated with the physics of the process are masked by heavy noise from the lathe. These lead to difficulties in selecting features to be extracted, and obtaining reliable data of these features. These obstacles are overcome by deep neural networks that are capable of learning and extracting meaningful features in an automatous fashion. A transfer learning approach is adopted in this study by using high-performance, representative deep neural networks previously developed for image classification and recognition, including ShuffleNet, GoogLeNet, ResNet18, ResNet50, VGG16, and DenseNet201. They are applied to the TF images obtained during the machine turning operations for process classification. The VGG16 network yields the highest classification accuracy at 92.67% when applied to processes of 12 classes, which demonstrates the potential and feasibility of the proposed method for satisfactory monitoring performance. Lastly, it is shown that the classification accuracy can be improved by using shallower networks modified from the VGG-16 network to mitigate the overfitting issue. A classification accuracy of 95.58% is achieved by removing two convolutional layers.

Topological measurement of deep neural networks using persistent homology

The inner representation of deep neural networks (DNNs) is indecipherable, which makes it difficult to tune DNN models, control their training process, and interpret their outputs. In this paper, we propose a novel approach to investigate the inner representation of DNNs through topological data analysis (TDA). Persistent homology (PH), one of the outstanding methods in TDA, was employed for investigating the complexities of trained DNNs. We constructed clique complexes on trained DNNs and calculated the one-dimensional PH of DNNs. The PH reveals the combinational effects of multiple neurons in DNNs at different resolutions, which is difficult to be captured without using PH. Evaluations were conducted using fully connected networks (FCNs) and networks combining FCNs and convolutional neural networks (CNNs) trained on the MNIST and CIFAR-10 data sets. Evaluation results demonstrate that the PH of DNNs reflects both the excess of neurons and problem difficulty, making PH one of the prominent methods for investigating the inner representation of DNNs.

Real-time electricity price forecasting of wind farms with deep neural network transfer learning and hybrid datasets

The ability to forecast real-time electricity price for wind power is key to the operation of energy markets and hedging price risks. Recent research suggests new deep neural network (DNN) architectures can capture temporal dependencies in historical price data, along with the ability to automatically extract important features of the dataset. However, most existing price prediction DNN representations still utilize basic architecture designs and either no pre-training, or simple training approaches. This work studies both the effect of transfer learning on three network representations and different source domains, as well as the mechanism of transfer learning. It is shown that transfer learning improves accuracy across all network representations. The best performance is obtained with a GRU-based architecture, termed GRU-TL, that has been pre-trained from a hybrid dataset of all wind farms in the same subzone. This model outperforms all statistical and deep learning benchmarks by an average of 6.7% in the mean absolute percent error (MAPE) metric. The underlying mechanism of transfer learning enables the pre-trained DNN representation to learn the features of the target dataset more accurately.

Robust high-dimensional memory-augmented neural networks

Traditional neural networks require enormous amounts of data to build their complex mappings during a slow training procedure that hinders their abilities for relearning and adapting to new data. Memory-augmented neural networks enhance neural networks with an explicit memory to overcome these issues. Access to this explicit memory, however, occurs via soft read and write operations involving every individual memory entry, resulting in a bottleneck when implemented using the conventional von Neumann computer architecture. To overcome this bottleneck, we propose a robust architecture that employs a computational memory unit as the explicit memory performing analog in-memory computation on high-dimensional (HD) vectors, while closely matching 32-bit software-equivalent accuracy. This is achieved by a content-based attention mechanism that represents unrelated items in the computational memory with uncorrelated HD vectors, whose real-valued components can be readily approximated by binary, or bipolar components. Experimental results demonstrate the efficacy of our approach on few-shot image classification tasks on the Omniglot dataset using more than 256,000 phase-change memory devices. Our approach effectively merges the richness of deep neural network representations with HD computing that paves the way for robust vector-symbolic manipulations applicable in reasoning, fusion, and compression.

DNNBrain: A Unifying Toolbox for Mapping Deep Neural Networks and Brains.

Deep neural networks (DNNs) have attained human-level performance on dozens of challenging tasks via an end-to-end deep learning strategy. Deep learning allows data representations that have multiple levels of abstraction; however, it does not explicitly provide any insights into the internal operations of DNNs. Deep learning's success is appealing to neuroscientists not only as a method for applying DNNs to model biological neural systems but also as a means of adopting concepts and methods from cognitive neuroscience to understand the internal representations of DNNs. Although general deep learning frameworks, such as PyTorch and TensorFlow, could be used to allow such cross-disciplinary investigations, the use of these frameworks typically requires high-level programming expertise and comprehensive mathematical knowledge. A toolbox specifically designed as a mechanism for cognitive neuroscientists to map both DNNs and brains is urgently needed. Here, we present DNNBrain, a Python-based toolbox designed for exploring the internal representations of DNNs as well as brains. Through the integration of DNN software packages and well-established brain imaging tools, DNNBrain provides application programming and command line interfaces for a variety of research scenarios. These include extracting DNN activation, probing and visualizing DNN representations, and mapping DNN representations onto the brain. We expect that our toolbox will accelerate scientific research by both applying DNNs to model biological neural systems and utilizing paradigms of cognitive neuroscience to unveil the black box of DNNs.

Deep Neural Networks as a computational model for early vision: Lateral masking and contour integration

Background: The deep neural network (DNN) models developed for image classification have been recently suggested as biologically inspired models for different brain functions. Here we apply a standard visual DNN model to explore early-vision mechanisms of integration, as reflected in the psychophysical phenomena of lateral masking, contrast summation, and contour integration. Methods: We used the standard ImageNet-trained VGG network model. The model correlate of perceptual distance was the L2 distance between the average response to images corresponding to different stimuli conditions. Higher values of the metric (indicating larger changes in the DNN representation) correspond to better discrimination. Results: For lateral masking, the model produced a close match to the basic behavioral data (Polat & Sagi 1993), with a facilitation of ~0.4 log units at 2.5 wave-length distance, as well as: (1) inhibition for very close flankers, (2) no facilitation for orthogonal flankers and decreased facilitation with deviation from collinearity, (3) more facilitation of a vertical configuration compared to horizontal and oblique, (4) scaling of the effects with wave-length, and more. These results were obtained with a growing facilitation and longer interaction range from the mid layers of the model and up, indicating hierarchical integration that possibly substitutes for the assumed lateral interactions in V1. For contrast summation we replicated the configuration dependent (smooth but not jagged) summation (response vs number of patches along a Gabor contour) power-law (Bonneh and Sagi 1998). For Gabor contour detection in noise (Field et. al. 1993) the model showed the known effects of spacing, contour smoothness, and scaling. For noise detection in natural images (Alam et al. 2014), perceptual thresholds were strongly correlated with model predictions (R=0.78, N=1080 images). Conclusions: These findings demonstrate effortless replication in a DNN of classic findings concerning early human visual processing, suggesting convergent evolution of biological and artificial vision.

Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations.

The reconstruction of visual information from human brain activity is a very important research topic in brain decoding. Existing methods ignore the structural information underlying the brain activities and the visual features, which severely limits their performance and interpretability. Here, we propose a hierarchically structured neural decoding framework by using multitask transfer learning of deep neural network (DNN) representations and a matrix-variate Gaussian prior. Our framework consists of two stages, Voxel2Unit and Unit2Pixel. In Voxel2Unit, we decode the functional magnetic resonance imaging (fMRI) data to the intermediate features of a pretrained convolutional neural network (CNN). In Unit2Pixel, we further invert the predicted CNN features back to the visual images. Matrix-variate Gaussian prior allows us to take into account the structures between feature dimensions and between regression tasks, which are useful for improving decoding effectiveness and interpretability. This is in contrast with the existing single-output regression models that usually ignore these structures. We conduct extensive experiments on two real-world fMRI data sets, and the results show that our method can predict CNN features more accurately and reconstruct the perceived natural images and faces with higher quality.

Neural Encoding and Decoding With Distributed Sentence Representations.

Building computational models to account for the cortical representation of language plays an important role in understanding the human linguistic system. Recent progress in distributed semantic models (DSMs), especially transformer-based methods, has driven advances in many language understanding tasks, making DSM a promising methodology to probe brain language processing. DSMs have been shown to reliably explain cortical responses to word stimuli. However, characterizing the brain activities for sentence processing is much less exhaustively explored with DSMs, especially the deep neural network-based methods. What is the relationship between cortical sentence representations against DSMs? What linguistic features that a DSM catches better explain its correlation with the brain activities aroused by sentence stimuli? Could distributed sentence representations help to reveal the semantic selectivity of different brain areas? We address these questions through the lens of neural encoding and decoding, fueled by the latest developments in natural language representation learning. We begin by evaluating the ability of a wide range of 12 DSMs to predict and decipher the functional magnetic resonance imaging (fMRI) images from humans reading sentences. Most models deliver high accuracy in the left middle temporal gyrus (LMTG) and left occipital complex (LOC). Notably, encoders trained with transformer-based DSMs consistently outperform other unsupervised structured models and all the unstructured baselines. With probing and ablation tasks, we further find that differences in the performance of the DSMs in modeling brain activities can be at least partially explained by the granularity of their semantic representations. We also illustrate the DSM's selectivity for concept categories and show that the topics are represented by spatially overlapping and distributed cortical patterns. Our results corroborate and extend previous findings in understanding the relation between DSMs and neural activation patterns and contribute to building solid brain-machine interfaces with deep neural network representations.

On the Stability Analysis of Deep Neural Network Representations of an Optimal State Feedback

Recent works have shown that the optimal state feedback for deterministic, nonlinear autonomous systems can be approximated by deep neural networks. In this article, we consider the stability of nonlinear systems controlled by such a network representation of the optimal feedback. First, we show that principal methods from stability theory readily applies. We then propose a novel method based on differential algebra techniques to study the robustness of a nominal trajectory with respect to perturbations of the initial conditions. It is, to the best of our knowledge, the first time that differential algebraic techniques are shown to allow for the high-order analysis of motion stability for a nonlinear system in general and for a neurocontrolled system in particular. We exemplify the proposed method in the 2-D case of the optimal control of a quadcopter and demonstrate it for different neural network architectures.

Hidden representations in deep neural networks: Part 2. Regression problems

Deep neural networks are an important class of machine learning models useful for representing complex input-output relationships. While their recent success is unparalleled, so is the inability to explain their internal representations. In this second part of a two-part series, we focus on understanding the hidden representations of deep neural networks and the underlying mechanisms for regression problems. We highlight challenges associated with deep neural networks with simple models that help us gain insight into the functioning of the hidden layers and the mechanism of the operation of the network. The article is structured in a tutorial-like fashion for the benefit of new practitioners so that they can appreciate nuances and the pitfalls involved in developing a deep neural network models for regression problems.