Deep Learning Community Research Articles

NEARSIDE: Structured kNowledge Extraction frAmework from SpecIes DEscriptions

Species descriptions are stored in textual form in corpora such as in floras and faunas, but this large amount of information cannot be used directly by algorithms, nor can it be linked to other data sources. The production of knowledge bases expressing structured data can benefit from collaborative and easy-to-use platforms like Xper3 (Vignes-Lebbe et al. 2017, Kerner and Vignes 2019, Saucède et al. 2021) but is very time-consuming at the human level. It is therefore mandatory for this task to make the information contained in species descriptions measurable and compatible with computer techniques. One of the most used data structures on the web and by the deep learning community is the triplet structure. Each piece of information is represented by a set of 3 elements (subject, predicate, object). One of the first steps towards species information accessibility is developing a text-to-triplet model, also known as text-to-graph, for monograph descriptions. In this work, we developed NEARSIDE, a text-to-graph model adapted to biology corpora to create normalized morphological characteristic knowledge bases for species descriptions. In Natural Language Processing, deep learning models have proven to be effective in extracting knowledge from open domain corpora (Lample et al. 2016, Sutskever et al. 2014), especially since the emergence of attention-based models (Devlin et al. 2019b, Devlin et al. 2019a). Several works have been made also on biomedical corpora (Fries et al. 2017,Cho and Lee 2019). In our case, we propose a model adapted to floras. Fully supervised deep learning models require a large amount of annotated data for training, nevertheless, the annotation process for the text-to-triplet task implies an expensive human intervention. Distant supervision is a technique that can be used to reduce this cost. This paradigm uses a small annotated glossary to project classes at the word level on a new complex and longer text (see Fig. 1). Named Entity Recognition (NER) is an Natural Language Processing (NLP) task that consists of extracting and classifying words of interest from a text (Sutskever et al. 2014, Devlin et al. 2019b, Lample et al. 2016), while triplet extraction can be compared to the Relation Extraction task (RE) which consists of extracting the words and the semantic relations between pairs of words. Distantly supervised NER is an often studied subject in the literature in comparison to distantly supervised RE (Liang et al. 2020, Meng et al. 2021) simply because NER is a subtask to RE and distant annotations generation is less expensive for the NER task (see Fig. 2). Our first contribution is creating a distantly annotated species description dataset for Named Entity Recognition with a well-balanced test set that allows us to bypass several biases that can be induced by the distant annotation and that are often observed in NER datasets (Taillé et al. 2021). In this dataset, each word of interest will be classified into one of 15 classes, each class being a specific kind of organ or descriptor. Our second contribution is proposing a distantly supervised model trained on our dataset, since fauna and flora corpora are particularly long and use a very specific technical vocabulary. We develop a context-oriented model adapted to this data by pretraining the language model. Thus the encoder of our model provides contextualized vectors for each extracted word that can be used to measure description similarities between different species. Our model reaches 96% accuracy in named entity classification on the test set. Our third contribution is the triplet construction module that can directly be applied to our model's outputs. This module is based on class dependency rules that are inspired by Xper3’s data representation format (see Fig. 3). Finally, NEARSIDE is an end-to-end structured knowledge extraction framework from unstructured species description corpora, that can be applied to several data sources. Thus making species descriptions from different corpora easily linked, compared and measured.

A convergence analysis of Nesterov’s accelerated gradient method in training deep linear neural networks

As the training process of deep neural networks involves expensive computational cost, speeding up the convergence is of great importance. Nesterov’s accelerated gradient (NAG) is one of the most popular accelerated optimizers in the deep learning community, which often exhibits improved convergence performance over gradient descent (GD) in practice. However, theoretical investigations of NAG mainly focus on the convex setting. Since the optimization landscape of the neural network is non-convex, little is known about the convergence and acceleration of NAG. Nowadays, some works make progress towards understanding the convergence of NAG in training over-parameterized neural networks, where the number of the parameters exceeds that of the training instances. Nonetheless, previous studies are limited to the two-layer neural network, which are far from explaining the remarkable success of NAG in optimizing deep neural networks. In this paper, we investigate the convergence of NAG in training two architectures of deep linear networks: deep fully-connected linear neural networks and deep linear ResNets. Based on the over-parameterization regime, we first analyze the residual dynamics induced by the training trajectory of NAG for a deep fully-connected linear neural network under random Gaussian initialization. Our results show that NAG can converge to the global minimum at a (1-O(1/κ))t rate when the width is near-linear in the depth of the network, where t is the number of iterations and κ>1 is a constant depending on the condition number of the feature matrix. Compared to the (1-O(1/κ))t rate of GD, NAG achieves an acceleration over GD. For deep linear ResNets, we utilize the same analytical approach and obtain a similar convergence result, while the width requirement is independent of the depth. To the best of our knowledge, these are the first theoretical guarantees for the convergence and acceleration of NAG in training deep neural networks. Numerical results show the acceleration of NAG compared to GD in terms of iterations. In addition, we conduct experiments to evaluate the effect of the depth on the convergence rate of NAG, which validate our derived conditions of the width. We hope our results may shed light on understanding the optimization behavior of NAG for modern deep neural networks.

Practical Implementation and Challenges of Artificial Intelligence-Driven Electronic Health Record Evaluation: Protected Health Information.

Detecting protected health information in electronic health record systems is often an early step in health care analytics, and it is a nontrivial problem. Specific challenges include finding clinician names and diseases, which lack a fixed format and are often context-dependent. The general problem of finding entities, termed named-entity recognition, has received a substantial amount of attention in the natural language processing and deep learning communities. This paper begins by outlining recent methods for finding protected health information, and it then introduces a hybrid system which combines regular expressions with a natural language processing framework called FLAIR. FLAIR is open-source, it includes state-of-the-art deep learning models, and it supports straightforward development of new models for language tasks including named-entity recognition. Finally, there is a discussion of how to apply the system to structured text in a database table as well as unstructured text in clinical notes.

Collective intelligence for deep learning: A survey of recent developments

In the past decade, we have witnessed the rise of deep learning to dominate the field of artificial intelligence. Advances in artificial neural networks alongside corresponding advances in hardware accelerators with large memory capacity, together with the availability of large datasets enabled practitioners to train and deploy sophisticated neural network models that achieve state-of-the-art performance on tasks across several fields spanning computer vision, natural language processing, and reinforcement learning. However, as these neural networks become bigger, more complex, and more widely used, fundamental problems with current deep learning models become more apparent. State-of-the-art deep learning models are known to suffer from issues that range from poor robustness, inability to adapt to novel task settings, to requiring rigid and inflexible configuration assumptions. Collective behavior, commonly observed in nature, tends to produce systems that are robust, adaptable, and have less rigid assumptions about the environment configuration. Collective intelligence, as a field, studies the group intelligence that emerges from the interactions of many individuals. Within this field, ideas such as self-organization, emergent behavior, swarm optimization, and cellular automata were developed to model and explain complex systems. It is therefore natural to see these ideas incorporated into newer deep learning methods. In this review, we will provide a historical context of neural network research’s involvement with complex systems, and highlight several active areas in modern deep learning research that incorporate the principles of collective intelligence to advance its capabilities. We hope this review can serve as a bridge between the complex systems and deep learning communities.

EEG-GAT: Graph Attention Networks for Classification of Electroencephalogram (EEG) Signals.

Graph neural networks (GNN) are an emerging framework in the deep learning community. In most GNN applications, the graph topology of data samples is provided in the dataset. Specifically, the graph shift operator (GSO), which could be adjacency, graph Laplacian, or their normalizations, is known a priori. However we often have no knowledge of the grand-truth graph topology underlying real-world datasets. One example of this is to extract subject-invariant features from physiological electroencephalogram (EEG) to predict a cognitive task. Previous methods use electrode sites to represent a node in the graph and connect them in various ways to hand-engineer a GSO e.g., i) each pair of electrode sites is connected to form a complete graph, ii) a specific number of electrode sites are connected to form a k-nearest neighbor graph, iii) each pair of electrode site is connected only if the Euclidean distance is within a heuristic threshold. In this paper, we overcome this limitation by parameterizing the GSO using a multi-head attention mechanism to explore the functional neural connectivity subject to a cognitive task between different electrode sites, and simultaneously learn the unsupervised graph topology in conjunction with the parameters of graph convolutional kernels.

Intrusion prevention system using convolutional neural network for wireless sensor network

Now-a-days, there is exponential growth in the field of wireless sensor network. In wireless sensor networks (WSN’s), most of communication happen through wireless media hence probability of attacks increases drastically. With the help of intrusion prevention system, we can classify user activities into two categories, normal and suspicious activity. There is need to design effective intrusion prevention system by exploring deep learning for WSN. This research aims to deal with proposing algorithms and techniques for intrusion prevention system using deep packet inspection based on deep learning. In this, we have proposed deep learning model using convolutional neural network. The proposed model includes two steps, intrusion detection and intrusion prevention. The proposed model learns useful feature representations from large amount of labeled data and then classifies them. In this work, convolutional neural network is used to prevent intrusion for WSN. To evaluate and check the effectiveness of the proposed system, the wireless sensor network dataset (WSNDS) dataset is used and the tests are performed. The test results show that proposed system has an accuracy of 97% and works better than existing system. The proposed work can be used as future benchmark for the deep learning and intrusion prevention research communities.

Unraveling the Hidden Environmental Impacts of AI Solutions for Environment Life Cycle Assessment of AI Solutions

In the past ten years, artificial intelligence has encountered such dramatic progress that it is now seen as a tool of choice to solve environmental issues and, in the first place, greenhouse gas emissions (GHG). At the same time, the deep learning community began to realize that training models with more and more parameters require a lot of energy and, as a consequence, GHG emissions. To our knowledge, questioning the complete net environmental impacts of AI solutions for the environment (AI for Green) and not only GHG, has never been addressed directly. In this article, we propose to study the possible negative impacts of AI for Green. First, we review the different types of AI impacts; then, we present the different methodologies used to assess those impacts and show how to apply life cycle assessment to AI services. Finally, we discuss how to assess the environmental usefulness of a general AI service and point out the limitations of existing work in AI for Green.

EfficientUNet: Modified encoder‐decoder architecture for the lung segmentation in chest x‐ray images

AbstractChest x‐rays are a widely used medical imaging technique for the early detection of pulmonary diseases. However, the increasing workload and shortage of radiologists have motivated the research community to explore the possibilities in automated diagnosis. Segmentation is the most critical and essential procedure to improve the automatic diagnosis. Irregular lung shape and size with the overlapped lung regions make the identification of lung boundaries difficult. Further, the dense abnormalities often have high‐intensity values that can be interpreted as lung boundaries. UNet is the most popular segmentation architecture among the deep learning community. Despite its high performance, there is scope for improvement. Most of the proposed segmentation models for lung segmentation in chest x‐rays are trained and tested on a same dataset. It is reported that the model trained on one dataset fails when tested on another dataset. The aim of this study is to develop an efficient architecture for accurate lung segmentation in chest x‐rays. The proposed model uses the pre‐trained encoder, and the decoder uses residual learning and batch normalization to improve the performance. The publicly available Shenzhen, Montgomery County, and the NIH datasets are used for the evaluation of the proposed model. It is also validated by cross‐dataset generalization. The proposed model obtained the 96.14% DSC and 92.67% JI value on the Shenzhen dataset, while 98.48% DSC and 97.01% JI value on the MC dataset. For cross‐dataset evaluation, the EfficientUNet reported the DSC value of 95.97% and JI value of 92.34% for the MC dataset. While the DSC value of 94.02% and JI value of 88.86% on the NIH dataset. The obtained results across the different metrics show the efficiency of the proposed model. Unlike, the other state‐of‐the‐art models, the proposed model is evaluated on the lungs that are highly irregular in shape and size due to different pulmonary abnormalities.

PBIL for optimizing inception module in convolutional neural networks

AbstractInception module is one of the most used variants in convolutional neural networks. It has a large portfolio of success cases in computer vision. In the past years, diverse inception flavours, differing in the number of branches, the size and the number of the kernels, have appeared in the scientific literature. They are proposed based on the expertise of the practitioners without any optimization process. In this work, an implementation of population-based incremental learning is proposed for automatic optimization of the hyperparameters of the inception module. This hyperparameters optimization undertakes classification of the MNIST database of handwritten digit images. This problem is widely used as a benchmark in classification, and therefore, the learned best configurations for the Inception module will be of wide use in the deep learning community. In order to reduce the carbon footprint of the optimization process, policies for reducing the redundant evaluations have been undertaken. As a consequence of this work, an evaluation of configurations of the inception module and a mechanism for optimizing hyperparameters in deep learning architectures are stated.

TSGB: Target-Selective Gradient Backprop for Probing CNN Visual Saliency.

The explanation for deep neural networks has drawn extensive attention in the deep learning community over the past few years. In this work, we study the visual saliency, a.k.a. visual explanation, to interpret convolutional neural networks. Compared to iteration based saliency methods, single backward pass based saliency methods benefit from faster speed, and they are widely used in downstream visual tasks. Thus, we focus on single backward pass based methods. However, existing methods in this category struggle to successfully produce fine-grained saliency maps concentrating on specific target classes. That said, producing faithful saliency maps satisfying both target-selectiveness and fine-grainedness using a single backward pass is a challenging problem in the field. To mitigate this problem, we revisit the gradient flow inside the network, and find that the entangled semantics and original weights may disturb the propagation of target-relevant saliency. Inspired by those observations, we propose a novel visual saliency method, termed Target-Selective Gradient Backprop (TSGB), which leverages rectification operations to effectively emphasize target classes and further efficiently propagate the saliency to the image space, thereby generating target-selective and fine-grained saliency maps. The proposed TSGB consists of two components, namely, TSGB-Conv and TSGB-FC, which rectify the gradients for convolutional layers and fully-connected layers, respectively. Extensive qualitative and quantitative experiments on the ImageNet and Pascal VOC datasets show that the proposed method achieves more accurate and reliable results than the other competitive methods. Code is available at https://github.com/123fxdx/CNNvisualizationTSGB.

FVC: An End-to-End Framework Towards Deep Video Compression in Feature Space.

Deep video compression is attracting increasing attention from both deep learning and video processing community. Recent learning-based approaches follow the hybrid coding paradigm to perform pixel space operations for reducing redundancy along both spatial and temporal dimentions, which leads to inaccurate motion estimation or less effective motion compensation. In this work, we propose a feature-space video coding framework (FVC), which performs all major operations (i.e., motion estimation, motion compression, motion compensation and residual compression) in the feature space. Specifically, a new deformable compensation module, which consists of motion estimation, motion compression and motion compensation, is proposed for more effective motion compensation. In our deformable compensation module, we first perform motion estimation in the feature space to produce the motion information (i.e., the offset maps). Then the motion information is compressed by using the auto-encoder style network. After that, we use the deformable convolution operation to generate the predicted feature for motion compensation. Finally, the residual information between the feature from the current frame and the predicted feature from the deformable compensation module is also compressed in the feature space. Motivated by the conventional codecs, in which the blocks with different sizes are used for motion estimation, we additionally propose two new modules called resolution-adaptive motion coding (RaMC) and resolution-adaptive residual coding (RaRC) to automatically cope with different types of motion and residual patterns at different spatial locations. Comprehensive experimental results demonstrate that our proposed framework achieves the state-of-the-art performance on three benchmark datasets including HEVC, UVG and MCL-JCV.

A Trimodel SAR Semisupervised Recognition Method Based on Attention-Augmented Convolutional Networks

Semi-supervised learning (SSL) in synthetic aperture radars (SAR) is one of the research hotspots in the field of radar-image automatic target recognition (ATR). It can efficiently deal with challenging environments where there are insufficient labeled samples and large unlabeled samples in the SAR dataset. In recent years, consistency regularization methods in semi-supervised learning have shown considerable improvement in recognition accuracy and efficiency. Current consistency regularization approaches suffer from two main shortcomings: firstly, extracting all of the relevant information in the image target is difficult owing to the inability of conventional convolutional neural networks (CNN) to capture global relational information,; secondly, the standard teacher-student regularization methodology causes confirmation biases due to the high coupling between teacher and student models. This work adopts an innovative tri-model semi-supervised method based on attention-augmented convolutional networks to address the aforementioned obstacles. Specifically, we develop an attention mechanism incorporating a novel positional embedding method based on recurrent neural networks (RNN), and integrate this with a standard convolutional network as a feature extractor, to improve the network's ability to extract global feature information from images. Further, we address the confirmation bias problem by introducing a classmate model to the standard teacher-student structure and utilise the model to impose a weak consistency constraint (WCC) designed on the student to weaken the strong-coupling between the teacher and the student. Comparative experiments on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset show that our method outperforms state-of-the-art semi-supervised methods in terms of recognition accuracy, demonstrating its potential as a new benchmark approach for the deep learning and SAR research community.

Generation of Spacecraft Operations Procedures Using Deep Reinforcement Learning

The high cost of space mission operations has motivated several space agencies to prioritize the development of autonomous spacecraft command and control technologies. Deep reinforcement learning (DRL) techniques present one promising domain for the creation of autonomous agents for complex, multifaceted operations problems. This work examines the feasibility of adapting DRL-driven policy generation algorithms to problems in spacecraft decision-making, including strategies for framing spacecraft decision-making problems such as Markov decision processes, avenues for dimensionality reduction, and simplification using expert domain knowledge, sensitivity to hyperparameters, and robustness in the face of mismodeled environmental dynamics. In addition, consideration is given to ensuring the safety of these approaches by hybridizing them with correct-by-construction control techniques in a novel adaptation of shielded deep reinforcement learning. These strategies are demonstrated against a prototypical low-fidelity stationkeeping scenario and a high-fidelity attitude mode management scenario involving flight heritage attitude control and momentum management algorithms. DRL techniques are found to compare favorably to other black-box optimization tools or heuristic solutions for these problems and to require similar network sizes and training durations as widely used testing datasets in the deep learning community.

A novel multimodal fusion network based on a joint-coding model for lane line segmentation

There has recently been growing interest in utilizing multimodal sensors to achieve robust lane line segmentation. In this paper, we introduce a novel multimodal fusion architecture from an information theory perspective, and demonstrate its practical utility using Light Detection and Ranging (LiDAR) camera fusion networks. In particular, we develop a multimodal fusion network as a joint coding model, where each single node, layer, and pipeline is represented as a communication channel, and forward propagation equates to information transmission in the channel. This enables us to qualitatively and quantitatively analyze the effect of different fusion approaches. We argue that an optimal fusion architecture is related to essential capacity and allocation based on the source and channel models. To test this multimodal fusion hypothesis, we progressively formulate a series of multimodal models based on our proposed fusion methods and evaluate them on benchmark KITTI and the A2D2 datasets. Our optimal fusion network achieves 85%+ lane line accuracy and 98.7%+ overall accuracy. We conclude that the performance gap with state-of-the-art models demonstrates the potential of our fusion architecture to serve as a new benchmark resource for the multimodal deep learning community.

Airborne SAR Autofocus Based on Blurry Imagery Classification

Existing airborne SAR autofocus methods can be classified as parametric and non-parametric. Generally, non-parametric methods, such as the widely used phase gradient autofocus (PGA) algorithm, are only suitable for scenes with many dominant point targets, while the parametric ones are suitable for all types of scenes, in theory, but their efficiency is generally low. In practice, whether many dominant point targets are present in the scene is usually unknown, so determining what kind of algorithm should be selected is not straightforward. To solve this issue, this article proposes an airborne SAR autofocus approach combined with blurry imagery classification to improve the autofocus efficiency for ensuring autofocus precision. In this approach, we embed the blurry imagery classification based on a typical VGGNet in a deep learning community into the traditional autofocus framework as a preprocessing step before autofocus processing to analyze whether dominant point targets are present in the scene. If many dominant point targets are present in the scene, the non-parametric method is used for autofocus processing. Otherwise, the parametric one is adopted. Therefore, the advantage of the proposed approach is the automatic batch processing of all kinds of airborne measured data.

A A Pilot Review of State-of-the-Art Deep Learning Applications to Nuclear Engineering & Technology

Given the advent of big data (BD), the focus of this “pilot” review of the most recent literature is upon deep learning (DL) implementations/solutions to important, as well as inherently complex, nuclear engineering and technology (NET) problems. The potential advancement from implementing large/real-scale DL-based algorithms is critically detailed, as well as highlighted, by inspecting the relevant body of latest research which appears in two major – overlapping, yet also complementary to each other in terms of scope and audience - nuclear energy (NE)-dedicated scientific research means of publication, namely Elsevier’s “Annals of Nuclear Energy” and “Progress in Nuclear Energy” journals. It is demonstrated that, given their (I) accuracy, robustness-to-noise and resilience to incomplete/erroneous information, as well as (II) competence under non-stationary and transient nuclear reactor (NR) operating conditions, deep and convolutional neural networks (DNNs and CNNs, respectively), as well as their variants, are becoming increasingly important, in their own right, for the swift and reliable solution of NET problems. It will be of great interest for – while also advantageous to - the NE, BD, and computational intelligence (CI)/DL communities, to join forces, follow up, and contribute to future research in – as well as application to - this particularly interesting and fruitful collaboration between disciplines.

Molecular design in drug discovery: a comprehensive review of deep generative models.

Deep generative models have been an upsurge in the deep learning community since they were proposed. These models are designed for generating new synthetic data including images, videos and texts by fitting the data approximate distributions. In the last few years, deep generative models have shown superior performance in drug discovery especially de novo molecular design. In this study, deep generative models are reviewed to witness the recent advances of de novo molecular design for drug discovery. In addition, we divide those models into two categories based on molecular representations in silico. Then these two classical types of models are reported in detail and discussed about both pros and cons. We also indicate the current challenges in deep generative models for de novo molecular design. De novo molecular design automatically is promising but a long road to be explored.

Depth Completion and Super-Resolution with Arbitrary Scale Factors for Indoor Scenes.

Depth sensing has improved rapidly in recent years, which allows for structural information to be utilized in various applications, such as virtual reality, scene and object recognition, view synthesis, and 3D reconstruction. Due to the limitations of the current generation of depth sensors, the resolution of depth maps is often still much lower than the resolution of color images. This hinders applications, such as view synthesis or 3D reconstruction, from providing high-quality results. Therefore, super-resolution, which allows for the upscaling of depth maps while still retaining sharpness, has recently drawn much attention in the deep learning community. However, state-of-the-art deep learning methods are typically designed and trained to handle a fixed set of integer-scale factors. Moreover, the raw depth map collected by the depth sensor usually has many depth data missing or misestimated values along the edges and corners of observed objects. In this work, we propose a novel deep learning network for both depth completion and depth super-resolution with arbitrary scale factors. The experimental results on the Middlebury stereo, NYUv2, and Matterport3D datasets demonstrate that the proposed method can outperform state-of-the-art methods.

CondenseNet with exclusive lasso regularization.

Group convolution has been widely used in deep learning community to achieve computation efficiency. In this paper, we develop CondenseNet-elasso to eliminate feature correlation among different convolution groups and alleviate neural network’s overfitting problem. It applies exclusive lasso regularization on CondenseNet. The exclusive lasso regularizer encourages different convolution groups to use different subsets of input channels therefore learn more diversified features. Our experiment results on CIFAR10, CIFAR100 and Tiny ImageNet show that CondenseNets-elasso are more efficient than CondenseNets and other DenseNet’ variants.

Comparison of augmentation and pre-processing for deep learning and chemometric classification of infrared spectra

Infrared spectroscopy hampered by physical distortions from scattering and instrumental effects. Therefore, it is generally accepted that spectra should be pre-processed before further data analysis. Deep learning community offers augmentation techniques as an alternative approach to deal with variability in the data. In this paper we propose an Extended Multiplicative Signal Augmentation (EMSA) method for augmenting physical distortions in infrared spectra. In order to study the effect of pre-processing and augmentation techniques we combine them with a wide range of classifiers from chemometrics, machine learning and statistics for four different spectral data sets. While the conventional pre-processing strategies perform well with all classifiers evaluated, augmentation can replace pre-processing when combined with Deep Convolutional Neural Networks and is especially successful when applied to small data sets.