Practical Deep Learning Research Articles

Private and Secure Distributed Deep Learning: A Survey

Traditionally, deep learning practitioners would bring data into a central repository for model training and inference. Recent developments in distributed learning, such as federated learning and deep learning as a service (DLaaS) do not require centralized data and instead push computing to where the distributed datasets reside. These decentralized training schemes, however, introduce additional security and privacy challenges. This survey first structures the field of distributed learning into two main paradigms and then provides an overview of the recently published protective measures for each. This work highlights both secure training methods as well as private inference measures. Our analyses show that recent publications while being highly dependent on the problem definition, report progress in terms of security, privacy, and efficiency. Nevertheless, we also identify several current issues within the private and secure distributed deep learning (PSDDL) field that require more research. We discuss these issues and provide a general overview of how they might be resolved.

Prospects, Challenges and Guidelines for Practical Deep Learning in Geoengineering

Prospects, Challenges and Guidelines for Practical Deep Learning in Geoengineering

Accurate measurement of key structures in CBD patients using deep learning

The prevalence of common bile duct (CBD) stones presents a widespread and intricate clinical challenge that necessitates precise measurements for effective clinical management. In response, we developed a practical deep learning system designed to precisely measure key structures within endoscopic retrograde cholangiopancreatography (ERCP) images in CBD patients. This system excels in detecting key structures in CBD patients and provides comprehensive information about the quantity, size, location, and dimensions of the common bile duct. While the size, location, and dimensions of the CBD are derived from the scaling ratio of the estimated duodenoscope diameter, the quantity of stones is determined independently of this conversion between pixels and millimeters. A rigorous evaluation of the system’s performance was conducted utilizing diverse metrics such as the Dice coefficient, accuracy, area under the receiver operating characteristic (AUROC) curve, and a comprehensive confusion matrix. A dataset of 418 images from 403 patients was divided into three sets: 314 images (75%) from 299 patients (74%) for training, 52 images (12.5%) from 52 patients (13%) for validation, and 52 images (12.5%) from 52 patients (13%) for testing. The proposed system demonstrated remarkable results in key structure detection and measurement, with Dice coefficients and overall AUROCs of 95.6% and 95.0% for the duodenoscope, 94.6% and 94.0% for the bile duct, and 94.1% and 92.0% for stone identification, respectively. The system’s timely, precise measurements of CBD structures can significantly enhance clinical decision-making, improve patient outcomes, and streamline diagnostic and management processes.

A practical PINN framework for multi-scale problems with multi-magnitude loss terms

For multi-scale problems, the conventional physics-informed neural networks (PINNs) face some challenges in obtaining available predictions. In this paper, based on PINNs, we propose a practical deep learning framework for multi-scale problems by reconstructing the loss function and associating it with specialized neural network architectures. New PINN methods derived from the improved PINN framework differ from the conventional PINN method mainly in two aspects. First, the new methods use a novel loss function by modifying the standard loss function through a (grouping) regularization strategy. The regularization strategy implements a different power operation on each loss term so that all loss terms composing the loss function are of approximately the same order of magnitude, which makes all loss terms be optimized synchronously during the optimization process. Second, for the multi-frequency or high-frequency problems, in addition to using the modified loss function, new methods upgrade the neural network architecture from the common fully-connected neural network to specialized network architectures such as the Fourier feature architecture given in Ref. [1] and the integrated architecture developed by us. The combination of the above two techniques leads to a significant improvement in the computational accuracy of multi-scale problems. Several challenging numerical examples demonstrate the effectiveness of the proposed methods. The proposed methods not only significantly outperform the conventional PINN method in terms of computational efficiency and computational accuracy, but also compare favorably with the state-of-the-art methods in the recent literature. The improved PINN framework facilitates better application of PINNs to multi-scale problems. The data and code accompanying this paper are available at https://github.com/wangyong1301108/MMPINN.

A Practical Deep Learning Architecture for Large-Area Solid Wastes Monitoring Based on UAV Imagery

The development of global urbanization has brought about a significant amount of solid waste. These untreated wastes may be dumped in any corner, causing serious pollution to the environment. Thus, it is necessary to accurately obtain their distribution locations and detailed edge information. In this study, a practical deep learning network for recognizing solid waste piles over extensive areas using unmanned aerial vehicle (UAV) imagery has been proposed and verified. Firstly, a high-resolution dataset serving to solid waste detection was created based on UAV aerial data. Then, a dual-branch solid waste semantic segmentation model was constructed to address the characteristics of the integration of solid waste distribution with the environment and the irregular edge morphology. The Context feature branch is responsible for extracting high-level semantic features, while the Spatial feature branch is designed to capture fine-grained spatial details. After information fusion, the model obtained more comprehensive feature representation and segmentation ability. The effectiveness of the improvement was verified through ablation experiments and compared with 13 commonly used semantic segmentation models, demonstrating the advantages of the method in solid waste segmentation tasks, with an overall accuracy of over 94%, and a recall rate of 88.6%—much better than the best performing baselines. Finally, a spatial distribution map of solid waste over Jiaxing district, China was generated by the model inference, which assisted the environmental protection department in completing environmental management. The proposed method provides a feasible approach for the accurately monitoring of solid waste, so as to provide policy support for environmental protection.

Humidification potential optimization of various membranes for proton exchange membrane fuel cell: Experiments and deep learning assisted metaheuristics

This study addresses the durability and performance challenges in Proton Exchange Membrane Fuel Cell (PEMFC) technology, primarily focusing on optimal humidity management. Acknowledging that, beyond technical aspects, the broader commercialization of PEMFCs is critically influenced by factors such as the cost and availability of hydrogen, this research aims to provide a comprehensive solution to enhance PEMFC applicability. Utilizing Nafion (NR-212), reverse osmosis (RO), and pervaporation (PV) membranes, the study optimizes five key performance metrics: pressure drop (∆P), dew point approach temperature (DPAT), water recovery ratio (WRR), Water Flux (J), and coefficient of performance (COP). These optimizations are conducted considering variables like temperature, humidity, flowrate, and membrane material. A deep neural network (DNN) model, incorporating Bayesian surrogacy with Gaussian process, gradient boost regression trees, and random forest, is developed using experimental data. With a correlation coefficient of 0.986, the model demonstrates high accuracy in predicting performance metrics, subsequently guiding genetic algorithms for effective PEMFC humidity control. The results show significant improvements in all metrics, with optimal values achieved for NR-212, RO, and PV membranes. This study thus presents a novel, practical deep learning approach, considering both technological advancements and external economic factors, for enhancing PEMFC operations.

Backdoor attacks and defenses in federated learning: Survey, challenges and future research directions

Federated learning (FL) is an approach within the realm of machine learning (ML) that allows the use of distributed data without compromising personal privacy. In FL, it becomes evident that the training data among participants frequently exhibit heterogeneous distribution characteristics. This inherent heterogeneity poses a substantial challenge for the orchestration server as it strives to assess the reliability of each local model update. Due to this challenge, FL becomes susceptible to various potential risks, with the ominous backdoor attack standing out as one of the most menacing threats. Backdoor attacks involve the insertion of malicious functionality into a targeted model through poisoned updates from malicious clients. These attacks can cause the global model to misbehave on specific inputs while appearing normal in other instances. Although the backdoor attacks received significant attention for their potential impact on practical deep learning applications, their exploration within the realm of FL remains limited. This survey seeks to address this gap by offering an all-encompassing examination of prevailing backdoor attack tactics and defenses in the context of FL. We include an exhaustive analysis of diverse approaches to provide a comprehensive understanding of this intricate landscape. Furthermore, we also discuss the challenges and potential future directions for attacks and defenses in the context of FL.

Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme.

Improving the generalization performance of deep neural networks (DNNs) trained by minibatch stochastic gradient descent (SGD) has raised lots of concerns from deep learning practitioners. The standard simple random sampling (SRS) scheme used in minibatch SGD treats all training samples equally in gradient estimation. In this article, we study a new data selection method based on the intrinsic property of the training set to help DNNs have better generalization performance. Our theoretical analysis suggests that this new sampling scheme, called the nontypicality sampling scheme, boosts the generalization performance of DNNs through biasing the solution toward wider minima, under certain assumptions. We confirm our findings experimentally and show that more variants of minibatch SGD can also benefit from the new sampling scheme. Finally, we discuss an extension of the nontypicality sampling scheme that holds promise to enhance both generalization performance and convergence speed of minibatch SGD.

Waterwave: A GPU Memory Flow Engine for Concurrent DNN Training

Training Deep Neural Networks (DNN) concurrently is becoming increasingly important for deep learning practitioners, e.g., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">hyperparameter optimization (HPO)</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">neural architecture search (NAS)</i> . The GPU memory capacity is the impediment that prohibits multiple DNNs from being trained on the same GPU due to the large memory usage during training. In this paper, we propose Waterwave a GPU memory flow engine for concurrent deep learning training. Firstly, to address the memory explosion brought by the long time lag between memory allocation and deallocation time, we develop an allocator tailored for multi-streams. By making the allocator aware of the stream information, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">prioritized allocation</i> is conducted based on the chunk's <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">synchronization</i> attributes, allowing us to provide useable memory after scheduling rather than waiting it to be really released after GPU computation. Secondly, Waterwave partitions the compute graph to a set of continuous <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">node groups</i> and then performs finer-grained scheduling: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NodeGroup pipeline execution</i> , to guarantee a proper memory requests order. Waterwave can accomplish up to 96.8% of the maximum batch size of solo training. Additionally, in scenarios with high memory demand, Waterwave can outperform existing spatial sharing and temporal sharing by up to 12x and 1.49x, respectively.

Deep learning-derived spatial organization features on histology images predicts prognosis in colorectal liver metastasis patients after hepatectomy

Histopathological images of colorectal liver metastases (CRLM) contain rich morphometric information that may predict patients' outcomes. However, to our knowledge, no study has reported any practical deep learning framework based on the histology images of CRLM, and their direct association with prognosis remains largely unknown. In this study, we developed a deep learning-based framework for fully automated tissue classification and quantification of clinically relevant spatial organization features (SOFs) in H&E-stained images of CRLM. The SOFs based risk-scoring system demonstrated a strong and robust prognostic value that is independent of the current clinical risk score (CRS) system in independent clinical cohorts. Our framework enables fully automated tissue classification of H&E images of CRLM, which could significantly reduce assessment subjectivity and the workload of pathologists. The risk-scoring system provides a time- and cost-efficient tool to assist clinical decision-making for patients with CRLM, which could potentially be implemented in clinical practice.

Experimental Observations of the Topology of Convolutional Neural Network Activations

Topological data analysis (TDA) is a branch of computational mathematics, bridging algebraic topology and data science, that provides compact, noise-robust representations of complex structures. Deep neural networks (DNNs) learn millions of parameters associated with a series of transformations defined by the model architecture resulting in high-dimensional, difficult to interpret internal representations of input data. As DNNs become more ubiquitous across multiple sectors of our society, there is increasing recognition that mathematical methods are needed to aid analysts, researchers, and practitioners in understanding and interpreting how these models' internal representations relate to the final classification. In this paper we apply cutting edge techniques from TDA with the goal of gaining insight towards interpretability of convolutional neural networks used for image classification. We use two common TDA approaches to explore several methods for modeling hidden layer activations as high-dimensional point clouds, and provide experimental evidence that these point clouds capture valuable structural information about the model's process. First, we demonstrate that a distance metric based on persistent homology can be used to quantify meaningful differences between layers and discuss these distances in the broader context of existing representational similarity metrics for neural network interpretability. Second, we show that a mapper graph can provide semantic insight as to how these models organize hierarchical class knowledge at each layer. These observations demonstrate that TDA is a useful tool to help deep learning practitioners unlock the hidden structures of their models.

Source-free and black-box domain adaptation via distributionally adversarial training

Source-free unsupervised domain adaptation is one class of practical deep learning methods which generalize in the target domain without transferring data from source domain. However, existing source-free domain adaptation methods rely on source model transferring. In many data-critical scenarios, the transferred source models may suffer from membership inference attacks and expose private data. In this paper, we aim to overcome a more practical and challenging setting where the source models cannot be transferred to the target domain. The source models are considered as queryable black-box models which only output hard labels. We use public third-party data to probe the source model and obtain supervision information, dispensing with transferring source model. To fill the gap between third-party data and target data, we further propose Distributionally Adversarial Training (DAT) to align the distribution of third-party data with target data, gain more informative query results and improve the data efficiency. We call this new framework Black-box Probe Domain Adaptation (BPDA) which adopts query mechanism and DAT to probe and refine supervision information. Experimental results on several domain adaptation datasets demonstrate the practicability and data efficiency of BPDA in query-only and source-free unsupervised domain adaptation.

Deep diffusion models for seismic processing

Seismic data processing involves techniques to deal with undesired effects that occur during acquisition and pre-processing. These effects mainly comprise coherent artefacts such as multiples, non-coherent signals such as electrical noise, and loss of signal information at the receivers that leads to incomplete traces. In the past years, there has been a remarkable increase of machine-learning-based solutions that have addressed the aforementioned issues. In particular, deep-learning practitioners have usually relied on heavily fine-tuned, customized discriminative algorithms. Although, these methods can provide solid results, they seem to lack semantic understanding of the provided data. Motivated by this limitation, in this work, we employ a generative solution, as it can explicitly model complex data distributions and hence, yield to a better decision-making process. In particular, we introduce diffusion models for three seismic applications: demultiple, denoising and interpolation. To that end, we run experiments on synthetic and on real data, and we compare the diffusion performance with standardized algorithms. We believe that our pioneer study not only demonstrates the capability of diffusion models, but also opens the door to future research to integrate generative models in seismic workflows.

Not getting in too deep: A practical deep learning approach to routine crystallisation image classification.

Using a relatively small training set of ~16 thousand images from macromolecular crystallisation experiments, we compare classification results obtained with four of the most widely-used convolutional deep-learning network architectures that can be implemented without the need for extensive computational resources. We show that the classifiers have different strengths that can be combined to provide an ensemble classifier achieving a classification accuracy comparable to that obtained by a large consortium initiative. We use eight classes to effectively rank the experimental outcomes, thereby providing detailed information that can be used with routine crystallography experiments to automatically identify crystal formation for drug discovery and pave the way for further exploration of the relationship between crystal formation and crystallisation conditions.

The Societal and Scientific Importance of Inclusivity, Diversity, and Equity in Machine Learning for Chemistry.

While the introduction of practical deep learning has driven progress across scientific fields, recent research highlighted that the requirement of deep learning for ever-increasing computational resources and data has potential negative impacts on the scientific community and society as a whole. An ever-growing need for more computational resources may exacerbate the concentration of funding, the exclusiveness of research, and thus the inequality between countries, sectors, and institutions. Here, I introduce recent concerns and considerations of the machine learning research community that could affect chemistry and present potential solutions, including more detailed assessments of model performance, increased adherence to open science and open data practices, an increase in multinational and multi-institutional collaboration, and a focus on thematic and cultural diversity.

DeepAPP: A Deep Reinforcement Learning Framework for Mobile Application Usage Prediction

This paper aims to predict a set of apps a user will open on her mobile device in the next time slot. Such an information is essential for many smartphone operations, e.g., app pre-loading and content pre-caching, to improve user experience. However, it is hard to build an explicit model that accurately captures the complex environment context and predicts a set of apps at one time. This paper presents a deep reinforcement learning framework, named as DeepAPP, which learns a model-free predictive neural network from historical app usage data. Meanwhile, an online updating strategy is designed to adapt the predictive network to the time-varying app usage behavior. To transform DeepAPP into a practical deep reinforcement learning system, several challenges are addressed by developing a context representation method for complex contextual environment, a general agent for overcoming data sparsity and a lightweight personalized agent for minimizing the prediction time. Extensive experiments on a large-scale anonymized app usage dataset reveal that DeepAPP provides high accuracy (precision 70.6 percent and recall of 62.4 percent) and reduces the prediction time of the state-of-the-art by 6.58×. A field experiment of 29 participants demonstrates DeepAPP can effectively reduce launch time of apps.

Anti-occlusion multi-object surveillance based on improved deep learning approach and multi-feature enhancement for unmanned smart grid safety

Unattended substations are the basis of intelligent substations, which require remote surveillance and control. Limited by the number of visual sensors, remote manual monitoring is incomplete and inefficient. Onsite workers and intruders are easily hidden by the smart grid facility, which affects the safety surveillance of personnel and equipment. The traditional kernelized correlation filter (KCF) method has a poor ability to adapt to the practical environment. This paper presents an anti-occlusion framework on the basis of imaging techniques to solve the problem of optimized surveillance. The novelty is further strengthened by its more practical Deep Learning model and tracking methods. Firstly, a multi-feature fusion model of the HOG feature and color feature is proposed to enhance target characteristics as the target is severely blocked. Secondly, a target classifier training and fast detection method based on improved CNN is introduced. Lastly, to overcome the drawbacks of the KCF tracking algorithm, such as its inability to scale adaptive and blind updates, a new adaptive learning rate strategy is proposed for occlusion tracking. The effects on the OTB-2013 dataset demonstrate that the improved technique has better accuracy and robustness when compared to KCF methods.

Evaluating online health information quality using machine learning and deep learning: A systematic literature review.

Due to the large volume of online health information, while quality remains dubious, understanding the usage of artificial intelligence to evaluate health information and surpass human-level performance is crucial. However, the existing studies still need a comprehensive review highlighting the vital machine, and Deep learning techniques for the automatic health information evaluation process. Therefore, this study outlines the most recent developments and the current state of the art regarding evaluating the quality of online health information on web pages and specifies the direction of future research. In this article, a systematic literature is conducted according to the PRISMA statement in eight online databases PubMed, Science Direct, Scopus, ACM, Springer Link, Wiley Online Library, Emerald Insight, and Web of Science to identify all empirical studies that use machine and deep learning models for evaluating the online health information quality. Furthermore, the selected techniques are compared based on their characteristics, such as health quality criteria, quality measurement tools, algorithm type, and achieved performance. The included papers evaluate health information on web pages using over 100 quality criteria. The results show no universal quality dimensions used by health professionals and machine or deep learning practitioners while evaluating health information quality. In addition, the metrics used to assess the model performance are not the same as those used to evaluate human performance. This systemic review offers a novel perspective in approaching the health information quality in web pages that can be used by machine and deep learning practitioners to tackle the problem more effectively.

Advancements in enhancing cyber-physical system security: Practical deep learning solutions for network traffic classification and integration with security technologies.

Traditional network analysis frequently relied on manual examination or predefined patterns for the detection of system intrusions. As soon as there was increase in the evolution of the internet and the sophistication of cyber threats, the ability for the identification of attacks promptly became more challenging. Network traffic classification is a multi-faceted process that involves preparation of datasets by handling missing and redundant values. Machine learning (ML) models have been employed to classify network traffic effectively. In this article, we introduce a hybrid Deep learning (DL) model which is designed for enhancing the accuracy of network traffic classification (NTC) within the domain of cyber-physical systems (CPS). Our novel model capitalizes on the synergies among CPS, network traffic classification (NTC), and DL techniques. The model is implemented and evaluated in Python, focusing on its performance in CPS-driven network security. We assessed the model's effectiveness using key metrics such as accuracy, precision, recall, and F1-score, highlighting its robustness in CPS-driven security. By integrating sophisticated hybrid DL algorithms, this research contributes to the resilience of network traffic classification in the dynamic CPS environment.

Deep Learning in Diverse Intelligent Sensor Based Systems.

Deep learning has become a predominant method for solving data analysis problems in virtually all fields of science and engineering. The increasing complexity and the large volume of data collected by diverse sensor systems have spurred the development of deep learning methods and have fundamentally transformed the way the data are acquired, processed, analyzed, and interpreted. With the rapid development of deep learning technology and its ever-increasing range of successful applications across diverse sensor systems, there is an urgent need to provide a comprehensive investigation of deep learning in this domain from a holistic view. This survey paper aims to contribute to this by systematically investigating deep learning models/methods and their applications across diverse sensor systems. It also provides a comprehensive summary of deep learning implementation tips and links to tutorials, open-source codes, and pretrained models, which can serve as an excellent self-contained reference for deep learning practitioners and those seeking to innovate deep learning in this space. In addition, this paper provides insights into research topics in diverse sensor systems where deep learning has not yet been well-developed, and highlights challenges and future opportunities. This survey serves as a catalyst to accelerate the application and transformation of deep learning in diverse sensor systems.