Training Deep Neural Networks Research Articles

PATH-13. LEARNED RESIZING WITH EFFICIENT TRAINING (LRET) FACILITATES IMPROVED PERFORMANCE OF LARGE-SCALE BRAIN TUMOR HISTOLOGY IMAGE CLASSIFICATION MODELS

Abstract BACKGROUND Histologic examination is vital in oncology research and diagnostics. The adoption of digital scanning of whole slide images (WSI) has created an opportunity to leverage deep learning-based image classification methods to enhance diagnosis and risk stratification. However, technical limitations prevent training and deployment of accurate comprehensive multiclass deep convolutional neural networks (DCNN) models for histopathology image classification. The input dimensions of DCNN architectures are small compared to the typical pathologist field of view, degrading performance by excluding important architectural features. Furthermore, data requirements for comprehensive models are sufficiently large to overwhelm the system memory during training. METHODS A method termed Learned Resizing with Efficient Training (LRET) was developed to address the main limitations of traditional histopathology classification model training. The LRET method couples efficient training techniques with image resizing to facilitate seamless integration of larger histology image patches into state-of-the-art classification models while preserving important structural information. The LRET method was coupled with two distinct resizing techniques to train three diverse histology image datasets using five different DCNN architectures. Performance metrics were compared on cross validation and hold out test sets. RESULTS LRET-trained models were flexible to multiple input patch dimensions and DCNN models. We demonstrated performance improvement across all datasets while significantly reducing the training time and resources over traditional methods. Using a large-scale, multiclass brain tumor classification dataset consisting of 74 distinct histopathologic classes, LRET-trained models outperformed existing methods by 15-28% in accuracy, yielding 94% accuracy for the best model. CONCLUSION The LRET method for DCNN training significantly enhances the performance of large-scale multiclass histopathology image classification. The implications of this work extend to broader applications within medical imaging and beyond, where efficient integration of high-resolution images into deep learning pipelines is paramount for driving advancements research and clinical practice.

A blueprint for precise and fault-tolerant analog neural networks

Analog computing has reemerged as a promising avenue for accelerating deep neural networks (DNNs) to overcome the scalability challenges posed by traditional digital architectures. However, achieving high precision using analog technologies is challenging, as high-precision data converters are costly and impractical. In this work, we address this challenge by using the residue number system (RNS) and composing high-precision operations from multiple low-precision operations, thereby eliminating the need for high-precision data converters and information loss. Our study demonstrates that the RNS-based approach can achieve ≥99% FP32 accuracy with 6-bit integer arithmetic for DNN inference and 7-bit for DNN training. The reduced precision requirements imply that using RNS can achieve several orders of magnitude higher energy efficiency while maintaining the same throughput compared to conventional analog hardware with the same precision. We also present a fault-tolerant dataflow using redundant RNS to protect the computation against noise and errors inherent within analog hardware.

A stochastic gradient method with variance control and variable learning rate for Deep Learning

In this paper we study a stochastic gradient algorithm which rules the increase of the mini-batch size in a predefined fashion and automatically adjusts the learning rate by means of a monotone or non-monotone line search procedure. The mini-batch size is incremented at a suitable a priori rate throughout the iterative process in order that the variance of the stochastic gradients is progressively reduced. The a priori rate is not subject to restrictive assumptions, allowing for the possibility of a slow increase in the mini-batch size. On the other hand, the learning rate can vary non-monotonically throughout the iterations, as long as it is appropriately bounded. Convergence results for the proposed method are provided for both convex and non-convex objective functions. Moreover it can be proved that the algorithm enjoys a global linear rate of convergence on strongly convex functions. The low per-iteration cost, the limited memory requirements and the robustness against the hyperparameters setting make the suggested approach well-suited for implementation within the deep learning framework, also for GPGPU-equipped architectures. Numerical results on training deep neural networks for multi-class image classification show a promising behaviour of the proposed scheme with respect to similar state of the art competitors.

DGPCD: a benchmark for typical official-style Dougong in ancient Chinese wooden architecture

Dougong, a distinctive component of ancient Chinese wooden architecture, holds significant importance for the preservation and restoration of such structures. In particular, the northern official-style buildings represent the pinnacle of ancient Chinese construction techniques. In the realm of cultural heritage preservation, the application of deep learning has gradually expanded, demonstrating remarkable effectiveness. Point cloud serving as a crucial source for Dougong, encapsulates various information, enabling support for tasks like Dougong point cloud classification and completion. The quality of Dougong datasets directly impacts the outcomes of DNNs (deep neural networks), as they serve as the foundational data support for these models. The typical official-style Dougong, with its standardized and repetitive structural patterns, is highly suitable for training DNNs to accurately recognize and analyze these complex architectural elements. However, due to the inherent characteristics of Dougong, such as coplanarity and occlusion, acquiring point cloud data is challenging, resulting in poor data quality and organizational difficulties. To address this issue, our study adopts a multi-source data fusion approach to tackle the challenges of insufficient data quantity and poor data quality. Further, through data augmentation, we enhance the dataset’s volume and generalize its characteristics. This effort culminates in the creation of the typical official-style Dougong Point Cloud Dataset (DG Dataset), poised to support deep learning tasks related to Dougong scenarios.

Fast generation of collision-free low-thrust trajectories for asteroid landing using deep neural networks

This study presents a fast generation method of time-optimal low-thrust trajectories based on deep neural networks (DNN) for the collision-free asteroid landings. An equivalent unconstrained differentiable problem is transformed via the introduction of smooth penalty function, so that it can be addressed with the optimal control framework. To efficiently solve the equivalent problem to generate sufficient dataset for training DNNs, a double homotopy method is developed. Specifically, the equivalent problem is connected to the time-optimal control problem without anti-collision through two consecutive simplifications, and a solving process with double backward continuation is also formulated. Besides, two DNN models are constructed to provide high-quality predictions for the minimum glide-slope constraint angle of an initial state, and the initial costates and landing time for time-optimal low-thrust trajectories with anti-collision constraints, respectively. As such, the efficiency of collision-free low-thrust landing trajectory generation is improved significantly. For the cases within the range of training dataset or within a certain extension, the low-thrust landing trajectories with a very small terminal landing error can be generated using the predicted initial costates. Meanwhile, with the high-quality initial costates, the high-precision landing trajectories can be obtained by the double homotopy method in only a few iterative steps. Finally, the promising results in a Bennu collision-free asteroid landing scenario validate the efficiency of the proposed method.

Shear-wave velocity prediction of tight reservoirs based on poroelasticity theory: A comparative study of deep neural network and rock physics model

The absence of shear-wave (S-wave) velocity in well log data limits the effective seismic characterization of lithology, petrophysical properties, and fluid distribution of tight reservoirs in the Chang 7 shale oil formations of Ordos Basin, west China. However, conventional rock physics models (RPMs), e.g., the Xu-White model, struggle to accurately obtain the key parameters such as pore aspect ratio in the process of S-wave velocity prediction. This work performs a comparative study of S-wave velocity prediction based on deep neural network (DNN) and RPM for such reservoirs. By employing the poroelasticity equations which are capable of describing elastic wave propagation in the complex reservoirs saturated with viscous fluid, the relation between elastic and petrophysical properties is established with a DNN-based method. This approach quantifies the connection between P-/S-wave velocities and rock properties. On the other hand, we reformulate the traditional Xu-White model by incorporating the decoupled Kuster-Toksöz (K-T) model and the poroelasticity equations with viscoelastic fluid in the modeling process. The variation of pore aspect ratio with depth is also considered to characterize the complex pore structures, which is appropriate for improving the accuracy of prediction. Log data from the three wells are considered for verification, of which results show that the S-wave velocity prediction methods help improve the final result. Aided by the DNN-based method, the prediction exhibits a better performance if the training data is sufficient. In particular, provided a small amount of measured S-wave data (even ten data points), the trained DNN model can also provide reasonable predictions.

A new deep neural network for forecasting: Deep dendritic artificial neural network

Deep artificial neural networks have become a good alternative to classical forecasting methods in solving forecasting problems. Popular deep neural networks classically use additive aggregation functions in their cell structures. It is available in the literature that the use of multiplicative aggregation functions in shallow artificial neural networks produces successful results for the forecasting problem. A type of high-order shallow artificial neural network that uses multiplicative aggregation functions is the dendritic neuron model artificial neural network, which has successful forecasting performance. In this study, the transformation of the dendritic neuron model turned into a multi-output architecture. A new dendritic cell based on the multi-output dendritic neuron model and a new deep artificial neural network is proposed. The training of this new deep dendritic artificial neural network is carried out with the differential evolution algorithm. The forecasting performance of the deep dendritic artificial neural network is compared with basic classical forecasting methods and some recent shallow and deep artificial neural networks over stock market time series. As a result, it has been observed that deep dendritic artificial neural network produces very successful forecasting results for the forecasting problem.

Extremization to fine tune physics informed neural networks for solving boundary value problems

We propose a novel method for fast and accurate training of physics-informed neural networks (PINNs) to find solutions to boundary value problems (BVPs) and initial boundary value problems (IBVPs). By combining the methods of training deep neural networks (DNNs) and Extreme Learning Machines (ELMs), we develop a model which has the expressivity of DNNs with the fine-tuning ability of ELMs. We showcase the superiority of our proposed method by solving several BVPs and IBVPs which include linear and non-linear ordinary differential equations (ODEs), partial differential equations (PDEs) and coupled PDEs. The examples we consider include a stiff coupled ODE system where traditional numerical methods fail, a 3+1D non-linear PDE, Kovasznay flow and Taylor–Green vortex solutions to incompressible Navier–Stokes equations and pure advection solution of 1+1 D compressible Euler equation.The Theory of Functional Connections (TFC) is used to exactly impose initial and boundary conditions (IBCs) of (I)BVPs on PINNs. We propose a modification to the TFC framework named Reduced TFC and show a significant improvement in the training and inference time of PINNs compared to IBCs imposed using TFC. Furthermore, Reduced TFC is shown to be able to generalize to more complex boundary geometries which is not possible with TFC. We also introduce a method of applying boundary conditions at infinity for BVPs and numerically solve the pure advection in 1+1 D Euler equations using these boundary conditions.

Revealing principles of autonomous thermal soaring in windy conditions using vulture-inspired deep reinforcement-learning

Thermal soaring, a technique used by birds and gliders to utilize updrafts of hot air, is an appealing model-problem for studying motion control and how it is learned by animals and engineered autonomous systems. Thermal soaring has rich dynamics and nontrivial constraints, yet it uses few control parameters and is becoming experimentally accessible. Following recent developments in applying reinforcement learning methods for training deep neural-network (deep-RL) models to soar autonomously both in simulation and real gliders, here we develop a simulation-based deep-RL system to study the learning process of thermal soaring. We find that this process has learning bottlenecks, we define a new efficiency metric and use it to characterize learning robustness, we compare the learned policy to data from soaring vultures, and find that the neurons of the trained network divide into function clusters that evolve during learning. These results pose thermal soaring as a rich yet tractable model-problem for the learning of motion control.

Tricking AI chips into simulating the human brain: A detailed performance analysis

In recent years, significant strides in Artificial Intelligence (AI) have led to various practical applications, primarily centered around training and deployment of deep neural networks (DNNs). These applications, however, require considerable computational resources, predominantly reliant on modern Graphics-Processing Units (GPUs). Yet, the quest for larger and faster DNNs has spurred the creation of specialized AI chips and efficient Machine-Learning (ML) software tools like TensorFlow and PyTorch have been developed for striking a balance between usability and performance. Simultaneously, the field of computational neuroscience shares a similar quest for increased computational power to simulate more extensive and detailed brain models, while also keeping usability high. Although GPUs have also entered this field, programming complexity remains high, resulting in cumbersome simulations. Inspired by AI progress, we introduce a workflow for easily accelerating brain simulations using TensorFlow and evaluate the performance of various, cutting-edge AI chips – including the Graphcore Intelligence-Processing Unit (IPU), GroqChip, Nvidia GPU with Tensor Cores, and Google Tensor-Processing Unit (TPU) – when simulating a biologically detailed as well as simpler brain models. Our model simulations explore the architectural tradeoffs of a modern-day CPU and these four AI platforms by varying computational density, memory requirements and floating-point numerical accuracy. Results show that the GroqChip achieves the best performance for small networks, yet is unable to simulate large-scale networks. At the scale of mammalian brains, the GPU, IPU and TPU achieve speedups ranging from 29x to 1,208x times over CPU runtimes. Remarkably, the TPU sets a new record for the largest, real-time simulation of the inferior-olivary nucleus in the brain. Reduced-accuracy floating-point implementations make some simulation results unreliable for brain research, notably for the GroqChip. Consequently, this work underscores the potential of ML libraries for accelerating brain simulations as well as the critical role of AI-chip numerical accuracy for biophysically realistic brain models.

Accuracy Assessment and Reinforcement Training of Deep Neural Networks in the Context of Accelerating a Simulation-Based Toolpath Generation Method

Accuracy Assessment and Reinforcement Training of Deep Neural Networks in the Context of Accelerating a Simulation-Based Toolpath Generation Method

Spiking neural networks with consistent mapping relations allow high-accuracy inference

Spike-based neuromorphic hardware has demonstrated substantial potential in low energy consumption and efficient inference. However, the direct training of deep spiking neural networks is challenging, and conversion-based methods still require substantial time delay owing to unresolved conversion errors. We determine that the primary source of the conversion errors stems from the inconsistency between the mapping relationship of traditional activation functions and the input-output dynamics of spike neurons. To counter this, we introduce the Consistent ANN-SNN Conversion (CASC) framework. It includes the Consistent IF (CIF) neuron model, specifically contrived to minimize the influence of the stable point's upper bound, and the wake-sleep conversion (WSC) method, synergistically ensuring the uniformity of neuron behavior. This method theoretically achieves a loss-free conversion, markedly diminishing time delays and improving inference performance in extensive classification and object detection tasks. Our approach offers a viable pathway toward more efficient and effective neuromorphic systems.

Toward Interpretable Cell Image Representation and Abnormality Scoring for Cervical Cancer Screening Using Pap Smears.

Screening is critical for prevention and early detection of cervical cancer but it is time-consuming and laborious. Supervised deep convolutional neural networks have been developed to automate pap smear screening and the results are promising. However, the interest in using only normal samples to train deep neural networks has increased owing to the class imbalance problems and high-labeling costs that are both prevalent in healthcare. In this study, we introduce a method to learn explainable deep cervical cell representations for pap smear cytology images based on one-class classification using variational autoencoders. Findings demonstrate that a score can be calculated for cell abnormality without training models with abnormal samples, and we localize abnormality to interpret our results with a novel metric based on absolute difference in cross-entropy in agglomerative clustering. The best model that discriminates squamous cell carcinoma (SCC) from normals gives 0.908±0.003 area under operating characteristic curve (AUC) and one that discriminates high-grade epithelial lesion (HSIL) 0.920±0.002 AUC. Compared to other clustering methods, our method enhances the V-measure and yields higher homogeneity scores, which more effectively isolate different abnormality regions, aiding in the interpretation of our results. Evaluation using an external dataset shows that our model can discriminate abnormality without the need for additional training of deep models.

Thermal buckling of the concrete structures reinforced by nanocomposites: Introducing advanced deep neural networks for thermal buckling problem

The thermal buckling behavior of concrete annular sector plates reinforced with nanocomposites is investigated in this work. The thermal buckling issue is stated using mathematical modeling, taking into account the geometric linearity and material heterogeneity caused by the nanocomposite reinforcement. The governing differential equations that are obtained from the theoretical model are solved using the discrete singular convolution (DSC) technique. To generate accurate buckling load predictions, the DSC method’s great accuracy in handling complicated boundary conditions and discontinuities is used. Furthermore, using mathematical modeling to create extensive datasets, the research aims to train deep neural networks (DNNs) to anticipate thermal buckling reactions. These datasets provide a strong basis for DNN training as they include a wide range of factors, such as geometric configurations, loading situations, and material properties. By combining DNN predictions with DSC solutions, this unique technique for solving this difficult issue seeks to improve the accuracy and efficiency of thermal buckling analysis. The findings show how well sophisticated mathematical methods and machine learning models work together, opening the door to creative solutions for the design and analysis of concrete structures reinforced with nanocomposite under complicated stress.

ReSmooth: Detecting and Utilizing OOD Samples When Training With Data Augmentation.

Data augmentation (DA) is a widely used technique for enhancing the training of deep neural networks. Recent DA techniques which achieve state-of-the-art performance always meet the need for diversity in augmented training samples. However, an augmentation strategy that has a high diversity usually introduces out-of-distribution (OOD) augmented samples and these samples consequently impair the performance. To alleviate this issue, we propose ReSmooth, a framework that first detects OOD samples in augmented samples and then leverages them. To be specific, we first use a Gaussian mixture model (GMM) to fit the loss distribution of both the original and augmented samples and accordingly split these samples into in-distribution (ID) samples and OOD samples. Then we start a new training where ID and OOD samples are incorporated with different smooth labels. By treating ID samples and OOD samples unequally, we can make better use of the diverse augmented data. Furthermore, we incorporate our ReSmooth framework with negative DA (NDA) strategies. By properly handling their intentionally created OOD samples, the classification performance of NDAs is largely ameliorated. Experiments on several classification benchmarks show that ReSmooth can be easily extended to the existing augmentation strategies [such as RandAugment (RA), rotate, and jigsaw] and improve on them. Our code is available at https://github.com/Chenyang4/ReSmooth.

Automatic Ground-Truth Image Labeling for Deep Neural Network Training and Evaluation Using Industrial Robotics and Motion Capture

Abstract The United States Navy (USN) intends to increase the amount of uncrewed aircraft in a carrier air wing. To support this increase, carrier-based uncrewed aircraft will be required to have some level of autonomy as there will be situations where a human cannot be in/on the loop. However, there is no existing and approved method to certify autonomy within Naval Aviation. In support of generating certification evidence for autonomy, the United States Naval Academy (USNA) has created a training and evaluation system (TES) to provide quantifiable metrics for feedback performance in autonomous systems. The preliminary use case for this work focuses on autonomous aerial refueling. Prior demonstrations of autonomous aerial refueling have leveraged a deep neural network (DNN) for processing visual feedback to approximate the relative position of an aerial refueling drogue. The training and evaluation system proposed in this work simulates the relative motion between the aerial refueling drogue and feedback camera system using industrial robotics. Ground-truth measurements of the pose between the camera and drogue are measured using a commercial motion capture system. Preliminary results demonstrate calibration methods providing ground-truth measurements with millimeter precision. Leveraging this calibration, the proposed system is capable of providing large-scale datasets for DNN training and evaluation against a precise ground truth.

Isolated Pulsar Population Synthesis with Simulation-based Inference

We combine pulsar population synthesis with simulation-based inference (SBI) to constrain the magnetorotational properties of isolated Galactic radio pulsars. We first develop a framework to model neutron star birth properties and their dynamical and magnetorotational evolution. We specifically sample initial magnetic field strengths, B, and spin periods, P, from lognormal distributions and capture the late-time magnetic field decay with a power law. Each lognormal is described by a mean, μlogB,μlogP , and standard deviation, σlogB,σlogP , while the power law is characterized by the index, a late. We subsequently model the stars’ radio emission and observational biases to mimic detections with three radio surveys, and we produce a large database of synthetic P– Ṗ diagrams by varying our five magnetorotational input parameters. We then follow an SBI approach that focuses on neural posterior estimation and train deep neural networks to infer the parameters’ posterior distributions. After successfully validating these individual neural density estimators on simulated data, we use an ensemble of networks to infer the posterior distributions for the observed pulsar population. We obtain μlogB=13.10−0.10+0.08 , σlogB=0.45−0.05+0.05 and μlogP=−1.00−0.21+0.26 , σlogP=0.38−0.18+0.33 for the lognormal distributions and alate=−1.80−0.61+0.65 for the power law at the 95% credible interval. We contrast our results with previous studies and highlight uncertainties of the inferred a late value. Our approach represents a crucial step toward robust statistical inference for complex population synthesis frameworks and forms the basis for future multiwavelength analyses of Galactic pulsars.

A non-monotone trust-region method with noisy oracles and additional sampling

In this work, we introduce a novel stochastic second-order method, within the framework of a non-monotone trust-region approach, for solving the unconstrained, nonlinear, and non-convex optimization problems arising in the training of deep neural networks. The proposed algorithm makes use of subsampling strategies that yield noisy approximations of the finite sum objective function and its gradient. We introduce an adaptive sample size strategy based on inexpensive additional sampling to control the resulting approximation error. Depending on the estimated progress of the algorithm, this can yield sample size scenarios ranging from mini-batch to full sample functions. We provide convergence analysis for all possible scenarios and show that the proposed method achieves almost sure convergence under standard assumptions for the trust-region framework. We report numerical experiments showing that the proposed algorithm outperforms its state-of-the-art counterpart in deep neural network training for image classification and regression tasks while requiring a significantly smaller number of gradient evaluations.

Phishing Website Detection

Phishing websites have emerged as a serious security risk. Phishing is the starting point for many cyberattacks that compromise the confidentiality, integrity, and availability of customer and business data. Decades of effort have gone into developing novel methods for automatically identifying phishing websites. Modern systems aren't very adept at spotting new phishing threats and require a lot of manual feature engineering, even though they can produce better outcomes. Thus, an open problem in this discipline is to identify tactics that can swiftly handle zero-day phishing attempts and automatically recognize phishing websites. The web page that the URL hosts has a plethora of information that can be utilized to assess the maliciousness of the web server. One useful technique for spotting phishing emails is machine learning. Additionally, it does away with the drawbacks of the earlier approach. After a careful analysis of the literature, we proposed a novel approach that combines a machine learning algorithm with feature extraction to identify phishing websites. Using the gathered dataset, this study aims to train deep neural networks and machine learning models to detect phishing websites.

BAMFORESTS: Bamberg Benchmark Forest Dataset of Individual Tree Crowns in Very-High-Resolution UAV Images

The anthropogenic climate crisis results in the gradual loss of tree species in locations where they were previously able to grow. This leads to increasing workloads and requirements for foresters and arborists as they are forced to restructure their forests and city parks. The advancements in computer vision (CV)—especially in supervised deep learning (DL)—can help cope with these new tasks. However, they rely on large, carefully annotated datasets to produce good and generalizable models. This paper presents BAMFORESTS: a dataset with 27,160 individually delineated tree crowns in 105 ha of very-high-resolution UAV imagery gathered with two different sensors from two drones. BAMFORESTS covers four areas of coniferous, mixed, and deciduous forests and city parks. The labels contain instance segmentations of individual trees, and the proposed splits are balanced by tree species and vitality. Furthermore, the dataset contains the corrected digital surface model (DSM), representing tree heights. BAMFORESTS is annotated in the COCO format and is especially suited for training deep neural networks (DNNs) to solve instance segmentation tasks. BAMFORESTS was created in the BaKIM project and is freely available under the CC BY 4.0 license.