Training Deep Neural Networks Research Articles

Reducing the Effect of Incorrect Lithology Labels on the Training of Deep Neural Networks for Lithology Identification

Reducing the Effect of Incorrect Lithology Labels on the Training of Deep Neural Networks for Lithology Identification

DNNShifter: An efficient DNN pruning system for edge computing

Deep neural networks (DNNs) underpin many machine learning applications. Production quality DNN models achieve high inference accuracy by training millions of DNN parameters which has a significant resource footprint. This presents a challenge for resources operating at the extreme edge of the network, such as mobile and embedded devices that have limited computational and memory resources. To address this, models are pruned to create lightweight, more suitable variants for these devices. Existing pruning methods are unable to provide similar quality models compared to their unpruned counterparts without significant time costs and overheads or are limited to offline use cases. Our work rapidly derives suitable model variants while maintaining the accuracy of the original model. The model variants can be swapped quickly when system and network conditions change to match workload demand. This paper presents DNNShifter, an end-to-end DNN training, spatial pruning, and model switching system that addresses the challenges mentioned above. At the heart of DNNShifter is a novel methodology that prunes sparse models using structured pruning - combining the accuracy-preserving benefits of unstructured pruning with runtime performance improvements of structured pruning. The pruned model variants generated by DNNShifter are smaller in size and thus faster than dense and sparse model predecessors, making them suitable for inference at the edge while retaining near similar accuracy as of the original dense model. DNNShifter generates a portfolio of model variants that can be swiftly interchanged depending on operational conditions. DNNShifter produces pruned model variants up to 93x faster than conventional training methods. Compared to sparse models, the pruned model variants are up to 5.14x smaller and have a 1.67x inference latency speedup, with no compromise to sparse model accuracy. In addition, DNNShifter has up to 11.9x lower overhead for switching models and up to 3.8x lower memory utilisation than existing approaches. DNNShifter is available for public use from https://github.com/blessonvar/DNNShifter.

Learned mappings for targeted free energy perturbation between peptide conformations.

Targeted free energy perturbation uses an invertible mapping to promote configuration space overlap and the convergence of free energy estimates. However, developing suitable mappings can be challenging. Wirnsberger et al. [J. Chem. Phys. 153, 144112 (2020)] demonstrated the use of machine learning to train deep neural networks that map between Boltzmann distributions for different thermodynamic states. Here, we adapt their approach to the free energy differences of a flexible bonded molecule, deca-alanine, with harmonic biases and different spring centers. When the neural network is trained until "early stopping"-when the loss value of the test set increases-we calculate accurate free energy differences between thermodynamic states with spring centers separated by 1 Å and sometimes 2 Å. For more distant thermodynamic states, the mapping does not produce structures representative of the target state, and the method does not reproduce reference calculations.

Fighting selection bias in statistical learning: application to visual recognition from biased image databases

In practice, and especially when training deep neural networks, visual recognition rules are often learned based on various sources of information. On the other hand, the recent deployment of facial recognition systems with uneven performances on different population segments has highlighted the representativeness issues induced by a naive aggregation of the datasets. In this paper, we show how biasing models can remedy these problems. Based on the (approximate) knowledge of the biasing mechanisms at work, our approach consists in reweighting the observations, so as to form a nearly debiased estimator of the target distribution. One key condition is that the supports of the biased distributions must partly overlap, and cover the support of the target distribution. In order to meet this requirement in practice, we propose to use a low dimensional image representation, shared across the image databases. Finally, we provide numerical experiments highlighting the relevance of our approach.

Hybrid morphological-convolutional neural networks for computer-aided diagnosis.

Training deep Convolutional Neural Networks (CNNs) presents challenges in terms of memory requirements and computational resources, often resulting in issues such as model overfitting and lack of generalization. These challenges can only be mitigated by using an excessive number of training images. However, medical image datasets commonly suffer from data scarcity due to the complexities involved in their acquisition, preparation, and curation. To address this issue, we propose a compact and hybrid machine learning architecture based on the Morphological and Convolutional Neural Network (MCNN), followed by a Random Forest classifier. Unlike deep CNN architectures, the MCNN was specifically designed to achieve effective performance with medical image datasets limited to a few hundred samples. It incorporates various morphological operations into a single layer and uses independent neural networks to extract information from each signal channel. The final classification is obtained by utilizing a Random Forest classifier on the outputs of the last neural network layer. We compare the classification performance of our proposed method with three popular deep CNN architectures (ResNet-18, ShuffleNet-V2, and MobileNet-V2) using two training approaches: full training and transfer learning. The evaluation was conducted on two distinct medical image datasets: the ISIC dataset for melanoma classification and the ORIGA dataset for glaucoma classification. Results demonstrate that the MCNN method exhibits reliable performance in melanoma classification, achieving an AUC of 0.94 (95% CI: 0.91 to 0.97), outperforming the popular CNN architectures. For the glaucoma dataset, the MCNN achieved an AUC of 0.65 (95% CI: 0.53 to 0.74), which was similar to the performance of the popular CNN architectures. This study contributes to the understanding of mathematical morphology in shallow neural networks for medical image classification and highlights the potential of hybrid architectures in effectively learning from medical image datasets that are limited by a small number of case samples.

A paradigm shift in the prevention and diagnosis of oral squamous cell carcinoma.

Oral squamous cell carcinoma (OSCC) is a widespread disease with only 50%-60% 5-year survival. Individuals with potentially malignant precursor lesions are at high risk. Survival could be increased by effective, affordable, and simple screening methods, along with a shift from incisional tissue biopsies to non-invasive brush biopsies for cytology diagnosis, which are easy to perform in primary care. Along with the explainable, fast, and objective artificial intelligence characterisation of cells through deep learning, an easy-to-use, rapid, and cost-effective methodology for finding high-risk lesions is achievable. The collection of cytology samples offers the further opportunity of explorative genomic analysis. Our prospective multicentre study of patients with leukoplakia yields a vast number of oral keratinocytes. In addition to cytopathological analysis, whole-slide imaging and the training of deep neural networks, samples are analysed according to a single-cell RNA sequencing protocol, enabling mapping of the entire keratinocyte transcriptome. Mapping the changes in the genetic profile, based on mRNA expression, facilitates the identification of biomarkers that predict cancer transformation. This position paper highlights non-invasive methods for identifying patients with oral mucosal lesions at risk of malignant transformation. Reliable non-invasive methods for screening at-risk individuals bring the early diagnosis of OSCC within reach. The use of biomarkers to decide on a targeted therapy is most likely to improve the outcome. With the large-scale collection of samples following patients over time, combined with genomic analysis and modern machine-learning-based approaches for finding patterns in data, this path holds great promise.

Range estimation in a weakly range-dependent waveguide in the South China Sea using single hydrophone and multi-task trained deep neural network

In this paper, an underwater source range estimation method using a multi-task trained deep neural network is proposed. Unlike existing machine learning frameworks, in this case, the physical quantities utilized for the training and testing are not identical. The training data parameter is the phase of the complex sound pressure, which is independent of the source and receiver depth and can be generated easily by an acoustic propagation model with only a few environmental parameters, decreasing computational costs significantly. The testing data parameter is the realistically-measured complex sound pressure. Instead of simply using the range as the only supervision signal, two other parameters — the phase mode and the bottom sound speed — are applied as auxiliary labels during the training process, embedding additional information about the sound field into the deep model. The projection between the sound field features and the range extracted from the training data by the deep model is applied to range estimation task on testing data. The effectiveness of the proposed approach is verified for a weakly range-dependent waveguide in the South China Sea by localizing several upsweep signals within a frequency band of 20–200 Hz. Testing results illustrate that the proposed method outperforms the classical matched impulse response approach and deep-learning-based single-label trained models, with the mean absolute percentage error (MAPE) decreasing by 24.63% and 59.46%, respectively. The mechanism of promotion is explained by comparing visualized hidden features of the neural network using t-distributed stochastic neighbor embedding (t-SNE) between the single-task and multi-task approach, showing that the latter helps to construct a more reasonable data manifold in high-dimensional space and thus ensures better performance of the deep model. We also fine-tune the network with a few realistic samples and further improve the performance by up to 25.34% against the baseline.

Multiscale modeling of woven composites by deep learning neural networks and its application in design optimization

Due to complex yarn geometries and multiscale nature, it is very challenging to predict mechanical properties and strength constants of woven composites in an accuracy and efficient way. In this paper, mechanics of structure genome (MSG) is employed to predict effective material properties and strength constants of woven composites. The microscale analysis is performed to compute the effective properties and strength constants of yarns based on a hexagonal packed microstructure. Then, the mesoscale analysis is performed to compute the effective material properties and strength constants of woven composites. The accuracy and efficiency of the MSG method are compared with the computational homogenization method. In addition, simulation data using the MSG model is generated to train deep learning neural networks (DNN) models. The trained models provide ultra-efficient surrogate models with high accuracy. The DNN models are further applied to a design optimization problem to tailor geometry parameters to enhance the mechanical performance of plain woven composites, and the optimization can be efficiently solved with gradients from the DNN models. The developed computational framework can be applied to general woven composites featuring complex microscale and mesoscale features.

Semi-supervised and weakly-supervised deep neural networks and dataset for fish detection in turbid underwater videos

Fish are key members of marine ecosystems, and they have a significant share in the healthy human diet. Besides, fish abundance is an excellent indicator of water quality, as they have adapted to various levels of oxygen, turbidity, nutrients, and pH. To detect various fish in underwater videos, Deep Neural Networks (DNNs) can be of great assistance. However, training DNNs is highly dependent on large, labeled datasets, while labeling fish in turbid underwater video frames is a laborious and time-consuming task, hindering the development of accurate and efficient models for fish detection. To address this problem, firstly, we have collected a dataset called FishInTurbidWater, which consists of a collection of video footage gathered from turbid waters, and quickly and weakly (i.e., giving higher priority to speed over accuracy) labeled them in a 4-times fast-forwarding software. Next, we designed and implemented a semi-supervised contrastive learning fish detection model that is self-supervised using unlabeled data, and then fine-tuned with a small fraction (20%) of our weakly labeled FishInTurbidWater data. At the next step, we trained, using our weakly labeled data, a novel weakly-supervised ensemble DNN with transfer learning from ImageNet. The results show that our semi-supervised contrastive model leads to more than 20 times faster turnaround time between dataset collection and result generation, with reasonably high accuracy (89%). At the same time, the proposed weakly-supervised ensemble model can detect fish in turbid waters with high (94%) accuracy, while still cutting the development time by a factor of four, compared to fully-supervised models trained on carefully labeled datasets. Our dataset and code are publicly available at the hyperlink FishInTurbidWater.

STADIA: Photonic Stochastic Gradient Descent for Neural Network Accelerators

Deep Neural Networks (DNNs) have demonstrated great success in many fields such as image recognition and text analysis. However, the ever-increasing sizes of both DNN models and training datasets make deep leaning extremely computation- and memory-intensive. Recently, photonic computing has emerged as a promising technology for accelerating DNNs. While the design of photonic accelerators for DNN inference and forward propagation of DNN training has been widely investigated, the architectural acceleration for equally important backpropagation of DNN training has not been well studied. In this paper, we propose a novel silicon photonic-based backpropagation accelerator for high performance DNN training. Specifically, a general-purpose photonic gradient descent unit named STADIA is designed to implement the multiplication, accumulation, and subtraction operations required for computing gradients using mature optical devices including Mach-Zehnder Interferometer (MZI) and Mircoring Resonator (MRR), which can significantly reduce the training latency and improve the energy efficiency of backpropagation. To demonstrate efficient parallel computing, we propose a STADIA-based backpropagation acceleration architecture and design a dataflow by using wavelength-division multiplexing (WDM). We analyze the precision of STADIA by quantifying the precision limitations imposed by losses and noises. Furthermore, we evaluate STADIA with different element sizes by analyzing the power, area and time delay for photonic accelerators based on DNN models such as AlexNet, VGG19 and ResNet. Simulation results show that the proposed architecture STADIA can achieve significant improvement by 9.7× in time efficiency and 147.2× in energy efficiency, compared with the most advanced optical-memristor based backpropagation accelerator.

Prediction and optimization of 3D-printed sandwich beams with chiral cores

This study pursues two primary objectives concerning sandwich structures with chiral cores. Firstly, it delves into the realm of machine learning-assisted predictions regarding the behavior of these structures under compressive loads. In this way, three key geometric parameters of the repetitive chiral unit cells-thickness, angle, and diameter-are considered as highly influential design factors. Accordingly, a custom method for designing experiments was employed, resulting in achieving 27 totally different sandwich beams. These beam structures were made up of Poly Lactic Acid (PLA) for both their face sheets and cores, employing Fused Deposition Modeling (FDM) printing technology. Subsequently, they underwent compressive loading, where the resulting mechanical responses forming the foundational dataset for training Deep Neural Networks (DNNs). Remarkably, the DNNs were trained utilizing both Bayesian and conjugate gradient algorithms. The outcomes notably demonstrated that DNNs trained with the Bayesian algorithm exhibited superior accuracy in predicting stress-strain responses of the beams. Transitioning to the study's second objective, Response Surface Methodology (RSM) was harnessed to optimize Young's modulus and Specific Energy Absorption (SEA). The application of RSM impressively showcased its robustness in attaining optimal design values for these mechanical properties over the defined range of design parameters. This comprehensive exploration effectively reveals the potential of machine learning predictions and optimization techniques, providing valuable insights into the intricate mechanical behaviors of sandwich structures featuring innovative cores.

Skeleton-guided generation of synthetic noisy point clouds from as-built BIM to improve indoor scene understanding

The limited amount of high-quality training data available in indoor understanding with deep learning is a major problem. A possible solution to this problem is to use synthetic data to improve network training. In this study, a fully automatic method to generate synthetic noisy point clouds from as-built building information modeling (BIM) models is presented and it assesses the potential of these synthetic point clouds to improve deep neural network training. Based on a skeleton-guided strategy, all hypothetical scanning sites are located along the central axis of the buildings, which are obtained through equidistant sampling. Then, the synthetic labeled point cloud is generated station-by-station, and data augmentation is achieved using a random combination of data from different stations. The proposed approach involves generating over 44 sets of synthetic noisy point clouds based on BIM models. The performance of state-of-the-art (SOTA) deep learning methods in understanding indoor scenes enhanced by the synthetic point clouds is thoroughly assessed, and the effectiveness of various combinations of real and synthetic datasets is investigated. The experimental results demonstrate that leveraging synthetic point clouds generated from BIM models leads to a remarkable 5%–10% improvement in 3D semantic segmentation accuracy. The research signifies the value of synthetic point clouds as an effective tool for improving deep neural network training. All simulation datasets are publicly available, including original BIM models, full synthetic point clouds, and point clouds after IHPR processing, accessible via the BIMSyn Dataset link. In future research, an exploration of how synthetic point clouds will be further improved by considering specific characteristics of objects such as color, material reflectance, and illumination.

Active learning with label quality control.

Training deep neural networks requires a large number of labeled samples, which are typically provided by crowdsourced workers or professionals at a high cost. To obtain qualified labels, samples need to be relabeled for inspection to control the quality of the labels, which further increases the cost. Active learning methods aim to select the most valuable samples for labeling to reduce labeling costs. We designed a practical active learning method that adaptively allocates labeling resources to the most valuable unlabeled samples and the most likely mislabeled labeled samples, thus significantly reducing the overall labeling cost. We prove that the probability of our proposed method labeling more than one sample from any redundant sample set in the same batch is less than 1/k, where k is the number of the k-fold experiment used in the method, thus significantly reducing the labeling resources wasted on redundant samples. Our proposed method achieves the best level of results on benchmark datasets, and it performs well in an industrial application of automatic optical inspection.

Non-ideal program-time conservation in charge trap flash for deep learning

Training deep neural networks (DNNs) is computationally intensive, but arrays of non-volatile memories such as charge trap flash (CTF) can accelerate DNN operations using in-memory computing. Specifically, the resistive processing unit (RPU) architecture uses a voltage-threshold program with stochastic encoded pulse trains and analog memory features to accelerate vector-vector outer product and weight update for gradient descent algorithms. Although CTF, offering high precision, has been regarded as an excellent choice for implementing RPU, the accumulation of charge due to the applied stochastic pulse trains is ultimately of critical significance in determining the final weight update. In this paper, we report on the non-ideal program-time conservation in CTF through pulsing input measurements. We experimentally measure the effect of pulse width and pulse gap, keeping the total ON-time of the input pulse train constant, and report three non-idealities: (1) the cumulative shift reduces when total ON-time is fragmented into a larger number of shorter pulses, (2) the cumulative shift drops abruptly for pulse widths <2 s, (3) the cumulative shift depends on the gap between consecutive pulses and the shift reduction recovers for smaller gaps. We present an explanation based on a transient tunneling field enhancement due to blocking oxide trap-charge dynamics to explain these non-idealities. Identifying and modeling the responsible mechanisms and predicting their system-level effects during learning is critical. This non-ideal accumulation is expected to affect algorithms and architectures relying on devices for implementing mathematically equivalent functions for in-memory computing-based acceleration.

Generating synthetic gait patterns based on benchmark datasets for controlling prosthetic legs

BackgroundProsthetic legs help individuals with an amputation regain locomotion. Recently, deep neural network (DNN)-based control methods, which take advantage of the end-to-end learning capability of the network, have been proposed. One prominent challenge for these learning-based approaches is obtaining data for the training, particularly for the training of a mid-level controller. In this study, we propose a method for generating synthetic gait patterns (vertical load and lower limb joint angles) using a generative adversarial network (GAN). This approach enables a mid-level controller to execute ambulation modes that are not included in the training datasets.MethodsThe conditional GAN is trained on benchmark datasets that contain the gait data of individuals without amputation; synthetic gait patterns are generated from the user input. Further, a DNN-based controller for the generation of impedance parameters is trained using the synthetic gait pattern and the corresponding synthetic stiffness and damping coefficients.ResultsThe trained GAN generated synthetic gait patterns with a coefficient of determination of 0.97 and a structural similarity index of 0.94 relative to benchmark data that were not included in the training datasets. We trained a DNN-based controller using the GAN-generated synthetic gait patterns for level-ground walking, standing-to-sitting motion, and sitting-to-standing motion. Four individuals without amputation participated in bypass testing and demonstrated the ambulation modes. The model successfully generated control parameters for the knee and ankle based on thigh angle and vertical load.ConclusionsThis study demonstrates that synthetic gait patterns can be used to train DNN models for impedance control. We believe a conditional GAN trained on benchmark datasets can provide reliable gait data for ambulation modes that are not included in its training datasets. Thus, designing gait data using a conditional GAN could facilitate the efficient and effective training of controllers for prosthetic legs.

Low Complexity Gradient Computation Techniques to Accelerate Deep Neural Network Training.

Deep neural network (DNN) training is an iterative process of updating network weights, called gradient computation, where (mini-batch) stochastic gradient descent (SGD) algorithm is generally used. Since SGD inherently allows gradient computations with noise, the proper approximation of computing weight gradients within SGD noise can be a promising technique to save energy/time consumptions during DNN training. This article proposes two novel techniques to reduce the computational complexity of the gradient computations for the acceleration of SGD-based DNN training. First, considering that the output predictions of a network (confidence) change with training inputs, the relation between the confidence and the magnitude of the weight gradient can be exploited to skip the gradient computations without seriously sacrificing the accuracy, especially for high confidence inputs. Second, the angle diversity-based approximations of intermediate activations for weight gradient calculation are also presented. Based on the fact that the angle diversity of gradients is small (highly uncorrelated) in the early training epoch, the bit precision of activations can be reduced to 2-/4-/8-bit depending on the resulting angle error between the original gradient and quantized gradient. The simulations show that the proposed approach can skip up to 75.83% of gradient computations with negligible accuracy degradation for CIFAR-10 dataset using ResNet-20. Hardware implementation results using 65-nm CMOS technology also show that the proposed training accelerator achieves up to 1.69x energy efficiency compared with other training accelerators.

FlexBlock: A Flexible DNN Training Accelerator With Multi-Mode Block Floating Point Support

When training deep neural networks (DNNs), expensive floating point arithmetic units are used in GPUs or custom neural processing units (NPUs). To reduce the burden of floating point arithmetic, community has started exploring the use of more efficient data representations, e.g., block floating point (BFP). The BFP format allows a group of values to share an exponent, which effectively reduces the memory footprint and enables cheaper fixed point arithmetic for multiply-accumulate (MAC) operations. However, existing BFP-based DNN accelerators are targeted for a specific precision, making them less versatile. In this paper, we present FlexBlock, a DNN training accelerator with three BFP modes, possibly different among activation, weight, and gradient tensors. By configuring FlexBlock to a lower BFP precision, the number of MACs handled by the core increases by up to 4× in 8-bit mode or 16× in 4-bit mode compared to 16-bit mode. To reach this theoretical upper bound, FlexBlock maximizes the core utilization at various precision levels or layer types, and allows dynamic precision control to keep throughput at its peak without sacrificing training accuracy. We evaluate the effectiveness of FlexBlock using representative DNNs on CIFAR, ImageNet and WMT14 datasets. As a result, training in FlexBlock significantly improves training speed by 1.5 5.3× and energy efficiency by 2.4 7.0× compared to other training accelerators.

Neural-optimal tuning of a controller for a parallel robot

In this article, a double strategy is proposed to find the optimal gains of a cascaded PI controller to minimize the trajectory position error in a five-bar parallel robot. The first strategy employs Differential Evolution to tune constant gains during the execution time of the desired trajectory. Once Differential Evolution achieves convergence on the solution by finding the vector of optimal gains that minimize the position tracking error, all the position error data and current of the two brushless motors are saved. In the second strategy, the data generated in the first strategy is used to train a Deep Neural Network. After that, the trained Deep Neural Network replaces the constant gains of the first strategy with time-varying gains for the desired trajectory. Three working scenarios are proposed to test the generalization of the Deep Neural Network. In the first scenario, a training trajectory is executed. In the second one, a testing trajectory of the Deep Neural Network is evaluated. In the third one, a mass change is generated in the middle of the cycle. The results show that the Deep Neural Network is robust to different trajectories and mass changes during the execution of pick and place tasks.

VoxelScape: Large Scale Simulated 3D Point Cloud Dataset of Urban Traffic Environments

Having a profound understanding of the surrounding environment is considered one of the crucial tasks for the reliable operation of future self-driving cars. Light Detection and Ranging (LiDAR) sensor plays a critical role in achieving such understanding due to its capability to perceive the world in 3D. Similar to 2D perception tasks, current state-of-the-art methods in 3D perception tasks rely on deep neural networks (DNNs). However, the performance of 3D perception tasks, specially point-wise semantic segmentation, is not on par with their 2D counterparts. One of the main reasons is the lack of publicly available labelled 3D point cloud datasets (PCDs) from 3D LiDAR sensors. In this work, we are introducing the VoxelScape dataset, a large-scale simulated 3D PCD with 100K annotated point cloud scans. The annotations in the VoxelScape dataset include both point-wise semantic labels and 3D bounding box labels. Additionally, we used a number of baseline approaches to validate the transferability of VoxelScape to real 3D PCD for two challenging 3D perception tasks. The promising results have shown that training DNNs on VoxelScape boosted the performance of the 3D perception tasks on the real PCD. Furthermore, we are also releasing the proposed data generation pipeline for the research community to facilitate realistic simulation of 3D LiDAR point cloud data for different scenarios beyond those covered in our VoxelScape dataset. The VoxelScape dataset and the corresponding LiDAR simulation codes are publicly available at https://voxel-scape.github.io/dataset

Deep Neural Network Direct Stiffness Method: Fundamental Steps into Beam Design

AbstractThis paper presents a novel approach for performing beam element analysis that considers the nonlinear deformation behavior of various RHS and SHS sections made of mild to high‐strength steel: the Deep Neural Network Direct Stiffness method (DNN‐DSM), which uses Deep Neural Networks (DNN), a subset of machine learning algorithms and more general artificial intelligence approaches, to predict the nonlinear stiffness matrix terms in a beam element formulation for implementation in the direct stiffness matrix (DSM). These predictions are made using trained DNN models drawn from an extensive pool of geometric‐material nonlinear simulations with additional imperfections (GMNIA) using shell‐based models. Initial implementations of this method are able to predict the nonlinear load‐displacement and moment‐rotation behavior of various cross sections with high accuracy. This combines the precision of shell analysis with the computational efficiency of beam element analysis. Previous publications have already demonstrated the suitability and advantages of this method, albeit on a small scale for local buckling prediction. This work goes beyond previous studies and focuses on finite beam element design. This includes a review of the modeling approaches of DNN‐DSM using experimental and numerical results from literature.