Training Deep Neural Networks Research Articles

Thermal buckling of the concrete structures reinforced by nanocomposites: Introducing advanced deep neural networks for thermal buckling problem

The thermal buckling behavior of concrete annular sector plates reinforced with nanocomposites is investigated in this work. The thermal buckling issue is stated using mathematical modeling, taking into account the geometric linearity and material heterogeneity caused by the nanocomposite reinforcement. The governing differential equations that are obtained from the theoretical model are solved using the discrete singular convolution (DSC) technique. To generate accurate buckling load predictions, the DSC method’s great accuracy in handling complicated boundary conditions and discontinuities is used. Furthermore, using mathematical modeling to create extensive datasets, the research aims to train deep neural networks (DNNs) to anticipate thermal buckling reactions. These datasets provide a strong basis for DNN training as they include a wide range of factors, such as geometric configurations, loading situations, and material properties. By combining DNN predictions with DSC solutions, this unique technique for solving this difficult issue seeks to improve the accuracy and efficiency of thermal buckling analysis. The findings show how well sophisticated mathematical methods and machine learning models work together, opening the door to creative solutions for the design and analysis of concrete structures reinforced with nanocomposite under complicated stress.

ReSmooth: Detecting and Utilizing OOD Samples When Training With Data Augmentation.

Data augmentation (DA) is a widely used technique for enhancing the training of deep neural networks. Recent DA techniques which achieve state-of-the-art performance always meet the need for diversity in augmented training samples. However, an augmentation strategy that has a high diversity usually introduces out-of-distribution (OOD) augmented samples and these samples consequently impair the performance. To alleviate this issue, we propose ReSmooth, a framework that first detects OOD samples in augmented samples and then leverages them. To be specific, we first use a Gaussian mixture model (GMM) to fit the loss distribution of both the original and augmented samples and accordingly split these samples into in-distribution (ID) samples and OOD samples. Then we start a new training where ID and OOD samples are incorporated with different smooth labels. By treating ID samples and OOD samples unequally, we can make better use of the diverse augmented data. Furthermore, we incorporate our ReSmooth framework with negative DA (NDA) strategies. By properly handling their intentionally created OOD samples, the classification performance of NDAs is largely ameliorated. Experiments on several classification benchmarks show that ReSmooth can be easily extended to the existing augmentation strategies [such as RandAugment (RA), rotate, and jigsaw] and improve on them. Our code is available at https://github.com/Chenyang4/ReSmooth.

Automatic Ground-Truth Image Labeling for Deep Neural Network Training and Evaluation Using Industrial Robotics and Motion Capture

Abstract The United States Navy (USN) intends to increase the amount of uncrewed aircraft in a carrier air wing. To support this increase, carrier-based uncrewed aircraft will be required to have some level of autonomy as there will be situations where a human cannot be in/on the loop. However, there is no existing and approved method to certify autonomy within Naval Aviation. In support of generating certification evidence for autonomy, the United States Naval Academy (USNA) has created a training and evaluation system (TES) to provide quantifiable metrics for feedback performance in autonomous systems. The preliminary use case for this work focuses on autonomous aerial refueling. Prior demonstrations of autonomous aerial refueling have leveraged a deep neural network (DNN) for processing visual feedback to approximate the relative position of an aerial refueling drogue. The training and evaluation system proposed in this work simulates the relative motion between the aerial refueling drogue and feedback camera system using industrial robotics. Ground-truth measurements of the pose between the camera and drogue are measured using a commercial motion capture system. Preliminary results demonstrate calibration methods providing ground-truth measurements with millimeter precision. Leveraging this calibration, the proposed system is capable of providing large-scale datasets for DNN training and evaluation against a precise ground truth.

Isolated Pulsar Population Synthesis with Simulation-based Inference

We combine pulsar population synthesis with simulation-based inference (SBI) to constrain the magnetorotational properties of isolated Galactic radio pulsars. We first develop a framework to model neutron star birth properties and their dynamical and magnetorotational evolution. We specifically sample initial magnetic field strengths, B, and spin periods, P, from lognormal distributions and capture the late-time magnetic field decay with a power law. Each lognormal is described by a mean, μlogB,μlogP , and standard deviation, σlogB,σlogP , while the power law is characterized by the index, a late. We subsequently model the stars’ radio emission and observational biases to mimic detections with three radio surveys, and we produce a large database of synthetic P– Ṗ diagrams by varying our five magnetorotational input parameters. We then follow an SBI approach that focuses on neural posterior estimation and train deep neural networks to infer the parameters’ posterior distributions. After successfully validating these individual neural density estimators on simulated data, we use an ensemble of networks to infer the posterior distributions for the observed pulsar population. We obtain μlogB=13.10−0.10+0.08 , σlogB=0.45−0.05+0.05 and μlogP=−1.00−0.21+0.26 , σlogP=0.38−0.18+0.33 for the lognormal distributions and alate=−1.80−0.61+0.65 for the power law at the 95% credible interval. We contrast our results with previous studies and highlight uncertainties of the inferred a late value. Our approach represents a crucial step toward robust statistical inference for complex population synthesis frameworks and forms the basis for future multiwavelength analyses of Galactic pulsars.

A non-monotone trust-region method with noisy oracles and additional sampling

In this work, we introduce a novel stochastic second-order method, within the framework of a non-monotone trust-region approach, for solving the unconstrained, nonlinear, and non-convex optimization problems arising in the training of deep neural networks. The proposed algorithm makes use of subsampling strategies that yield noisy approximations of the finite sum objective function and its gradient. We introduce an adaptive sample size strategy based on inexpensive additional sampling to control the resulting approximation error. Depending on the estimated progress of the algorithm, this can yield sample size scenarios ranging from mini-batch to full sample functions. We provide convergence analysis for all possible scenarios and show that the proposed method achieves almost sure convergence under standard assumptions for the trust-region framework. We report numerical experiments showing that the proposed algorithm outperforms its state-of-the-art counterpart in deep neural network training for image classification and regression tasks while requiring a significantly smaller number of gradient evaluations.

Phishing Website Detection

Phishing websites have emerged as a serious security risk. Phishing is the starting point for many cyberattacks that compromise the confidentiality, integrity, and availability of customer and business data. Decades of effort have gone into developing novel methods for automatically identifying phishing websites. Modern systems aren't very adept at spotting new phishing threats and require a lot of manual feature engineering, even though they can produce better outcomes. Thus, an open problem in this discipline is to identify tactics that can swiftly handle zero-day phishing attempts and automatically recognize phishing websites. The web page that the URL hosts has a plethora of information that can be utilized to assess the maliciousness of the web server. One useful technique for spotting phishing emails is machine learning. Additionally, it does away with the drawbacks of the earlier approach. After a careful analysis of the literature, we proposed a novel approach that combines a machine learning algorithm with feature extraction to identify phishing websites. Using the gathered dataset, this study aims to train deep neural networks and machine learning models to detect phishing websites.

BAMFORESTS: Bamberg Benchmark Forest Dataset of Individual Tree Crowns in Very-High-Resolution UAV Images

The anthropogenic climate crisis results in the gradual loss of tree species in locations where they were previously able to grow. This leads to increasing workloads and requirements for foresters and arborists as they are forced to restructure their forests and city parks. The advancements in computer vision (CV)—especially in supervised deep learning (DL)—can help cope with these new tasks. However, they rely on large, carefully annotated datasets to produce good and generalizable models. This paper presents BAMFORESTS: a dataset with 27,160 individually delineated tree crowns in 105 ha of very-high-resolution UAV imagery gathered with two different sensors from two drones. BAMFORESTS covers four areas of coniferous, mixed, and deciduous forests and city parks. The labels contain instance segmentations of individual trees, and the proposed splits are balanced by tree species and vitality. Furthermore, the dataset contains the corrected digital surface model (DSM), representing tree heights. BAMFORESTS is annotated in the COCO format and is especially suited for training deep neural networks (DNNs) to solve instance segmentation tasks. BAMFORESTS was created in the BaKIM project and is freely available under the CC BY 4.0 license.

A comprehensive exploration of approximate DNN models with a novel floating-point simulation framework

This paper introduces TorchAxf11https://github.com/knuscent/TorchAxf., a framework for fast simulation of diverse approximate deep neural network (DNN) models, including spiking neural networks (SNNs). The proposed framework utilizes various approximate adders and multipliers, supports industrial standard reduced precision floating-point formats, such as bfloat16, and accommodates user-customized precision representations. Leveraging GPU acceleration on the PyTorch framework, TorchAxf accelerates approximate DNN training and inference. In addition, it allows seamless integration of arbitrary approximate arithmetic algorithms with C/C++ behavioral models to emulate approximate DNN hardware accelerators.We utilize the proposed TorchAxf framework to assess twelve popular DNN models under approximate multiply-and-accumulate (MAC) operations. Through comprehensive experiments, we determine the suitable degree of floating-point arithmetic approximation for these DNN models without significant accuracy loss and offer the optimal reduced precision formats for each DNN model. Additionally, we demonstrate that approximate-aware re-training can rectify errors and enhance pre-trained DNN models under reduced precision formats. Furthermore, TorchAxf, operating on GPU, remarkably reduces simulation time for complex DNN models using approximate arithmetic by up to 131.38× compared to the baseline optimized CPU implementation. Finally, we compare the proposed framework with state-of-the-art frameworks to highlight its superiority.

Scalable bio-inspired training of Deep Neural Networks with FastHebb

Recent work on sample efficient training of Deep Neural Networks (DNNs) proposed a semi-supervised methodology based on biologically inspired Hebbian learning, combined with traditional backprop-based training. Promising results were achieved on various computer vision benchmarks, in scenarios of scarce labeled data availability. However, current Hebbian learning solutions can hardly address large-scale scenarios due to their demanding computational cost. In order to tackle this limitation, in this contribution, we investigate a novel solution, named FastHebb (FH), based on the reformulation of Hebbian learning rules in terms of matrix multiplications, which can be executed more efficiently on GPU. Starting from Soft-Winner-Takes-All (SWTA) and Hebbian Principal Component Analysis (HPCA) learning rules, we formulate their improved FH versions: SWTA-FH and HPCA-FH. We experimentally show that the proposed approach accelerates training speed up to 70 times, allowing us to gracefully scale Hebbian learning experiments on large datasets and network architectures such as ImageNet and VGG.

Entropy-based deep neural network training optimization for optical coherence tomography imaging

ABSTRACT This paper presents an optimization technique for the number of training epochs needed for deep learning models. The proposed method eliminates the need for separate validation data and significantly decreases training epochs. Using a four-class Optical Coherence Tomography (OCT) image dataset encompassing Choroidal Neovascularization (CNV), Diabetic Macular Edema (DME), Drusen, and Normal retina categories, we evaluated twelve architectures. These include general-purpose models (Alexnet, VGG11, VGG13, VGG16, VGG19, ResNet-18, ResNet-34, and ResNet-50) and OCT image-specific models (RetiNet, AOCT-NET, DeepOCT, and Octnet). The proposed technique reduced training epochs ranging from 4.35% to 58.27% for all architectures except Alexnet. Although the overall increase in accuracy ranges from 0.28% to 12.6%, with some architectures experiencing minor improvements, this is seen as acceptable considering the substantial reduction in training time. By achieving higher accuracy with fewer training epochs and eliminating the need for separate validation data, our methodology streamlines early stopping significantly. Statistical evaluations via Shapiro-Wilk and Kruskal-Wallis tests further affirm these results, showcasing the potential of this novel technique for efficient deep learning practices in scenarios constrained by time or computational resources.

Multiple Image Distortion DNN Modeling Individual Subject Quality Assessment

A recent research direction is focused on training Deep Neural Networks (DNNs) to replicate individual subject assessments of media quality. These DNNs are referred to as Artificial Intelligence-based Observers (AIOs). An AIO is designed to simulate, in real-time, the quality ratings of a specific individual, enabling an automatic quality assessment that accounts for subjects characteristics and preferences. Training AIOs is a promising but challenging research area due to the greater noise in individual raw opinion scores compared to the Mean Opinion Score (MOS). Effective learning from noisy labels necessitates the training of complex models on large-scale datasets. Unfortunately, this is challenging for AIOs as the media quality assessment community lacks extensive datasets that include individual opinion scores. To address the complexity of the task, we first created a dataset comprising two million samples, with synthetic labels derived from human annotation. We then trained a customized network for image quality assessment, named Multi-Distortion ResNet50 (MDResNet50), on this dataset. The weights of the MDResNet50 were subsequently utilized to initialize the learning process of each AIO, thereby avoiding the need to train a complex model from scratch on a small-scale dataset with raw individual opinion scores. Computational experiments show that our approach significantly advances the state-of-the-art in the AIO research. In particular: i) we demonstrate through a simulation the ability of AIOs to mimic two well-known behavioral characteristics of a subject, i.e.,bias and inconsistency, when scoring the media quality; ii) we train and release DNN-based AIOs that, compared to the state-of-the-art, exhibit a higher performance with a statistical significance in assessing multiple image distortions; iii) we train AIOs that more accurately mimic the sensitivity of real subjects to noise and color saturation, and also better predict the opinion score distribution compared to the state-of-the-art AIOs.

Boosting few-shot rare skin disease classification via self-supervision and distribution calibration.

Due to the difficulty in obtaining clinical samples and the high cost of labeling, rare skin diseases are characterized by data scarcity, making training deep neural networks for classification challenging. In recent years, few-shot learning has emerged as a promising solution, enabling models to recognize unseen disease classes by limited labeled samples. However, most existing methods ignored the fine-grained nature of rare skin diseases, resulting in poor performance when generalizing to highly similar classes. Moreover, the distributions learned from limited labeled data are biased, severely impairing the model's generalizability. This paper proposes a self-supervision distribution calibration network (SS-DCN) to address the above issues. Specifically, SS-DCN adopts a multi-task learning framework during pre-training. By introducing self-supervised tasks to aid in supervised learning, the model can learn more discriminative and transferable visual representations. Furthermore, SS-DCN applied an enhanced distribution calibration (EDC) strategy, which utilizes the statistics of base classes with sufficient samples to calibrate the bias distribution of novel classes with few-shot samples. By generating more samples from the calibrated distribution, EDC can provide sufficient supervision for subsequent classifier training. The proposed method is evaluated on three public skin disease datasets(i.e., ISIC2018, Derm7pt, and SD198), achieving significant performance improvements over state-of-the-art methods.

BmmW: A DNN-based joint BLE and mmWave radar system for accurate 3D localization with goal-oriented communication

Bluetooth Low Energy (BLE) has emerged as one of the reference technologies for the development of indoor localization systems, due to its increasing ubiquity, low-cost hardware, and to the introduction of direction-finding enhancements improving its ranging performance. However, the intrinsic narrowband nature of BLE makes this technology susceptible to multipath and channel interference. As a result, it is still challenging to achieve decimetre-level localization accuracy, which is necessary when developing location-based services for smart factories and workspaces. To address this challenge, we present BmmW, an indoor localization system that augments the ranging estimates obtained with BLE5.1’s constant tone extension feature with mmWave radar measurements to provide 3D localization of a mobile tag with decimetre-level accuracy. Specifically, BmmW embeds a deep neural network (DNN) that is jointly trained with both BLE and mmWave measurements, practically leveraging the strengths of both technologies. In fact, mmWave radars can locate objects and people with decimetre-level accuracy, but their effectiveness in monitoring stationary targets and multiple objects is limited, and they also suffer from a fast signal attenuation limiting the usable range to a few meters. We evaluate BmmW’s performance experimentally, and show that its joint DNN training scheme allows to track mobile tags with a mean 3D localization accuracy of 10 cm when combining angle-of-arrival BLE measurements with mmWave radar data. We further assess two variations of BmmW: BmmW-Lite and BmmW-Lite+, both tailored for single-antenna BLE devices, which eliminates the necessity for bulky and expensive multi-antenna arrays and represents a cost-effective solution that is easy to integrate into compact IoT devices. In contrast to classic BmmW (which utilizes angle-of-arrival info), BmmW-Lite uses raw in-phase/quadrature (I/Q) measurements, and achieves a mean localization accuracy of 36 cm, thus facilitating precise object tracking in indoor environments even when using budget-friendly single-antenna BLE devices. BmmW-Lite+ extends BmmW-Lite by allowing the localization task to be transferred from the edge to the cloud due to device memory and power constraints. To this end, BmmW-Lite+ employs a goal-oriented communication paradigm that compresses initial BLE features into a more compact semantic format at the edge device, which allows to minimize the amount of data that needs to be sent to the cloud. Our experimental results show that BmmW-Lite+ can compress raw BLE features by up to 12% of their initial size (hence allowing to save network bandwidth and minimize energy consumption), with negligible impact on the localization accuracy.

Learning Gaze-aware Compositional GAN from Limited Annotations

Gaze-annotated facial data is crucial for training deep neural networks (DNNs) for gaze estimation. However, obtaining these data is labor-intensive and requires specialized equipment due to the challenge of accurately annotating the gaze direction of a subject. In this work, we present a generative framework to create annotated gaze data by leveraging the benefits of labeled and unlabeled data sources. We propose a Gaze-aware Compositional GAN that learns to generate annotated facial images from a limited labeled dataset. Then we transfer this model to an unlabeled data domain to take advantage of the diversity it provides. Experiments demonstrate our approach's effectiveness in generating within-domain image augmentations in the ETH-XGaze dataset and cross-domain augmentations in the CelebAMask-HQ dataset domain for gaze estimation DNN training. We also show additional applications of our work, which include facial image editing and gaze redirection.

Cross-domain structural damage identification using transfer learning strategy

In damage identification of civil structures, training deep neural networks (DNNs) often requires a large volume of annotated training data. However, artificial damage to real-world structures is always forbidden, resulting in insufficient data for training. Transfer learning provides a solution that allows structural damage information in a data-rich domain to be transferred and shared as the prior knowledge to a data-scarce domain, thereby indirectly augmenting available training data in the latter domain. To this end, a multi-task transfer learning strategy has been adopted for the purpose of achieving cross-domain damage identification between a plane frame in the source domain and a three-dimensional frame in the target domain. The strategy can share the learning knowledge from the domain with sufficient training data to the target domain with insufficient data. Thereby, the DNN (named DNN#2) in the target domain can be trained on a small amount of training data. Owing to their ability in data feature extraction, stacked auto-encoders (SAEs) are used to construct the desirable DNN#1 and DNN#2 corresponding to different damage identification tasks, named Task#1 and Task#2, in the two domains. The two DNNs share the hidden layers in order to share damage feature information between the source and target domains. The training datasets of the two domains are first used to jointly pre-train the auto-encoders’ parameters in an unsupervised learning way. Afterwards, a supervised fine-tuning step is carried out to retraining the entire SAEs for better performance. By these means, Task#2 receives some prior knowledge from Task#1, and thus, it is accomplished on limited training data when Task#1 is synchronously achieved. The analysis results demonstrate that the proposed multi-task learning strategy requires only a single training session to simultaneously realize the cross-domain damage identification of two different steel frames.

Cost-minimization and Implicit Innovative Training for Energy Prediction using Deep Neural Network

To alleviate adverse environmental impacts, power stations and energy grids must optimize resource application for power generation. Accordingly, soothsaying guests' energy consumption has come integral to every energy operation system. exercising data from smart homes, energy operation information can train deep neural networks to anticipate unborn energy demands. As the frugality advances, both energy product and consumption have steadily increased over the times. Amidst global enterprises over energy force and environmental challenges, this study introduces a new vaticination approach using neural networks. By using statistical data from the energy assiduity, these networks directly read changes in energy product and consumption trends. Numerical findings validate the efficacity of this neural network- grounded vaticination system, emphasizing its significance in energy conservation sweats. Predicting energy consumption stands as a pivotal bid in energy conservation enterprise. Support vector retrogression, famed for its efficacity in handlingnon-linear data retrogression challenges, has surfaced as a prominent tool for soothsaying structure energy consumption. Through analysis of literal data, it's apparent that the relationship between lighting energy consumption and its impacting factors is non-linear.

Hierarchical Training of Deep Neural Networks Using Early Exiting.

Deep neural networks (DNNs) provide state-of-the-art accuracy for vision tasks, but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime, and privacy concerns. In this study, a novel hierarchical training method for DNNs is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime, and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that, due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud, and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification, respectively, when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved, while the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy DNNs on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

IoT device type identification using training deep quantum neural networks optimized with a chimp optimization algorithm for enhancing IoT security

IoT networks can be defined as groups of physically connected things and devices that can connect to the Internet and exchange data with one another. Since enabling an increasing number of internets of things devices to connect with their networks, organizations have become more vulnerable to safety issues and attacks. A major drawback of previous research is that it can find out prior seen types only, also any new device types are considered anomalous. In this manuscript, IoT device type detection utilizing Training deep quantum neural networks optimized with a Chimp optimization algorithm for enhancing IOT security (IOT-DTI-TDQNN-COA-ES) is proposed. The proposed method entails three phases namely data collection, feature extraction and detection. For Data collection phase, real network traffic dataset from different IoT device types are collected. For feature mining phase, the internet traffic features are extracted through automated building extraction (ABE) method. IoT device type identification phase, Training deep quantum neural networks (TDQNN) optimized with Chimp optimization algorithm (COA) is utilized to detect the category of IoT devices as known and unknown device. IoT network is implemented in Python. Then the simulation performance of the proposed IOT-DTI-TDQNN-COA-ES method attains higher accuracy as26.82% and 23.48% respectively, when compared with the existing methods.

VLSI architecture of stochastic genetic algorithm for real time training of deep neural network

VLSI architecture of stochastic genetic algorithm for real time training of deep neural network

Machine-Learning driven STM images prediction of doped/defective graphene: Towards optimized tools for 2D nanomaterials characterization

Scanning tunneling microscopy (STM) is a key characterization technique that allows for visualization of specific features at nanostructures, given its ability to reveal changes in local charge densities in doping or defective sites. Besides the experimental acquisition of STM data from real samples, there is the theoretical route, which allows for STM images simulation from ab-initio calculations on defined nanostructures. This work presents an alternative route for theoretical STM image prediction by means of Machine Learning (ML) methods, based on the training of deep convolutional neural networks using simulated STM data. The ML-based STM predictions for defective, B- and N-doped graphene quantum dots are compared to ab-initio STM simulations, in terms of accuracy of structural rearrangements, charge density localizations, and computational resources requirements. The results demonstrate that STM images predicted by ML-methods are accurate at doping sites, whereas they show good similarity to simulations of large structural vacancies. This work represents a novel alternative for the use of ML methods in image-based characterization routines of doped/defective graphene, and potentially, for other 2-dimensional nanomaterials.