Training Deep Neural Networks Research Articles

Accuracy Assessment and Reinforcement Training of Deep Neural Networks in the Context of Accelerating a Simulation-Based Toolpath Generation Method

Accuracy Assessment and Reinforcement Training of Deep Neural Networks in the Context of Accelerating a Simulation-Based Toolpath Generation Method

Toward Interpretable Cell Image Representation and Abnormality Scoring for Cervical Cancer Screening Using Pap Smears.

Screening is critical for prevention and early detection of cervical cancer but it is time-consuming and laborious. Supervised deep convolutional neural networks have been developed to automate pap smear screening and the results are promising. However, the interest in using only normal samples to train deep neural networks has increased owing to the class imbalance problems and high-labeling costs that are both prevalent in healthcare. In this study, we introduce a method to learn explainable deep cervical cell representations for pap smear cytology images based on one-class classification using variational autoencoders. Findings demonstrate that a score can be calculated for cell abnormality without training models with abnormal samples, and we localize abnormality to interpret our results with a novel metric based on absolute difference in cross-entropy in agglomerative clustering. The best model that discriminates squamous cell carcinoma (SCC) from normals gives 0.908±0.003 area under operating characteristic curve (AUC) and one that discriminates high-grade epithelial lesion (HSIL) 0.920±0.002 AUC. Compared to other clustering methods, our method enhances the V-measure and yields higher homogeneity scores, which more effectively isolate different abnormality regions, aiding in the interpretation of our results. Evaluation using an external dataset shows that our model can discriminate abnormality without the need for additional training of deep models.

ReSmooth: Detecting and Utilizing OOD Samples When Training With Data Augmentation.

Data augmentation (DA) is a widely used technique for enhancing the training of deep neural networks. Recent DA techniques which achieve state-of-the-art performance always meet the need for diversity in augmented training samples. However, an augmentation strategy that has a high diversity usually introduces out-of-distribution (OOD) augmented samples and these samples consequently impair the performance. To alleviate this issue, we propose ReSmooth, a framework that first detects OOD samples in augmented samples and then leverages them. To be specific, we first use a Gaussian mixture model (GMM) to fit the loss distribution of both the original and augmented samples and accordingly split these samples into in-distribution (ID) samples and OOD samples. Then we start a new training where ID and OOD samples are incorporated with different smooth labels. By treating ID samples and OOD samples unequally, we can make better use of the diverse augmented data. Furthermore, we incorporate our ReSmooth framework with negative DA (NDA) strategies. By properly handling their intentionally created OOD samples, the classification performance of NDAs is largely ameliorated. Experiments on several classification benchmarks show that ReSmooth can be easily extended to the existing augmentation strategies [such as RandAugment (RA), rotate, and jigsaw] and improve on them. Our code is available at https://github.com/Chenyang4/ReSmooth.

Isolated Pulsar Population Synthesis with Simulation-based Inference

We combine pulsar population synthesis with simulation-based inference (SBI) to constrain the magnetorotational properties of isolated Galactic radio pulsars. We first develop a framework to model neutron star birth properties and their dynamical and magnetorotational evolution. We specifically sample initial magnetic field strengths, B, and spin periods, P, from lognormal distributions and capture the late-time magnetic field decay with a power law. Each lognormal is described by a mean, μlogB,μlogP , and standard deviation, σlogB,σlogP , while the power law is characterized by the index, a late. We subsequently model the stars’ radio emission and observational biases to mimic detections with three radio surveys, and we produce a large database of synthetic P– Ṗ diagrams by varying our five magnetorotational input parameters. We then follow an SBI approach that focuses on neural posterior estimation and train deep neural networks to infer the parameters’ posterior distributions. After successfully validating these individual neural density estimators on simulated data, we use an ensemble of networks to infer the posterior distributions for the observed pulsar population. We obtain μlogB=13.10−0.10+0.08 , σlogB=0.45−0.05+0.05 and μlogP=−1.00−0.21+0.26 , σlogP=0.38−0.18+0.33 for the lognormal distributions and alate=−1.80−0.61+0.65 for the power law at the 95% credible interval. We contrast our results with previous studies and highlight uncertainties of the inferred a late value. Our approach represents a crucial step toward robust statistical inference for complex population synthesis frameworks and forms the basis for future multiwavelength analyses of Galactic pulsars.

Deep Learning Based Detection of Toxic Mushrooms in Karnataka

Mushrooms are fungi that play significant roles in ecosystems, food, and medicine. However, distinguishing toxic mushrooms from non-toxic ones is challenging, as their visual characteristics can be misleading. Deep learning, a subset of artificial intelligence, offers a promising approach to detecting and classifying mushrooms based on their toxicity. By training deep neural networks on datasets of labelled mushroom images, it becomes possible to create models capable of accurately differentiating between toxic and non-toxic mushrooms. Mushrooms are fungi that play significant roles in ecosystems, food, and medicine. However, manually distinguishing toxic mushrooms from non-toxic ones is a challenging and time-consuming task. This research work is to help the local farmers and exporters to identify the difference between the toxic and the edible mushroom which will eventually boost production and will directly have an impact on the economic growth in India. This paper compared different deep learning algorithms such as Support Vector Machine (SVM), Resnet50, YOLO V5 and Alexnet, which have been found to be the most useful in classifying the mushrooms effectively as toxic and non-toxic respectively. The patterns in the images of the mushrooms have been predicted through the above classifiers. The validation techniques show SVM to be the most effective with an accuracy of 83%.

Phishing Website Detection

Phishing websites have emerged as a serious security risk. Phishing is the starting point for many cyberattacks that compromise the confidentiality, integrity, and availability of customer and business data. Decades of effort have gone into developing novel methods for automatically identifying phishing websites. Modern systems aren't very adept at spotting new phishing threats and require a lot of manual feature engineering, even though they can produce better outcomes. Thus, an open problem in this discipline is to identify tactics that can swiftly handle zero-day phishing attempts and automatically recognize phishing websites. The web page that the URL hosts has a plethora of information that can be utilized to assess the maliciousness of the web server. One useful technique for spotting phishing emails is machine learning. Additionally, it does away with the drawbacks of the earlier approach. After a careful analysis of the literature, we proposed a novel approach that combines a machine learning algorithm with feature extraction to identify phishing websites. Using the gathered dataset, this study aims to train deep neural networks and machine learning models to detect phishing websites.

BAMFORESTS: Bamberg Benchmark Forest Dataset of Individual Tree Crowns in Very-High-Resolution UAV Images

The anthropogenic climate crisis results in the gradual loss of tree species in locations where they were previously able to grow. This leads to increasing workloads and requirements for foresters and arborists as they are forced to restructure their forests and city parks. The advancements in computer vision (CV)—especially in supervised deep learning (DL)—can help cope with these new tasks. However, they rely on large, carefully annotated datasets to produce good and generalizable models. This paper presents BAMFORESTS: a dataset with 27,160 individually delineated tree crowns in 105 ha of very-high-resolution UAV imagery gathered with two different sensors from two drones. BAMFORESTS covers four areas of coniferous, mixed, and deciduous forests and city parks. The labels contain instance segmentations of individual trees, and the proposed splits are balanced by tree species and vitality. Furthermore, the dataset contains the corrected digital surface model (DSM), representing tree heights. BAMFORESTS is annotated in the COCO format and is especially suited for training deep neural networks (DNNs) to solve instance segmentation tasks. BAMFORESTS was created in the BaKIM project and is freely available under the CC BY 4.0 license.

A comprehensive exploration of approximate DNN models with a novel floating-point simulation framework

This paper introduces TorchAxf11https://github.com/knuscent/TorchAxf., a framework for fast simulation of diverse approximate deep neural network (DNN) models, including spiking neural networks (SNNs). The proposed framework utilizes various approximate adders and multipliers, supports industrial standard reduced precision floating-point formats, such as bfloat16, and accommodates user-customized precision representations. Leveraging GPU acceleration on the PyTorch framework, TorchAxf accelerates approximate DNN training and inference. In addition, it allows seamless integration of arbitrary approximate arithmetic algorithms with C/C++ behavioral models to emulate approximate DNN hardware accelerators.We utilize the proposed TorchAxf framework to assess twelve popular DNN models under approximate multiply-and-accumulate (MAC) operations. Through comprehensive experiments, we determine the suitable degree of floating-point arithmetic approximation for these DNN models without significant accuracy loss and offer the optimal reduced precision formats for each DNN model. Additionally, we demonstrate that approximate-aware re-training can rectify errors and enhance pre-trained DNN models under reduced precision formats. Furthermore, TorchAxf, operating on GPU, remarkably reduces simulation time for complex DNN models using approximate arithmetic by up to 131.38× compared to the baseline optimized CPU implementation. Finally, we compare the proposed framework with state-of-the-art frameworks to highlight its superiority.

Scalable bio-inspired training of Deep Neural Networks with FastHebb

Recent work on sample efficient training of Deep Neural Networks (DNNs) proposed a semi-supervised methodology based on biologically inspired Hebbian learning, combined with traditional backprop-based training. Promising results were achieved on various computer vision benchmarks, in scenarios of scarce labeled data availability. However, current Hebbian learning solutions can hardly address large-scale scenarios due to their demanding computational cost. In order to tackle this limitation, in this contribution, we investigate a novel solution, named FastHebb (FH), based on the reformulation of Hebbian learning rules in terms of matrix multiplications, which can be executed more efficiently on GPU. Starting from Soft-Winner-Takes-All (SWTA) and Hebbian Principal Component Analysis (HPCA) learning rules, we formulate their improved FH versions: SWTA-FH and HPCA-FH. We experimentally show that the proposed approach accelerates training speed up to 70 times, allowing us to gracefully scale Hebbian learning experiments on large datasets and network architectures such as ImageNet and VGG.

Entropy-based deep neural network training optimization for optical coherence tomography imaging

ABSTRACT This paper presents an optimization technique for the number of training epochs needed for deep learning models. The proposed method eliminates the need for separate validation data and significantly decreases training epochs. Using a four-class Optical Coherence Tomography (OCT) image dataset encompassing Choroidal Neovascularization (CNV), Diabetic Macular Edema (DME), Drusen, and Normal retina categories, we evaluated twelve architectures. These include general-purpose models (Alexnet, VGG11, VGG13, VGG16, VGG19, ResNet-18, ResNet-34, and ResNet-50) and OCT image-specific models (RetiNet, AOCT-NET, DeepOCT, and Octnet). The proposed technique reduced training epochs ranging from 4.35% to 58.27% for all architectures except Alexnet. Although the overall increase in accuracy ranges from 0.28% to 12.6%, with some architectures experiencing minor improvements, this is seen as acceptable considering the substantial reduction in training time. By achieving higher accuracy with fewer training epochs and eliminating the need for separate validation data, our methodology streamlines early stopping significantly. Statistical evaluations via Shapiro-Wilk and Kruskal-Wallis tests further affirm these results, showcasing the potential of this novel technique for efficient deep learning practices in scenarios constrained by time or computational resources.

Multiple Image Distortion DNN Modeling Individual Subject Quality Assessment

A recent research direction is focused on training Deep Neural Networks (DNNs) to replicate individual subject assessments of media quality. These DNNs are referred to as Artificial Intelligence-based Observers (AIOs). An AIO is designed to simulate, in real-time, the quality ratings of a specific individual, enabling an automatic quality assessment that accounts for subjects characteristics and preferences. Training AIOs is a promising but challenging research area due to the greater noise in individual raw opinion scores compared to the Mean Opinion Score (MOS). Effective learning from noisy labels necessitates the training of complex models on large-scale datasets. Unfortunately, this is challenging for AIOs as the media quality assessment community lacks extensive datasets that include individual opinion scores. To address the complexity of the task, we first created a dataset comprising two million samples, with synthetic labels derived from human annotation. We then trained a customized network for image quality assessment, named Multi-Distortion ResNet50 (MDResNet50), on this dataset. The weights of the MDResNet50 were subsequently utilized to initialize the learning process of each AIO, thereby avoiding the need to train a complex model from scratch on a small-scale dataset with raw individual opinion scores. Computational experiments show that our approach significantly advances the state-of-the-art in the AIO research. In particular: i) we demonstrate through a simulation the ability of AIOs to mimic two well-known behavioral characteristics of a subject, i.e.,bias and inconsistency, when scoring the media quality; ii) we train and release DNN-based AIOs that, compared to the state-of-the-art, exhibit a higher performance with a statistical significance in assessing multiple image distortions; iii) we train AIOs that more accurately mimic the sensitivity of real subjects to noise and color saturation, and also better predict the opinion score distribution compared to the state-of-the-art AIOs.

BmmW: A DNN-based joint BLE and mmWave radar system for accurate 3D localization with goal-oriented communication

Bluetooth Low Energy (BLE) has emerged as one of the reference technologies for the development of indoor localization systems, due to its increasing ubiquity, low-cost hardware, and to the introduction of direction-finding enhancements improving its ranging performance. However, the intrinsic narrowband nature of BLE makes this technology susceptible to multipath and channel interference. As a result, it is still challenging to achieve decimetre-level localization accuracy, which is necessary when developing location-based services for smart factories and workspaces. To address this challenge, we present BmmW, an indoor localization system that augments the ranging estimates obtained with BLE5.1’s constant tone extension feature with mmWave radar measurements to provide 3D localization of a mobile tag with decimetre-level accuracy. Specifically, BmmW embeds a deep neural network (DNN) that is jointly trained with both BLE and mmWave measurements, practically leveraging the strengths of both technologies. In fact, mmWave radars can locate objects and people with decimetre-level accuracy, but their effectiveness in monitoring stationary targets and multiple objects is limited, and they also suffer from a fast signal attenuation limiting the usable range to a few meters. We evaluate BmmW’s performance experimentally, and show that its joint DNN training scheme allows to track mobile tags with a mean 3D localization accuracy of 10 cm when combining angle-of-arrival BLE measurements with mmWave radar data. We further assess two variations of BmmW: BmmW-Lite and BmmW-Lite+, both tailored for single-antenna BLE devices, which eliminates the necessity for bulky and expensive multi-antenna arrays and represents a cost-effective solution that is easy to integrate into compact IoT devices. In contrast to classic BmmW (which utilizes angle-of-arrival info), BmmW-Lite uses raw in-phase/quadrature (I/Q) measurements, and achieves a mean localization accuracy of 36 cm, thus facilitating precise object tracking in indoor environments even when using budget-friendly single-antenna BLE devices. BmmW-Lite+ extends BmmW-Lite by allowing the localization task to be transferred from the edge to the cloud due to device memory and power constraints. To this end, BmmW-Lite+ employs a goal-oriented communication paradigm that compresses initial BLE features into a more compact semantic format at the edge device, which allows to minimize the amount of data that needs to be sent to the cloud. Our experimental results show that BmmW-Lite+ can compress raw BLE features by up to 12% of their initial size (hence allowing to save network bandwidth and minimize energy consumption), with negligible impact on the localization accuracy.

Learning Gaze-aware Compositional GAN from Limited Annotations

Gaze-annotated facial data is crucial for training deep neural networks (DNNs) for gaze estimation. However, obtaining these data is labor-intensive and requires specialized equipment due to the challenge of accurately annotating the gaze direction of a subject. In this work, we present a generative framework to create annotated gaze data by leveraging the benefits of labeled and unlabeled data sources. We propose a Gaze-aware Compositional GAN that learns to generate annotated facial images from a limited labeled dataset. Then we transfer this model to an unlabeled data domain to take advantage of the diversity it provides. Experiments demonstrate our approach's effectiveness in generating within-domain image augmentations in the ETH-XGaze dataset and cross-domain augmentations in the CelebAMask-HQ dataset domain for gaze estimation DNN training. We also show additional applications of our work, which include facial image editing and gaze redirection.

Cross-domain structural damage identification using transfer learning strategy

In damage identification of civil structures, training deep neural networks (DNNs) often requires a large volume of annotated training data. However, artificial damage to real-world structures is always forbidden, resulting in insufficient data for training. Transfer learning provides a solution that allows structural damage information in a data-rich domain to be transferred and shared as the prior knowledge to a data-scarce domain, thereby indirectly augmenting available training data in the latter domain. To this end, a multi-task transfer learning strategy has been adopted for the purpose of achieving cross-domain damage identification between a plane frame in the source domain and a three-dimensional frame in the target domain. The strategy can share the learning knowledge from the domain with sufficient training data to the target domain with insufficient data. Thereby, the DNN (named DNN#2) in the target domain can be trained on a small amount of training data. Owing to their ability in data feature extraction, stacked auto-encoders (SAEs) are used to construct the desirable DNN#1 and DNN#2 corresponding to different damage identification tasks, named Task#1 and Task#2, in the two domains. The two DNNs share the hidden layers in order to share damage feature information between the source and target domains. The training datasets of the two domains are first used to jointly pre-train the auto-encoders’ parameters in an unsupervised learning way. Afterwards, a supervised fine-tuning step is carried out to retraining the entire SAEs for better performance. By these means, Task#2 receives some prior knowledge from Task#1, and thus, it is accomplished on limited training data when Task#1 is synchronously achieved. The analysis results demonstrate that the proposed multi-task learning strategy requires only a single training session to simultaneously realize the cross-domain damage identification of two different steel frames.

Cost-minimization and Implicit Innovative Training for Energy Prediction using Deep Neural Network

To alleviate adverse environmental impacts, power stations and energy grids must optimize resource application for power generation. Accordingly, soothsaying guests' energy consumption has come integral to every energy operation system. exercising data from smart homes, energy operation information can train deep neural networks to anticipate unborn energy demands. As the frugality advances, both energy product and consumption have steadily increased over the times. Amidst global enterprises over energy force and environmental challenges, this study introduces a new vaticination approach using neural networks. By using statistical data from the energy assiduity, these networks directly read changes in energy product and consumption trends. Numerical findings validate the efficacity of this neural network- grounded vaticination system, emphasizing its significance in energy conservation sweats. Predicting energy consumption stands as a pivotal bid in energy conservation enterprise. Support vector retrogression, famed for its efficacity in handlingnon-linear data retrogression challenges, has surfaced as a prominent tool for soothsaying structure energy consumption. Through analysis of literal data, it's apparent that the relationship between lighting energy consumption and its impacting factors is non-linear.

Hierarchical Training of Deep Neural Networks Using Early Exiting.

Deep neural networks (DNNs) provide state-of-the-art accuracy for vision tasks, but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime, and privacy concerns. In this study, a novel hierarchical training method for DNNs is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime, and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that, due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud, and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification, respectively, when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved, while the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy DNNs on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

IoT device type identification using training deep quantum neural networks optimized with a chimp optimization algorithm for enhancing IoT security

IoT networks can be defined as groups of physically connected things and devices that can connect to the Internet and exchange data with one another. Since enabling an increasing number of internets of things devices to connect with their networks, organizations have become more vulnerable to safety issues and attacks. A major drawback of previous research is that it can find out prior seen types only, also any new device types are considered anomalous. In this manuscript, IoT device type detection utilizing Training deep quantum neural networks optimized with a Chimp optimization algorithm for enhancing IOT security (IOT-DTI-TDQNN-COA-ES) is proposed. The proposed method entails three phases namely data collection, feature extraction and detection. For Data collection phase, real network traffic dataset from different IoT device types are collected. For feature mining phase, the internet traffic features are extracted through automated building extraction (ABE) method. IoT device type identification phase, Training deep quantum neural networks (TDQNN) optimized with Chimp optimization algorithm (COA) is utilized to detect the category of IoT devices as known and unknown device. IoT network is implemented in Python. Then the simulation performance of the proposed IOT-DTI-TDQNN-COA-ES method attains higher accuracy as26.82% and 23.48% respectively, when compared with the existing methods.

VLSI architecture of stochastic genetic algorithm for real time training of deep neural network

VLSI architecture of stochastic genetic algorithm for real time training of deep neural network

Machine-Learning driven STM images prediction of doped/defective graphene: Towards optimized tools for 2D nanomaterials characterization

Scanning tunneling microscopy (STM) is a key characterization technique that allows for visualization of specific features at nanostructures, given its ability to reveal changes in local charge densities in doping or defective sites. Besides the experimental acquisition of STM data from real samples, there is the theoretical route, which allows for STM images simulation from ab-initio calculations on defined nanostructures. This work presents an alternative route for theoretical STM image prediction by means of Machine Learning (ML) methods, based on the training of deep convolutional neural networks using simulated STM data. The ML-based STM predictions for defective, B- and N-doped graphene quantum dots are compared to ab-initio STM simulations, in terms of accuracy of structural rearrangements, charge density localizations, and computational resources requirements. The results demonstrate that STM images predicted by ML-methods are accurate at doping sites, whereas they show good similarity to simulations of large structural vacancies. This work represents a novel alternative for the use of ML methods in image-based characterization routines of doped/defective graphene, and potentially, for other 2-dimensional nanomaterials.

Enhancing the automatic facies classification of Brazilian presalt acoustic image logs with SwinV2-Unet: Leveraging transfer learning and confident learning

Facies classification of image logs plays a vital role in reservoir characterization, especially in the heterogeneous and anisotropic carbonate formations of the Brazilian presalt region. Although manual classification remains the industry standard for handling the complexity and diversity of image logs, it has the notable disadvantages of being time consuming, labor intensive, subjective, and nonrepeatable. Recent advancements in machine learning offer promising solutions for automation and acceleration. However, previous attempts to train deep neural networks for facies identification have struggled to generalize to new data due to insufficient labeled data and the inherent intricacy of image logs. In addition, human errors in manual labels further hinder the performance of trained models. To overcome these challenges, we develop adopting the state-of-the-art SwinV2-Unet to provide depthwise facies classification for Brazilian presalt acoustic image logs. The training process incorporates transfer learning to mitigate overfitting and confident learning to address label errors. Through a k-fold cross-validation experiment, with each fold spanning more than 350 m, we achieve an impressive macro F1 score of 0.90 for out-of-sample predictions. This significantly surpasses the previous model modified from the widely recognized U-Net, which provides a macro F1 score of 0.68. These findings highlight the effectiveness of the used enhancements, including the adoption of an improved neural network and an enhanced training strategy. Moreover, our SwinV2-Unet enables a highly efficient and accurate facies analysis of the complex yet informative image logs, significantly advancing our understanding of hydrocarbon reservoirs, saving human effort, and improving productivity.