Network Training Research Articles

Non-intrusive load monitoring based on time-enhanced multidimensional feature visualization

In the research of non-intrusive load monitoring (NILM), the temporal characteristics of V–I trajectories are often overlooked, and using a single feature for identification may lead to insignificant differences between similar loads. Based on this, this paper proposes a non-intrusive load monitoring method based on time-enhanced multidimensional feature visualization. By adding a time axis to the V–I trajectory, it integrates the rate of change in voltage and current, power factor, and third harmonic to form a three-dimensional spatiotemporal color V–I trajectory, addressing the gap in dynamic characteristics. The ECA-ResNet34 network model is used for load identification, avoiding the problems of network degradation and training difficulties caused by the excessive depth of traditional convolutional neural networks (CNN), and achieving efficient monitoring of household loads. The method was validated on the PLAID dataset, achieving an average F1 score of 97.3%. Furthermore, utilizing transfer learning, the model trained and tested on the PLAID dataset was further tested on the WHITED dataset to increase the model’s universality and generalization ability, showing significant effects in identifying loads with similar V–I trajectories and multiple states.

Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per-second 3D imaging with deep learning

To reveal the fundamental aspects hidden behind a variety of transient events in mechanics, physics, and biology, the highly desired ability to acquire three-dimensional (3D) images with ultrafast temporal resolution has been long sought. As one of the most commonly employed 3D sensing techniques, fringe projection profilometry (FPP) reconstructs the depth of a scene from stereo images taken with sequentially structured illuminations. However, the imaging speed of current FPP methods is generally capped at several kHz, which is limited by the projector-camera hardware and the number of fringe patterns required for phase retrieval and unwrapping. Here we report a novel learning-based ultrafast 3D imaging technique, termed single-shot super-resolved FPP (SSSR-FPP), which enables ultrafast 3D imaging at 100,000 Hz. SSSR-FPP uses only one pair of low signal-to-noise ratio (SNR), low-resolution, and pixelated fringe patterns as input, while the high-resolution unwrapped phase and fringe orders can be deciphered with a specific trained deep neural network. Our approach exploits the significant speed gain achieved by reducing the imaging window of conventional high-speed cameras, while “regenerating” the lost spatial resolution through deep learning. To demonstrate the high spatio-temporal resolution of SSSR-FPP, we present 3D videography of several transient scenes, including rotating turbofan blades, exploding building blocks, and the reciprocating motion of a steam engine, etc., which were previously challenging or even impossible to capture with conventional methods. Experimental results establish SSSR-FPP as a significant step forward in the field of 3D optical sensing, offering new insights into a broad spectrum of dynamic processes across various scientific disciplines.

A method for reconstruction of structural health monitoring data using WGANGP with U-net generator

Data loss issue often occurs in structural health monitoring (SHM), which can undermine the reliability of structural condition diagnosis and prognosis. Neural networks are commonly employed for reconstruction of missing SHM data by modeling the correlation between intact and missing signals. The existing neural network methods use deeper architectures to capture complex data correlations. As network depth increases, the ability to preserve both low- and high-level features diminishes, and network training becomes challenging, which reduces data reconstruction accuracy. This study proposes a data reconstruction method that utilizes a Wasserstein generative adversarial network containing a gradient penalty term with a U-net generator. Multiple improvements are made to the generative adversarial network to enhance the reconstruction performance. First, the U-net is used as a generator, and the signal features at low and high levels are preserved based on the skip-connection technique. The U-net is pre-set with multiple layers to enhance the reconstruction accuracy. Second, a mean square error (MSE) term is added to the loss function. The MSE coefficient is proposed to balance adversarial training and reconstruction feature learning. Third, the Wasserstein distance is introduced to replace the cross-entropy loss function of traditional generative adversarial networks, avoiding gradient vanishing and exploding. The reconstruction performance of the proposed method is evaluated on a computational bridge model and Canton Tower, and the influence of reconstruction parameters on the accuracy of reconstruction results is discussed in detail. The reconstructed data by the proposed method closely matches the original data in both the time domain and frequency domain. The time–frequency characteristics of the acceleration data can be accurately reconstructed, demonstrating the effectiveness of the reconstructed signals in data analysis. By comparing with typical neural network methods, it is found that the proposed method has higher reconstruction accuracy.

Partitioned neural network training via synthetic intermediate labels

Abstract The proliferation of extensive neural network architectures, particularly deep learning models, presents a challenge in terms of resource-intensive training. GPU memory constraints have become a notable bottleneck in training such sizable models. Existing strategies, including data parallelism, model parallelism, pipeline parallelism, and fully sharded data parallelism, offer partial solutions. Model parallelism, in particular, enables the distribution of the entire model across multiple GPUs, yet the ensuing data communication between these partitions slows down training. Instead of using the entire model for training, this study advocates partitioning the model across GPUs and generating synthetic intermediate labels to train individual segments. These labels, produced through a random process, mitigate memory overhead and computational load. This approach results in a more efficient training process that minimizes data communication while maintaining model accuracy. The method is validated using 6-layer fully-connected networks, via the extended MNIST, CIFAR10 and CIFAR100 datasets. It is shown that the computational improvement to reach 90% of the cross-yield accuracy can be as high as 66%. Additionally, the improvement in training bandwidth compared to standard model parallelism is quantitatively demonstrated through an example scenario. This work contributes to mitigating the resource-intensive nature of training large neural networks, paving the way for more efficient deep learning model development.

Scalable network emulation on analog neuromorphic hardware

We present a novel software feature for the BrainScaleS-2 accelerated neuromorphic platform that facilitates the partitioned emulation of large-scale spiking neural networks. This approach is well suited for deep spiking neural networks and allows for sequential model emulation on undersized neuromorphic resources if the largest recurrent subnetwork and the required neuron fan-in fit on the substrate. We demonstrate the training of two deep spiking neural network models—using the MNIST and EuroSAT datasets—that exceed the physical size constraints of a single-chip BrainScaleS-2 system. The ability to emulate and train networks larger than the substrate provides a pathway for accurate performance evaluation in planned or scaled systems, ultimately advancing the development and understanding of large-scale models and neuromorphic computing architectures.

Automated corrosion surface quantification in steel transmission towers using UAV photogrammetry and deep convolutional neural networks

AbstractCorrosion in steel transmission towers poses a challenge to structural integrity and safety, requiring efficient detection methods. Traditional visual inspections are unsustainable due to the complexity and volume of structures. Their manual, qualitative, and subjective nature often leads to inconsistencies in maintenance planning. This study proposes a deep learning‐based approach for semantic segmentation of corroded areas on steel towers. Using the DeepLabv3+ model, the network was trained and validated on 999 field photographs. MobileNetV2, serving as the feature extractor, was chosen for its optimal balance between accuracy and computational efficiency, achieving a validation accuracy of 90.8% and a loss of 0.23. The trained network was applied to real‐world inspections using orthomosaics derived from photogrammetric reconstructions of the South‐East tower at the Torino Eremo broadcasting center. These photogrammetric products not only enabled precise segmentation of corroded areas but also provided the foundation for corrosion quantification with metrical accuracy, a critical advantage for maintenance planning. Unlike traditional image segmentation methods, which lack a spatial reference and precise scaling, the photogrammetric approach ensures that the corrosion extent and distribution are quantified in exact physical dimensions, enhancing the reliability of the analysis. The results show that deep learning‐based inspections can automate detection, providing reliable data and reducing reliance on manual inspections, enhancing efficiency, safety, and accuracy.

Influence of Trainees' Transfer Beliefs, Intentions, and Commitment on Transfer Readiness: Variable and Person‐Oriented Analyses

ABSTRACTTransfer beliefs are understudied in the training transfer field, whereas structural equation modelling (SEM) has been a widely used technique to study transfer models. New methodologies are needed to study training transfer and network analysis (NA) has emerged as a new approach that provides a visual representation of a given network. We explored the relation of transfer beliefs, intentions, commitment, and implementation intentions, and transfer using variable (SEM) and person‐oriented approaches (NA) according to groups of trainees based on their transfer readiness. The longitudinal design measured T1 before the training and T2 after the training (268 participants). T1 measured trainees' beliefs about transfer, commitment to transfer, and intention to transfer; T2 measured self‐reported transfer and implementation intention actions. The results of the NA confirmed the structure of the exploratory factor analysis. The NA model offered the visual representation of the network complimentary to the results obtained via SEM. Differentiating NA and SEM multigroup by cluster showed differences in models and architectures between clusters. We discussed the relations between beliefs and transfer, and also the implications of the combined use of the SEM and NA as novel approaches to study transfer.

Enabling high-throughput quantitative wood anatomy through a dedicated pipeline.

Throughout their lifetime, trees store valuable environmental information within their wood. Unlocking this information requires quantitative analysis, in most cases of the surface of wood. The conventional pathway for high-resolution digitization of wood surfaces and segmentation of wood features requires several manual and time consuming steps. We present a semi-automated high-throughput pipeline for sample preparation, gigapixel imaging, and analysis of the anatomy of the end-grain surfaces of discs and increment cores. The pipeline consists of a collaborative robot (Cobot) with sander for surface preparation, a custom-built open-source robot for gigapixel imaging (Gigapixel Woodbot), and a Python routine for deep-learning analysis of gigapixel images. The robotic sander allows to obtain high-quality surfaces with minimal sanding or polishing artefacts. It is designed for precise and consistent sanding and polishing of wood surfaces, revealing detailed wood anatomical structures by applying consecutively finer grits of sandpaper. Multiple samples can be processed autonomously at once. The custom-built open-source Gigapixel Woodbot is a modular imaging system that enables automated scanning of large wood surfaces. The frame of the robot is a CNC (Computer Numerical Control) machine to position a camera above the objects. Images are taken at different focuspoints, with a small overlap between consecutive imagesin the X-Y plane, and merged by mosaic stitching, into a gigapixel image. Multiple scans can be initiated through the graphical application, allowing the system to autonomously image several objects and large surfaces. Finally, a Python routine using a trained YOLOv8 deep learning network allows for fully automated analysis of the gigapixel images, here shown as a proof-of-concept for the quantification of vessels and rays on full disc surfaces and increment cores. We present fully digitized beech discs of 30-35cm diameter at a resolution of 2.25 m, for which we automatically quantified the number of vessels (up to 13 million) and rays. We showcase the same process for five 30cm length beech increment cores also digitized at a resolution of 2.25 m, and generated pith-to-bark profiles of vessel density. This pipeline allows researchers to perform high-detail analysis of anatomical features on large surfaces, test fundamental hypotheses in ecophysiology, ecology, dendroclimatology, and many more with sufficient sample replication.

Convolutional Neural Networks Assisted Peak Classification in Targeted LC-HRMS/MS for Equine Doping Control Screening Analyses.

Doping control screening analyses usually involve visual inspection of extracted ion chromatograms (EIC) by a trained analytical chemist, followed by further investigations if needed. This task is both highly repetitive and time-consuming, given the hundreds of compounds and metabolites to be screened in tens of thousands of samples per year. With the recent widespread adoption of machine learning in analytical chemistry and the training of high-performance convolutional neural networks (CNN), these operations can be automated with high accuracy and throughput. Applying this technology to doping control is challenging as the false negative rate (FNR) shall be equal to zero. In this study, we demonstrated that implementing a deep learning strategy for chromatogram classification in equine doping control can be feasible and accurate. We illustrated our findings with a CNN scoring model combined with a linear discriminant analysis (LDA) classifier trained on chromatogram images from our ultra-high-pressure liquid chromatography coupled to high-resolution tandem mass spectrometry (UHPLC-HRMS/MS)-based biotherapeutics screening method. We expect that artificial intelligence (AI) will be a valuable tool for doping control laboratories in the near future.

Constructive-synthesizing modeling of the direct current traction power supply system

A general constructive-synthesizing model of the DC traction power supply section has been developed. The model can be used to solve a number of problems related to reducing electricity consumption in both railway and urban public electric transport. The developed model is focused on determining the availability and nomenclature of traction substation equipment and rational use of recuperation energy. However, it can also be applied to solve other problems related to the designs of the traction power supply system. An example of the formed scheme of a linear power supply section with three substations is given. The presented generalized model allows to form a set of potentially possible schemes of sections of the power supply system, which can serve as the basis for further constructive modeling: traction power supply modes and the state of the power grid, equipment location options and train situation, determination of expert conclusions on the use of recuperation energy on these models, formation of a neuro-fuzzy or neural network of decision-making and, based on them control constructor. The developed modeling method is based on the capabilities of the сonstructive-synthesizing modeling in a new subject area. The terminal alphabet is semantically filled with images of electrical equipment, traction network and electricity consumers with appropriate attributes. The constructive-synthesizing modeling specification has been performed, which allows taking into account a significant number of capabilities and features of modern equipment of traction power supply systems, traction network sections and train situation. The given indi-vidual case of forming a structural diagram demonstrates the capabilities of the constructive-synthesizing modeling in relation to a number of tasks.

A protocol for trustworthy EEG decoding with neural networks

Deep learning solutions have rapidly emerged for EEG decoding, achieving state-of-the-art performance on a variety of decoding tasks. Despite their high performance, existing solutions do not fully address the challenge posed by the introduction of many hyperparameters, defining data pre-processing, network architecture, network training, and data augmentation. Automatic hyperparameter search is rarely performed and limited to network-related hyperparameters. Moreover, pipelines are highly sensitive to performance fluctuations due to random initialization, hindering their reliability. Here, we design a comprehensive protocol for EEG decoding that explores the hyperparameters characterizing the entire pipeline and that includes multi-seed initialization for providing robust performance estimates. Our protocol is validated on 9 datasets about motor imagery, P300, SSVEP, including 204 participants and 26 recording sessions, and on different deep learning models. We accompany our protocol with extensive experiments on the main aspects influencing it, such as the number of participants used for hyperparameter search, the split into sequential simpler searches (multi-step search), the use of informed vs. non-informed search algorithms, and the number of random seeds for obtaining stable performance. The best protocol included 2-step hyperparameter search via an informed search algorithm, with the final training and evaluation performed using 10 random initializations. The optimal trade-off between performance and computational time was achieved by using a subset of 3-5 participants for hyperparameter search. Our protocol consistently outperformed baseline state-of-the-art pipelines, widely across datasets and models, and could represent a standard approach for neuroscientists for decoding EEG in a trustworthy and reliable way.

Hessian-based mixed-precision quantization with transition aware training for neural networks.

Model quantization is widely used to realize the promise of ubiquitous embedded deep network inference. While mixed-precision quantization has shown promising performance, existing approaches often rely on time-consuming search process to determine the optimal bit configuration. To address this, we introduce Hessian-based Mixed-Precision Quantization Aware Training(HMQAT) to decrease the search overhead of bit configuration. By using the sensitivity metric of joint average Hessian trace and parameter size, HMQAT effectively guides the search process. We solve the optimization problem about bit configuration automatically using a Pareto frontier method. The lowest search overhead can be achieved through our scheme. Additionally, our approach incorporates quantization transition aware fine-tuning of scale factor. This strategy consistently ensures optimal inference performance along the accuracy-size Pareto frontier across multiple models. We extensively evaluated our method on ImageNet and CIFAR10. In particular, we show that compared to the baseline, HMQAT achieves a 10.34× reduction in model size while retaining 99.81% of the Top-1 accuracy on ResNet18 for ImageNet. Moreover, HMQAT surpassed the state-of-the-art mixed-precision quantization methods in compressing neural networks with reduced search cost and achieving a satisfying trade-off between size and accuracy. This study paves the way of deploying neural networks on lightweight devices.

Ultra-high-sensitive SPR device composed of CaF2 prism, Ag, graphene, and sensing medium for refractive index detection-artificial neural network training

Ultra-high-sensitive SPR device composed of CaF2 prism, Ag, graphene, and sensing medium for refractive index detection-artificial neural network training

GBM-Reservoir: Brain tumor (Glioblastoma Multiforme) MRI dataset collection with ground truth segmentation masks.

In this article, we present a brain tumor database collection comprising 23,049 samples, with each sample including four different types of MRI brain scans: FLAIR, T1, T1ce, and T2. Additionally, one or two segmentation masks (ground truth) are provided for each sample. The first mask is the raw output from the registration process and is provided for all samples, while the second mask, provided particularly for synthetic samples, is a post-processed version of the first, designed to simplify interpretation and optimize it for network training. These samples have been acquired via registration process of 438 samples available at the moment of registration from the original dataset provided by the BraTS 2022 Challenge. Registering each pair of existing brain scans results in two additional scans that retain a similar brain shape while featuring varying tumor locations. Consequently, by registering all possible pairs, a dataset originally consisting of n samples can be expanded to n2 samples. The original dataset was collected from different institutions under standard clinical conditions, but with different equipment and imaging protocols. As a result, the image quality is heterogeneous, reflecting the diversity of clinical practices across institutions. This dataset can be utilized for various tasks, such as developing fully automated segmentation algorithms for new, unseen brain tumor cases, particularly through deep learning-based approaches, since ground truth is provided for each sample.

Loss function inversion for improved crack segmentation in steel bridges using a CNN framework

Automating bridge visual inspection using deep learning algorithms for crack detection in images is a prominent way to make these inspections more effective. This paper addresses several challenges associated with crack detection: (1) data imbalance, caused by a small crack area as compared to the background, and (2) a high false positive rate, due to a large amount of crack-like features in the background. First, a new benchmark dataset is presented, containing images of cracks in steel bridges along with pixel-wise annotations. Secondly, the importance of incorporating background patches is examined to assess their impact on network performance when applied to high resolution images of cracks in steel bridges. Finally, a loss function is introduced that enables the use of a relatively large number of background patches in neural network training. The proposed approaches yield a significant reduction in false positive rates, thereby improving the overall performance of crack segmentation.

Highlights of the Marie Curie Training Network ONCORNET2.0

Highlights of the Marie Curie Training Network ONCORNET2.0 The ONCORNET2.0 (ONCOgenic GPCR Network of Excellence and Training) consortium (2020–2024) aims to consolidate an international training network of early stage researchers (ESRs) focused on drug discovery for oncogenic GPCRs. The project methods span a wide range of techniques and disciplines aimed at furthering our understanding of two receptors heavily involved in oncogenic processes.

From Sancus to Sancusq: staleness and quantization-aware full-graph decentralized training in graph neural networks

Graph neural networks (GNNs) have emerged due to their success at modeling graph data. Yet, it is challenging for GNNs to efficiently scale to large graphs. Thus, distributed GNNs come into play. To avoid communication caused by expensive data movement between workers, we propose Sancus and its advanced version Sancus, the staleness and quantization-aware communication-avoiding decentralized GNN system. By introducing a set of novel bounded embedding staleness metrics and adaptively skipping broadcasts, Sancus abstracts decentralized GNN processing as sequential matrix multiplication and uses historical embeddings via cache. To further mitigate the communication volume, Sancus conducts quantization-aware communication on embeddings to reduce the size of broadcast messages. Theoretically, we show bounded approximation errors of embeddings and gradients with a known fastest convergence guarantee. Empirically, we evaluate Sancus and Sancus with common GNN models via different system setups on large-scale benchmark datasets. Compared to SOTA works, Sancus can avoid up to 86% communication with 3.0× faster throughput on average without accuracy loss.

Posterior sampling for random noise attenuation via score-based generative models

Random noise attenuation is an ill-posed inverse problem with multiple solutions, especially in complicated field noise situations. We develop a method to sample stochastic solutions from the posterior distribution of seismic data for a given noisy input. Posterior sampling can be performed by Langevin dynamics with a conditional score function, which can be described as a trained score network (in score-based generative models) plus an analytical expression related to the noise distribution. Each solution from the posterior distribution is reasonable and of high quality. The numerous solutions we obtain may contain underground structural information of interest. We also achieve interactive posterior sampling by automatically estimating a noise level or manually setting it according to the noise level map of the field noise. Experiments on synthetic and field data verify the superiority of our posterior sampling approach.

Diagnosis of the Skin Cancer by Vision Transformers

Skin cancer, one of the most frequent cancers, requires early identification for efficient treatment and better survival. Early diagnosis relies on precise and quick skin lesion categorization into benign and malignant categories. This work uses Vision Transformers (ViTs) to classify skin cancer photos by modeling long-range relationships and capturing complicated visual patterns. ViTs, originally created for natural language processing, have showed great potential in picture classification tasks because to their self-attention processes, outperforming CNNs. A public collection of 270 skin lesion images—240 malignant and 30 benign—was used in this study. Preprocessing included scaling and normalizing the dataset to 384x384x3 and splitting it into 80% training and 20% testing sets. Transfer learning optimised a pre-trained ViT model for this job. To improve accuracy and avoid overfitting, hyperparameters were carefully selected for network training. Parallel computing accelerated training to 30 minutes and 20 seconds. Vision Transformers classify medical images well, according to the study. The ViT model outperformed numerous other methods with 98.15% accuracy on the test set. A confusion matrix analysis showed great sensitivity in detecting malignant lesions and low misclassification of benign patients. These findings show that ViTs can capture detailed medical picture aspects, making them useful for dermatological diagnoses. This study shows that Vision Transformers can improve diagnosis accuracy and lays the groundwork for their use in other medical imaging fields. In the battle against skin cancer and other illnesses that need early diagnosis, ViTs' scalability, efficiency, and precision are important. Future research might integrate ViTs with other deep learning architectures to improve robustness and flexibility. This study adds to the data supporting sophisticated AI in medical diagnostics and lays the groundwork for automated, reliable, and efficient healthcare solutions.

Real-Time High Dynamic Equalization Industrial Imaging Enhancement Based on Fully Convolutional Network

Severe reflections on the surfaces of smooth objects can result in low dynamic range and uneven illumination in images, which negatively impacts downstream tasks such as defect detection and QR code recognition on images of smooth workpieces. Consequently, this paper proposes a novel approach to real-time high dynamic equalization imaging based on a fully convolutional network, termed Multi-exposure Image Fusion with Multi-dimensional Attention Mechanism and Training Storage Units (MEF-AT). Specifically, this paper innovatively proposes using training storage units, which utilize intermediate results during network training as auxiliary images, to remove uneven illumination and enhance image dynamic range effectively. Furthermore, by integrating a multi-dimensional attention mechanism into the backbone network, the model can more efficiently extract and utilize critical image information. Additionally, this paper introduces a Deep Guided Filter (DGF) with learnable parameters, which upsample the weight maps generated by the network, thus better adapting to complex industrial scenarios and producing higher quality fused images. An image evaluation metric assessing the lighting uniformity is introduced to thoroughly evaluate the proposed method’s performance. Given the lack of an MEF dataset for smooth workpieces, this paper collects a new dataset for multi-exposure fusion tasks on metallic workpieces. Our method takes less than 4 ms to run four 2K images on a GPU 3090. Both qualitative and quantitative experimental results demonstrate our method’s superior comprehensive performance in proprietary industrial and public datasets.