Training Deep Networks Research Articles

Daily power generation forecasting for a grid-connected solar power plant using transfer learning technique

Deep learning is efficiently used for photovoltaic power generation forecasting to handle the intermittent nature of solar energy. However, big data are required for training deep networks which are not available for newly installed plants. Therefore, in this study, a novel strategy is proposed to train a deep learning model using a transfer learning technique to cop up with the unavailability of enough training datasets. A new 400 kWp solar power plant installed in the Himalayan region is considered as a case study to evaluate the proposed model. The proposed approach utilizes solar radiation data to train a deep neural network and then fine-tune the model using the power generation data from the plant. The network architecture is optimized using grey wolf optimizer to find the best suitable model for the data. The evaluation results show that the same model can achieve higher performance in generation forecasting with percentage error improved by 2% and R-value increased by 7.7% after applying transfer learning. Moreover, SHapley Additive exPlanation and Partial Dependence Plots are used to interpret the model behavior and showed that the model is mostly dependent on the previous generation values (up to 4 days) followed by the temperature and solar radiation.

An active object detection model with multi-step prediction based on deep q-learning network and innovative training algorithm

An active object detection model with multi-step prediction based on deep q-learning network and innovative training algorithm

Deep learning in the advanced core sample porosity determination with XCT image

X-ray computed tomography (XCT) for core sample porosity analysis has advanced significantly, integrating machine learning with traditional image processing and introducing methods such as Weighted Assignment by CT number calculation. However, several areas still require enhancement, particularly in image preprocessing and model optimization for specific deep learning approaches. Moreover, the application of the Weighted Assignment method to water-sensitive mudstone core samples in this study revealed the inconsistency with the porosity estimation, primarily due to the overemphasis on fracture effects. To address these limitations, this study proposed the ResNet algorithm, a deep learning model developed for XCT-based porosity estimation. ResNet's architecture, incorporating residual learning with shortcut connections, enables effective training of deep networks and enhances feature extraction. The ResNet-based model was trained on XCT images with known porosity values measured by Multi-Sensor Core Logging (MSCL), effectively predicting volume fraction porosity for materials with similar lithological characteristics. Analysis of testing losses, comparison of predicted versus actual porosity values and RRMSE, demonstrates the model's superior performance than traditional methods. This novel application of ResNet to porosity estimation highlights the potential of deep learning to significantly enhance accuracy by 97.82% compared with the weighted assignment method with RRMSE.

A Survey of Label-Efficient Deep Learning for 3D Point Clouds.

In the past decade, deep neural networks have achieved significant progress in point cloud learning. However, collecting large-scale precisely-annotated point clouds is extremely laborious and expensive, which hinders the scalability of existing point cloud datasets and poses a bottleneck for efficient exploration of point cloud data in various tasks and applications. Label-efficient learning offers a promising solution by enabling effective deep network training with much-reduced annotation efforts. This paper presents the first comprehensive survey of label-efficient learning of point clouds. We address three critical questions in this emerging research field: i) the importance and urgency of label-efficient learning in point cloud processing, ii) the subfields it encompasses, and iii) the progress achieved in this area. To this end, we propose a taxonomy that organizes label-efficient learning methods based on the data prerequisites provided by different types of labels. We categorize four typical label-efficient learning approaches that significantly reduce point cloud annotation efforts: data augmentation, domain transfer learning, weakly-supervised learning, and pretrained foundation models. For each approach, we outline the problem setup and provide an extensive literature review that showcases relevant progress and challenges. Finally, we share our views on the current research challenges and potential future directions.

In-context learning enables multimodal large language models to classify cancer pathology images

Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce.

Deep supervision network with contrastive learning for zero-shot sketch-based image retrieval

Zero-shot sketch-based image retrieval (ZS-SBIR) is an extremely challenging cross-modal retrieval task. In ZS-SBIR, hand-drawn sketches are used as queries to retrieve corresponding natural images in zero-shot scenarios. Existing methods utilize diverse loss functions to guide deep neural networks (DNNs) to align feature representations of both sketches and images. In general, these methods supervise only the last layer of DNNs and then update each layer of DNNs using back-propagate technology. However, this strategy cannot effectively optimize the intermediate layers of DNNs, potentially hindering retrieval performance. To address this issue, we propose a deep supervision network with contrastive learning (DSNCL) approach for ZS-SBIR. Specifically, we employ a novel deep supervision network training method that attaches multiple projection heads to the intermediate layers of DNNs. These projection heads map multi-level features to a normalized embedding space and are trained by contrastive learning. The proposed method instructs the intermediate layers of DNNs to learn the invariance of various data augmentation, thereby aligning the feature representations of both sketches and images. This significantly narrows its domain gap and semantic gap. Besides, we use contrastive learning to directly optimize the intermediate layers of DNNs, which effectively reduces the optimization difficulty of their intermediate layers. Furthermore, we investigate the cross-batch metric (CBM) learning mechanism, which stores samples of different batches for metric learning by constructing a semantic queue, to further improve the performance in ZS-SBIR applications. Comprehensive experimental results on the Sketchy and TU-Berlin datasets validate the superiority of our DSNCL method over existing state-of-the-art methods.

Building Extraction from Unmanned Aerial Vehicle (UAV) Data in a Landslide-Affected Scattered Mountainous Area Based on Res-Unet

Building extraction in landslide-affected scattered mountainous areas is essential for sustainable development, as it improves disaster risk management, fosters sustainable land use, safeguards the environment, and bolsters socio-economic advancement; however, this process entails considerable challenges. This study proposes a Res-Unet-based model to extract landslide-affected buildings from unmanned aerial vehicle (UAV) data in scattered mountain regions, leveraging the feature extraction capabilities of ResNet and the precise localization abilities of U-Net. A landslide-affected, scattered mountainous region within the Three Gorges Reservoir area was selected as a case study to validate the model’s performance. Experimental results indicate that Res-Unet displays high accuracy and robustness in building recognition, attaining accuracy (ACC), intersection-over-union (IOU), and F1-score values of 0.9849, 0.9785, and 0.9892, respectively. This enhancement can be attributed to the combined model, which amalgamates the skip connections, the symmetric architecture of U-Net, and the residual blocks of ResNet. This integration preserves low-level detail during recovery at higher levels, facilitating the extraction of multi-scale features while also mitigating the vanishing gradient problem prevalent in deep network training through the residual block structure, thus enabling the extraction of more complex features. The proposed Res-Unet approach shows significant potential for the accurate recognition and extraction of buildings in complex terrains through the efficient processing of remote sensing images.

Mixed Mutual Transfer for Long-Tailed Image Classification.

Real-world datasets often follow a long-tailed distribution, where a few majority (head) classes contain a large number of samples, while many minority (tail) classes contain significantly fewer samples. This imbalance creates an information disparity between head and tail classes, which can negatively impact the performance of deep networks. Some transfer knowledge techniques attempt to mitigate this gap by generating additional minority samples, either through oversampling the tail classes or transferring knowledge from the head classes to the tail classes. However, these methods often restrict the diversity of the generated minority samples, primarily focusing on transferring information only to the tail classes. This paper introduces a simple yet effective method for long-tailed classification, called mixed mutual transfer (MMT), which facilitates the mutual transfer of knowledge between head and tail classes by blending samples. The core idea of our method is to create new samples by blending head and tail samples. Head samples are selected using a uniform sampler that retains the long-tailed distribution, while tail samples are selected using a differential sampler that reverses the long-tailed distribution to alleviate imbalance. Our approach aims to diversify both tail and head classes. During the training phase, we utilize the generated samples to update the original dataset for training deep networks. Mixed mutual transfer simultaneously enhances the performance of both head and tail classes. Experimental results on various class-imbalanced datasets show that the proposed method significantly outperforms existing methods, demonstrating its effectiveness in improving the performance of long-tailed deep networks.

Pre-training transformer with dual-branch context content module for table detection in document images

BackgroundDocument images such as statistical reports and scientific journals are widely used in information technology. Accurate detection of table areas in document images is an essential prerequisite for tasks such as information extraction. However, because of the diversity in the shapes and sizes of tables, existing table detection methods adapted from general object detection algorithms, have not yet achieved satisfactory results. Incorrect detection results might lead to the loss of critical information. MethodsTherefore, we propose a novel end-to-end trainable deep network combined with a self-supervised pretraining transformer for feature extraction to minimize incorrect detections. To better deal with table areas of different shapes and sizes, we added a dual-branch context content attention module (DCCAM) to high-dimensional features to extract context content information, thereby enhancing the network's ability to learn shape features. For feature fusion at different scales, we replaced the original 3×3 convolution with a multilayer residual module, which contains enhanced gradient flow information to improve the feature representation and extraction capability. ResultsWe evaluated our method on public document datasets and compared it with previous methods, which achieved state-of-the-art results in terms of evaluation metrics such as recall and F1-score. https://github.com/YongZ-Lee/TD-DCCAM

Using acoustic emission technique for structural health monitoring of laminate composite: A novel CNN-LSTM framework

This research has developed a real-time end-to-end deep learning model for structural health monitoring (SHM) method for composite impact damage diagnosis based on Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM). The acoustic emission (AE) signals collected under low-velocity impacts by means of piezoelectric sensors on composite materials are used for training deep learning networks. Based on the impact load curves, specimens are categorized into minor failure, intermediate failure, and severe failure. The convolved signals are segmented and reconstructed at a given length for the following LSTM module. The average accuracies for basic CNN, CNN– Recurrent Neural Network (RNN), CNN-LSTM, and CNN– Gated Recurrent Unit (GRU) are respectively 88.7 %, 92.6 %, 98 %, and 95.4 %. A sensitivity analysis on sub-signal length was conducted on the CNN-LSTM model, revealing that the model achieved its best performance when the sub-signal length was set at 16. The model attained prediction accuracies of 97.4 %, 100 %, and 100 %, respectively, for minor failure, intermediate failure, and severe failure cases.

An Effective Res-Progressive Growing Generative Adversarial Network-Based Cross-Platform Super-Resolution Reconstruction Method for Drone and Satellite Images

In recent years, accurate field monitoring has been a research hotspot in the domains of aerial remote sensing and satellite remote sensing. In view of this, this study proposes an innovative cross-platform super-resolution reconstruction method for remote sensing images for the first time, aiming to make medium-resolution satellites capable of field-level detection through a super-resolution reconstruction technique. The progressive growing generative adversarial network (PGGAN) model, which has excellent high-resolution generation and style transfer capabilities, is combined with a deep residual network, forming the Res-PGGAN model for cross-platform super-resolution reconstruction. The Res-PGGAN architecture is similar to that of the PGGAN, but includes a deep residual module. The proposed Res-PGGAN model has two main benefits. First, the residual module facilitates the training of deep networks, as well as the extraction of deep features. Second, the PGGAN structure performs well in cross-platform sensor style transfer, allowing for cross-platform high-magnification super-resolution tasks to be performed well. A large pre-training dataset and real data are used to train the Res-PGGAN to improve the resolution of Sentinel-2’s 10 m resolution satellite images to 0.625 m. Three evaluation metrics, including the structural similarity index metric (SSIM), the peak signal-to-noise ratio (PSNR), and the universal quality index (UQI), are used to evaluate the high-magnification images obtained by the proposed method. The images generated by the proposed method are also compared with those obtained by the traditional bicubic method and two deep learning super-resolution reconstruction methods: the enhanced super-resolution generative adversarial network (ESRGAN) and the PGGAN. The results indicate that the proposed method outperforms all the comparison methods and demonstrates an acceptable performance regarding all three metrics (SSIM/PSNR/UQI: 0.9726/44.7971/0.0417), proving the feasibility of cross-platform super-resolution image recovery.

PolyMeme: Fine-Grained Internet Meme Sensing.

Internet memes are a special type of digital content that is shared through social media. They have recently emerged as a popular new format of media communication. They are often multimodal, combining text with images and aim to express humor, irony, sarcasm, or sometimes convey hatred and misinformation. Automatically detecting memes is important since it enables tracking of social and cultural trends and issues related to the spread of harmful content. While memes can take various forms and belong to different categories, such as image macros, memes with labeled objects, screenshots, memes with text out of the image, and funny images, existing datasets do not account for the diversity of meme formats, styles and content. To bridge this gap, we present the PolyMeme dataset, which comprises approximately 27 K memes from four categories. This was collected from Reddit and a part of it was manually labelled into these categories. Using the manual labels, deep learning networks were trained to classify the unlabelled images with an estimated error rate of 7.35%. The introduced meme dataset in combination with existing datasets of regular images were used to train deep learning networks (ResNet, ViT) on meme detection, exhibiting very high accuracy levels (98% on the test set). In addition, no significant gains were identified from the use of regular images containing text.

ICS-ResNet: A Lightweight Network for Maize Leaf Disease Classification

The accurate identification of corn leaf diseases is crucial for preventing disease spread and improving corn yield. Plant leaf images are often affected by factors such as complex backgrounds, climate, light, and sample data imbalance. To address these issues, we propose a lightweight convolutional neural network, ICS-ResNet, based on ResNet50. This network incorporates improved spatial and channel attention modules as well as a deep separable residual structure to enhance recognition accuracy. (1) The residual connections in the ResNet network prevent gradient loss during deep network training. (2) The improved channel attention (ICA) and spatial attention (ISA) modules fully utilize semantic information from different feature layers to accurately localize key features of the network. (3) To reduce the number of parameters and lower computational costs, we replace traditional convolutional computation with a depth-separable residual structure. (4) We also employ cosine annealing to dynamically adjust the learning rate, enhancing the network’s training stability, improving model convergence, and preventing local optima. Experiments on the corn dataset in Plant Village compare the proposed ICS-ResNet with eight popular networks: CSPNet, InceptionNet_v3, EfficientNet, ShuffleNet, MobileNet, ResNet50, ResNet101 and ResNet152. The results show that the ICS-ResNet achieves an accuracy of 98.87%, which is 5.03%, 3.18%, 1.13%, 1.81%, 1.13%, 0.68%, 0.44% and 0.60% higher than the other networks, respectively. Furthermore, the number of parameters and computations are reduced by 69.21% and 54.88%, respectively, compared to the original ResNet50 network, significantly improving the efficiency of corn leaf disease classification. The study provides strong technical support for sustainable agriculture and the promotion of agricultural science and technology innovation.

Data-driven hybrid modelling of waves at mid-frequencies range: Application to forward and inverse Helmholtz problems

In this paper, we introduce a novel hybrid approach that leverages both data and numerical simulations to address the challenges of solving forward and inverse wave problems, particularly in the mid-frequency range. Our method is tailored for efficiency and accuracy, considering the computationally intensive nature of these problems, which arise from the need for refined mesh grids and a high number of degrees of freedom. Our approach unfolds in multiple stages, each targeting a specific frequency range. Initially, we decompose the wave field into a grid of finely resolved points, designed to capture the intricate details at various wavenumbers within the frequency range of interest. Importantly, the distribution of these grid points remains consistent across different wavenumbers. Subsequently, we generate a substantial dataset comprising 1,000 maps covering the entire frequency range. Creating such a dataset, especially at higher frequencies, can pose a significant computational challenge. To tackle this, we employ a highly efficient enrichment-based finite element method, ensuring the dataset’s creation is computationally manageable. The dataset which encompasses 1000 different values of the wavenumbers with their corresponding wave simulation will be the basis to train a fully connected neural network. In the forward problem the neural network is trained such that a wave pattern is predicted for each value of the wavenumber. To address the inverse problem while upholding stability, we introduce latent variables to reduce the number of physical parameters. Our trained deep network undergoes rigorous testing for both forward and inverse problems, enabling a direct comparison between predicted solutions and their original counterparts. Once the network is trained, it becomes a powerful tool for accurately solving wave problems in a fraction of the CPU time required by alternative methods. Notably, our approach is supervised, as it relies on a dataset generated through the enriched finite element method, and hyperparameter tuning is carried out for both the forward and inverse networks.

On the Training and Generalization of Deep Operator Networks

On the Training and Generalization of Deep Operator Networks

Unsupervised speckle noise reduction technique for clinical ultrasound imaging.

Deep learning-based image enhancement has significant potential in the field of ultrasound image processing, as it can accurately model complicated nonlinear artifacts and noise, such as ultrasonic speckle patterns. However, training deep learning networks to acquire reference images that are clean and free of noise presents significant challenges. This study introduces an unsupervised deep learning framework, termed speckle-to-speckle (S2S), designed for speckle and noise suppression. This framework can complete its training without the need for clean (speckle-free) reference images. The proposed network leverages statistical reasoning for the mutual training of two in vivo images, each with distinct speckle patterns and noise. It then infers speckle- and noise-free images without needing clean reference images. This approach significantly reduces the time, cost, and effort experts need to invest in annotating reference images manually. The experimental results demonstrated that the proposed approach outperformed existing techniques in terms of the signal-to-noise ratio, contrast-to-noise ratio, structural similarity index, edge preservation index, and processing time (up to 86 times faster). It also performed excellently on images obtained from ultrasound scanners other than the ones used in this work. S2S demonstrates the potential of employing an unsupervised learning-based technique in medical imaging applications, where acquiring a ground truth reference is challenging.

Robust implicit regularization via weight normalization

Abstract Overparameterized models may have many interpolating solutions; implicit regularization refers to the hidden preference of a particular optimization method towards a certain interpolating solution among the many. A by now established line of work has shown that (stochastic) gradient descent tends to have an implicit bias towards low rank and/or sparse solutions when used to train deep linear networks, explaining to some extent why overparameterized neural network models trained by gradient descent tend to have good generalization performance in practice. However, existing theory for square-loss objectives often requires very small initialization of the trainable weights, which is at odds with the larger scale at which weights are initialized in practice for faster convergence and better generalization performance. In this paper, we aim to close this gap by incorporating and analysing gradient flow (continuous-time version of gradient descent) with weight normalization, where the weight vector is reparameterized in terms of polar coordinates, and gradient flow is applied to the polar coordinates. By analysing key invariants of the gradient flow and using Lojasiewicz’s Theorem, we show that weight normalization also has an implicit bias towards sparse solutions in the diagonal linear model, but that in contrast to plain gradient flow, weight normalization enables a robust bias that persists even if the weights are initialized at practically large scale. Experiments suggest that the gains in both convergence speed and robustness of the implicit bias are improved dramatically using weight normalization in overparameterized diagonal linear network models.

Empowering deep neural quantum states through efficient optimization

Computing the ground state of interacting quantum matter is a long-standing challenge, especially for complex two-dimensional systems. Recent developments have highlighted the potential of neural quantum states to solve the quantum many-body problem by encoding the many-body wavefunction into artificial neural networks. However, this method has faced the critical limitation that existing optimization algorithms are not suitable for training modern large-scale deep network architectures. Here, we introduce a minimum-step stochastic-reconfiguration optimization algorithm, which allows us to train deep neural quantum states with up to 106 parameters. We demonstrate our method for paradigmatic frustrated spin-1/2 models on square and triangular lattices, for which our trained deep networks approach machine precision and yield improved variational energies compared to existing results. Equipped with our optimization algorithm, we find numerical evidence for gapless quantum-spin-liquid phases in the considered models, an open question to date. We present a method that captures the emergent complexity in quantum many-body problems through the expressive power of large-scale artificial neural networks.

Enhancing visual autonomous navigation in row-based crops with effective synthetic data generation

IntroductionService robotics is recently enhancing precision agriculture enabling many automated processes based on efficient autonomous navigation solutions. However, data generation and in-field validation campaigns hinder the progress of large-scale autonomous platforms. Simulated environments and deep visual perception are spreading as successful tools to speed up the development of robust navigation with low-cost RGB-D cameras.Materials and methodsIn this context, the contribution of this work resides in a complete framework to fully exploit synthetic data for a robust visual control of mobile robots. A wide realistic multi-crops dataset is accurately generated to train deep semantic segmentation networks and enabling robust performance in challenging real-world conditions. An automatic parametric approach enables an easy customization of virtual field geometry and features for a fast reliable evaluation of navigation algorithms.Results and conclusionThe high quality of the generated synthetic dataset is demonstrated by an extensive experimentation with real crops images and benchmarking the resulting robot navigation both in virtual and real fields with relevant metrics.

Speech Emotion Recognition Using Dual-Stream Representation and Cross-Attention Fusion

Speech emotion recognition (SER) aims to recognize human emotions through in-depth analysis of audio signals. However, it remains challenging to encode emotional cues and to fuse the encoded cues effectively. In this study, dual-stream representation is developed, and both full training and fine-tuning of different deep networks are employed for encoding emotion patterns. Specifically, a cross-attention fusion (CAF) module is designed to integrate the dual-stream output for emotion recognition. Using different dual-stream encoders (fully training a text processing network and fine-tuning a pre-trained large language network), the CAF module is compared to other three fusion modules on three databases. The SER performance is quantified with weighted accuracy (WA), unweighted accuracy (UA), and F1-score (F1S). The experimental results suggest that the CAF outperforms the other three modules and leads to promising performance on the databases (EmoDB: WA, 97.20%; UA, 97.21%; F1S, 0.8804; IEMOCAP: WA, 69.65%; UA, 70.88%; F1S, 0.7084; RAVDESS: WA, 81.86%; UA, 82.75.21%; F1S, 0.8284). It is also found that fine-tuning a pre-trained large language network achieves superior representation than fully training a text processing network. In a future study, improved SER performance could be achieved through the development of a multi-stream representation of emotional cues and the incorporation of a multi-branch fusion mechanism for emotion recognition.