Weight Initialization Scheme Research Articles

Fine-tuning GAN Models with Unpaired Aerial Images for RGB to NDVI Translation in Vegetation Index Estimation

Calculating the Normalized Difference Vegetation Index (NDVI) requires expensive multispectral cameras, posing challenges such as high costs and the need for technical expertise. This research aims to develop a method to transform RGB images obtained from Unmanned Aerial Vehicle (UAV) into NDVI images. Our proposed method leverages the CycleGAN model, an unsupervised image-to-image translation framework, to learn the intricate mapping between RGB values and NDVI. The model was trained on unpaired datasets of RGB and NDVI images, sourced from paddy fields located in Gresik and Yogyakarta, Indonesia. This process successfully encapsulated the complex correlation between these two modalities. Various training strategies were systematically investigated, including weight initialization schemes, fine-tuning procedures, and learning rate policies, to optimize the model's performance. The fine-tuned CycleGAN demonstrated superior performance in creating synthetic NDVI images from unpaired dataset, surpassing other methods in terms of fidelity, quality, and structural coherence. The results were impressive, with a Normalized Root Mean Square Error (NRMSE) of 0.327, a Peak Signal-to-Noise Ratio (PSNR) of 16.330, an Oriented FAST and Rotated BRIEF (ORB) score of 0.859, and a Structural Similarity Index (SSIM) of 0.757. The best performing CycleGAN model was then deployed on a low-spec microcomputer device, specifically the Raspberry Pi 4B with an average computation time of 21.0077 seconds. Raspberry Pi 4B was chosen for its lightweight, compact dimensions, and compatibility with efficient battery power connections, making it suitable for drone deployment.

Enhancing COVID-19 disease severity classification through advanced transfer learning techniques and optimal weight initialization schemes

ObjectiveMedical imaging plays a central role in medicine today by supporting diagnosis and treatment. For small medical image datasets, training from scratch is a time-consuming process and transfer learning emerges as a solution. In these cases, ImageNet weights are usually used as initial weights, after which fine-tuning is performed. We propose a methodology for COVID-19 severity classification (mild, moderate, severe, critical) in chest X-ray images based on transfer learning using DenseNet121 architecture as a base model. MethodsThe novelty of the approach mainly lies in the investigation of three different weight initialization schemes (i) ImageNet (ii) CheXNeXt (iii) DeepCOVID-XR, which are similar in a graded manner in terms of image nature, with ImageNet being the least similar, CheXNeXt similar as they were trained on different lung diseases not including COVID-19 and DeepCOVID-XR the most similar as they were obtained by training specifically on COVID-19 X-ray images. ResultsThe results show that the worst results are achieved using ImageNet as initial weights (average AUC = 0.700), followed by the better results with DeepCOVID-XR (average AUC = 0.774) while CheXNeXt performed significantly better (average AUC = 0.917). The results of weaker performing classification models were improved when the severe and critical classes were merged, to account for the similarity between these classes in the dataset (average AUC for initialization schemes were ImageNet AUC = 0.867, CheXNeXt AUC = 0.900 and DeepCOVID-XR AUC = 0.794). ConclusionThis leads to a conclusion that, in the medical domain, where image datasets are usually small and highly imbalanced, if initial weights are chosen to be in nature similar to the new dataset, achieved results are better. However, there is no need to start from weights obtained during training on the same disease, as this may cause overfitting. There are significant discrepancies between ImageNet and medical imaging datasets and the results from this paper could help in guiding future implementations of transfer learning in medical applications.

Learnable weight initialization for volumetric medical image segmentation

Hybrid volumetric medical image segmentation models, combining the advantages of local convolution and global attention, have recently received considerable attention. While mainly focusing on architectural modifications, most existing hybrid approaches still use conventional data-independent weight initialization schemes which restrict their performance due to ignoring the inherent volumetric nature of the medical data. To address this issue, we propose a learnable weight initialization approach that utilizes the available medical training data to effectively learn the contextual and structural cues via the proposed self-supervised objectives. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach, leading to state-of-the-art segmentation performance. Our proposed data-dependent initialization approach performs favorably as compared to the Swin-UNETR model pretrained using large-scale datasets on multi-organ segmentation task. Our source code and models are available at: https://github.com/ShahinaKK/LWI-VMS.

Cooperative Spectrum Sensing Using Extreme Learning Machines for Cognitive Radio Networks

In this article, a technique for cooperative spectrum sensing (CSS) using the Extreme Learning Machine (ELM) is proposed. ELMs are feedforward neural networks where the hidden layer parameters are not tuned, and only output weights are optimized. The simulations were done for both the fading and non-fading environments. Different combination of activation function and weight initialization scheme is deployed for calculating channel occupancy detection. The obtained results are compared with popular fusion schemes and well-known Machine Learning (ML) techniques. Further, the proposed algorithm's performance is compared with some of the recent CSS techniques in the literature. The primary metrics for comparison were the receiver operating characteristic (ROC) curve, the area under the curve (AUC), detection performance, and energy consumption. We obtained the ELM model's computational performance during the training phase and calculated the channel detection probability. These results demonstrate the potential superiority of ELM over established methods. In particular, the work presented here shows a better trade-off between training time and detection performance.

Numerical Evaluation on Parametric Choices Influencing Segmentation Results in Radiology Images—A Multi-Dataset Study

Medical image segmentation has gained greater attention over the past decade, especially in the field of image-guided surgery. Here, robust, accurate and fast segmentation tools are important for planning and navigation. In this work, we explore the Convolutional Neural Network (CNN) based approaches for multi-dataset segmentation from CT examinations. We hypothesize that selection of certain parameters in the network architecture design critically influence the segmentation results. We have employed two different CNN architectures, 3D-UNet and VGG-16, given that both networks are well accepted in the medical domain for segmentation tasks. In order to understand the efficiency of different parameter choices, we have adopted two different approaches. The first one combines different weight initialization schemes with different activation functions, whereas the second approach combines different weight initialization methods with a set of loss functions and optimizers. For evaluation, the 3D-UNet was trained with the Medical Segmentation Decathlon dataset and VGG-16 using LiTS data. The quality assessment done using eight quantitative metrics enhances the probability of using our proposed strategies for enhancing the segmentation results. Following a systematic approach in the evaluation of the results, we propose a few strategies that can be adopted for obtaining good segmentation results. Both of the architectures used in this work were selected on the basis of general acceptance in segmentation tasks for medical images based on their promising results compared to other state-of-the art networks. The highest Dice score obtained in 3D-UNet for the liver, pancreas and cardiac data was 0.897, 0.691 and 0.892. In the case of VGG-16, it was solely developed to work with liver data and delivered a Dice score of 0.921. From all the experiments conducted, we observed that two of the combinations with Xavier weight initialization (also known as Glorot), Adam optimiser, Cross Entropy loss (GloCEAdam) and LeCun weight initialization, cross entropy loss and Adam optimiser LecCEAdam worked best for most of the metrics in a 3D-UNet setting, while Xavier together with cross entropy loss and Tanh activation function (GloCEtanh) worked best for the VGG-16 network. Here, the parameter combinations are proposed on the basis of their contributions in obtaining optimal outcomes in segmentation evaluations. Moreover, we discuss that the preliminary evaluation results show that these parameters could later on be used for gaining more insights into model convergence and optimal solutions.The results from the quality assessment metrics and the statistical analysis validate our conclusions and we propose that the presented work can be used as a guide in choosing parameters for the best possible segmentation results for future works.

Emergence of Network Motifs in Deep Neural Networks.

Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article, we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Simulations show that the final network topology is shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.

Pixel-Wise PolSAR Image Classification via a Novel Complex-Valued Deep Fully Convolutional Network

Although complex-valued (CV) neural networks have shown better classification results compared to their real-valued (RV) counterparts for polarimetric synthetic aperture radar (PolSAR) classification, the extension of pixel-level RV networks to the complex domain has not yet thoroughly examined. This paper presents a novel complex-valued deep fully convolutional neural network (CV-FCN) designed for PolSAR image classification. Specifically, CV-FCN uses PolSAR CV data that includes the phase information and uses the deep FCN architecture that performs pixel-level labeling. The CV-FCN architecture is trained in an end-to-end scheme to extract discriminative polarimetric features, and then the entire PolSAR image is classified by the trained CV-FCN. Technically, for the particularity of PolSAR data, a dedicated complex-valued weight initialization scheme is proposed to initialize CV-FCN. It considers the distribution of polarization data to conduct CV-FCN training from scratch in an efficient and fast manner. CV-FCN employs a complex downsampling-then-upsampling scheme to extract dense features. To enrich discriminative information, multi-level CV features that retain more polarization information are extracted via the complex downsampling scheme. Then, a complex upsampling scheme is proposed to predict dense CV labeling. It employs the complex max-unpooling layers to greatly capture more spatial information for better robustness to speckle noise. The complex max-unpooling layers upsample the real and the imaginary parts of complex feature maps based on the max locations maps retained from the complex downsampling scheme. In addition, to achieve faster convergence and obtain more precise classification results, a novel average cross-entropy loss function is derived for CV-FCN optimization. Experiments on real PolSAR datasets demonstrate that CV-FCN achieves better classification performance than other state-of-art methods.

CGAN-MBL for Reliability Assessment With Imbalanced Transmission Gear Data

In term of data-driven techniques for reliability assessment of transmission gears, it is necessary to collect sufficient data of each class in practice. However, the fact that highest and lowest reliability events rarely occur would result in a class imbalance problem. To resolve such an issue, we propose conditional generative adversarial network-mean-covariance balancing labeling (CGAN-MBL) to improve reliability assessment of transmission gears with insufficient and imbalanced data. First, we establish a CGAN-based model to generate credible instances with a similar distribution of original data. In the model, it is proven that by introducing a network weight initialization scheme, the ability to escape from local optimum is enhanced and global search ability of CGAN is strengthened; minibatch discriminator is designed to decrease mode collapse by distinguishing a batch of original and generated data iteratively. Second, in order to overcome the drawback of CGAN with unlabeled generated data, MBL is introduced to tag the generated data through exploring the nearest distance between generated instances and original class centers. Finally, synthetic data combined generated data with original ones train classifiers. Simulations show the effectiveness of the proposed approach for imbalance learning in real-world transmission gear data.

Breast cancer histology images classification: Training from scratch or transfer learning?

Abstract We demonstrated the ability of transfer learning in comparison with the fully-trained network on the histopathological imaging modality by considering three pre-trained networks: VGG16, VGG19, and ResNet50 and analyzed their behavior for magnification independent breast cancer classification. Concurrently, we examined the effect of training–testing data size on the performance of considered networks. A fine-tuned pre-trained VGG16 with logistic regression classifier yielded the best performance with 92.60% accuracy, 95.65% area under ROC curve (AUC), and 95.95% accuracy precision score (APS) for 90%–10% training–testing data splitting. Layer-wise fine-tuning and different weight initialization schemes can be a future aspect of this study.

Binary/ternary extreme learning machines

In this paper, a new hidden layer construction method for Extreme Learning Machines (ELMs) is investigated, aimed at generating a diverse set of weights. The paper proposes two new ELM variants: Binary ELM, with a weight initialization scheme based on { 0 , 1 } -weights; and Ternary ELM, with a weight initialization scheme based on { - 1 , 0 , 1 } -weights. The motivation behind this approach is that these features will be from very different subspaces and therefore each neuron extracts more diverse information from the inputs than neurons with completely random features traditionally used in ELM. Therefore, ideally it should lead to better ELMs. Experiments show that indeed ELMs with ternary weights generally achieve lower test error. Furthermore, the experiments show that the Binary and Ternary ELMs are more robust to irrelevant and noisy variables and are in fact performing implicit variable selection. Finally, since only the weight generation scheme is adapted, the computational time of the ELM is unaffected, and the improved accuracy, added robustness and the implicit variable selection of Binary ELM and Ternary ELM come for free.

An independent component analysis based weight initialization method for multilayer perceptrons

A novel algorithm for determining the optimal initial weights of multilayer perceptrons based on an independent component analysis is developed. The algorithm is able to initialize the hidden layer weights that extract the salient feature components from the input data. The initial output layer weights are evaluated in such a way that the output neurons are kept inside the active region. Real-world benchmark problems were used for validating the proposed algorithm. The simulation results indicate that the proposed algorithm gets a substantial reduction in the required training time as compared with the other well-known weight initialization schemes. The proposed weight initialization method is capable of speeding up the learning process of multilayer perceptrons effectively.

A stochastically motivated random initialization of pattern classifying MLPs

In this contribution, a new stochastically motivated random weight initialization scheme for pattern classifying Multi-Layer Perceptrons (MLPs) is presented. Its first aim is to ensure that all training examples and all nodes have an equal opportunity to contribute to the improvement of the network during the Error Back-Propagation (EBP) training. In addition, it pursues input scale invariance: if the network inputs were substituted by rescaled inputs, the initialization procedure should provide an equally well performing network. Finally, the new algorithm can initialize MLPs comprising both concentric (e.g., Gaussian) and squashing (e.g., sigmoidal) nodes. Experiments demonstrate that networks initialized using the proposed method train better than networks initialized using a standard random initialization scheme.