Random Initial Weights Research Articles

Multi-Step Ahead Water Level Forecasting Using Deep Neural Networks

Stream gauge height (water level) is a significant indicator for forecasting future floods. Flooding occurs when the water level exceeds the flood stage. Predicting imminent floods can save lives, protect infrastructure, and improve road traffic management and transportation. Deep neural networks have been increasingly used in this domain due to their predictive capabilities in capturing complex features and interdependencies. This study employs four distinct models—Multi-Layer Perceptron (MLP), Long Short-Term Memory (LSTM), transformer, and LSTNet—with MLP serving as the baseline model to forecast water levels. The models are trained using data from 20 distinct river gages across the state of Missouri to ensure consistent performance. Random search optimization is employed for hyperparameter tuning. The prediction intervals are set at 4, 6, 8, and 10 (each interval equivalent to 30 min) to ensure that performance results are robust and not due to random weight initialization or suboptimal hyperparameters and are consistent throughout different prediction intervals. The findings of this study indicate that the LSTNet model leads to a better performance than the other models, with a median RMSE of 0.00724, 0.00959, 0.01204, and 0.01230 for the 4, 6, 8, and 10 intervals, respectively. As climate change leads to localized storms driven by atmospheric shifts, water level fluctuations are becoming increasingly extreme, further exacerbating data drift in real-world datasets. The LSTNet model demonstrates superior performance in terms of RMSE, MAE, and the correlation coefficient across all prediction intervals when forecasting water levels under data drift conditions.

OffRoadSynth Open Dataset for Semantic Segmentation using Synthetic-Data-Based Weight Initialization for Autonomous UGV in Off-Road Environments

This article concerns the issue of image semantic segmentation for the machine vision system of an autonomous Unmanned Ground Vehicle (UGV) moving in an off-road environment. Determining the meaning (semantics) of the areas visible in the recorded image provides a complete understanding of the scene surrounding the autonomous vehicle. It is crucial for the correct determination of a passable route. Nowadays, semantic segmentation is generally solved using convolutional neural networks (CNN), which can take an image as input and output the segmented image. However, proper training of the neural network requires the use of large amounts of data, which becomes problematic in the situation of low availability of large, dedicated image data sets that consider various off-road situations - driving on various types of roads, surrounded by diverse vegetation and in various weather and light conditions. This study introduces a synthetic image dataset called “OffRoadSynth” to address the training data scarcity for off-road scenarios. It has been shown that pre-training the neural network on this synthetic dataset improves image segmentation accuracy compared to other methods, such as random network weight initialization or using larger, generic datasets. Results suggest that using a smaller but domain-dedicated set of synthetic images to initialize network weights for training on the target real-world dataset may be an effective approach to improving semantic segmentation results of images, including those from off-road environments.

Velocity Estimations in Blood Microflows via Machine Learning Symmetries

Improving velocity forecasts of blood microflows could be useful in biomedical applications. We focus on estimating the velocity of the blood in capillaries. Modeling blood microflow in capillaries is a complex process. In this paper, we use artificial intelligence techniques for this modeling: more precisely, artificial neural networks (ANNs). The selected model is able to accurately forecast the velocity, with an R2 of 0.8992 comparing the forecast with the actual velocity. A key part of ANN model creation is selecting the appropriate parameters for the ANN, such as the number of neurons, the number of layers and the type of training algorithm used. A grid approach with 327,600 simulations was used. It is shown that there are substantial, statistically significant differences when different types of ANN structures are used. It is also shown that the proposed model is robust regarding the initial random initialization of weights in the ANN. Additionally, the sensitivity of the selected models to additional noise was also tested.

Developing a multivariate time series forecasting framework based on stacked autoencoders and multi-phase feature

Time series forecasting across different domains has received massive attention as it eases intelligent decision-making activities. Recurrent neural networks and various deep learning algorithms have been applied to modeling and forecasting multivariate time series data. Due to intricate non-linear patterns and significant variations in the randomness of characteristics across various categories of real-world time series data, achieving effectiveness and robustness simultaneously poses a considerable challenge for specific deep-learning models. We have proposed a novel prediction framework with a multi-phase feature selection technique, a long short-term memory-based autoencoder, and a temporal convolution-based autoencoder to fill this gap. The multi-phase feature selection is applied to retrieve the optimal feature selection and optimal lag window length for different features. Moreover, the customized stacked autoencoder strategy is employed in the model. The first autoencoder is used to resolve the random weight initialization problem. Additionally, the second autoencoder models the temporal relation between non-linear correlated features with convolution networks and recurrent neural networks.Finally, the model's ability to generalize, predict accurately, and perform effectively is validated through experimentation with three distinct real-world time series datasets. In this study, we conducted experiments on three real-world datasets: Energy Appliances, Beijing PM2.5 Concentration, and Solar Radiation. The Energy Appliances dataset consists of 29 attributes with a training size of 15,464 instances and a testing size of 4239 instances. For the Beijing PM2.5 Concentration dataset, there are 18 attributes, with 34,952 instances in the training set and 8760 instances in the testing set. The Solar Radiation dataset comprises 11 attributes, with 22,857 instances in the training set and 9797 instances in the testing set. The experimental setup involved evaluating the performance of forecasting models using two distinct error measures: root mean square error and mean absolute error. To ensure robust evaluation, the errors were calculated at the identical scale of the data. The results of the experiments demonstrate the superiority of the proposed model compared to existing models, as evidenced by significant advantages in various metrics such as mean squared error and mean absolute error. For PM2.5 air quality data, the proposed model's mean absolute error is 7.51 over 12.45, about ∼40% improvement. Similarly, the mean square error for the dataset is improved from 23.75 to 11.62, which is ∼51%of improvement. For the solar radiation dataset, the proposed model resulted in ∼34.7% improvement in means squared error and ∼75% in mean absolute error. The recommended framework demonstrates outstanding capabilities in generalization and outperforms datasets spanning multiple indigenous domains.

TCN-GAWO: Genetic Algorithm Enhanced Weight Optimization for Temporal Convolutional Network

Abstract This article proposes a genetic algorithm (GA)-enhanced weight optimization method for temporal convolutional network (TCN-GAWO). TCN-GAWO combines the evolutionary process of the genetic algorithm with the gradient-based training and can achieve higher predication/fitting accuracy than traditional temporal convolutional network (TCN). Performances of TCN-GAWO are also more stable. In TCN-GAWO, multiple TCNs are generated with random initial weights first, then these TCNs are trained individually for given epochs, next the selection-crossover-mutation procedure is applied among TCNs to get the evolved offspring. Gradient-based training and selection-crossover-mutation are taken in turns until convergence. The TCN with the optimal performance is then selected. Performances of TCN-GAWO are thoroughly evaluated using realistic engineering data, including C-MAPSS dataset provided by NASA and jet engine lubrication oil dataset provided by airlines. Experimental results show that TCN-GAWO outperforms existing methods for both datasets, demonstrating the effectiveness and the wide range applicability of the proposed method in solving time series problems.

Improving extreme learning machine model using deep learning feature extraction and grey wolf optimizer: Application to image classification

The Extreme Learning Machine (ELM) is a highly efficient model for real-time network retraining due to its fast learning speed, unlike traditional machine learning methods. However, the performance of ELM can be negatively impacted by the random initialization of weights and biases. Moreover, poor input feature quality can further degrade performance, particularly with complex visual data. To overcome these issues, this paper proposes optimizing the input features as well as the initial weights and biases. We combine both Convolutional Neural Network (CNN) and Convolutional AutoEncoder (CAE) extracted features to optimize the quality of the input features. And we use our hybrid Grey Wolf Optimizer-Multi-Verse Optimizer (GWO-MVO) metaheuristic for initializing weights and biases by applying four fitness functions based on: the norm of the output weights, the error rate on the training set, and the error rate on the validation set. Our method is evaluated on image classification tasks using two benchmark datasets: CIFAR-10 and CIFAR-100. Since image quality may vary in real-world applications, we trained and tested our models on the dataset’s original and noisy versions. The results demonstrate that our method provides a robust and efficient alternative for image classification tasks, offering improved accuracy and reduced overfitting.

A simplified decision feedback Chebyshev function link neural network with intelligent initialization for underwater acoustic channel equalization

IntroductionIn shallow-water environments, the reliability of underwater communication links is often compromised by significant multipath effects. Some equalization techniques such as decision feedback equalizer, and deep neural network equalizer suffer from slow convergence and high computational complexity.MethodsTo address this challenge, this paper proposes a simplified decision feedback Chebyshev function link neural network equalizer (SDF-CFLNNE). The structure of the SDF-CFLNNE employs Chebyshev polynomial function expansion modules to directly and non-linearly transform the input signals into the output layer, without the inclusion of hidden layers. Additionally, it feeds the decision signal back to the input layer rather than the function expansion module, which significantly reduces computational complexity. Considering that, in the training phase of neural networks, the random initialization of weights and biases can substantially impact the training process and the ultimate performance of the network, this paper proposes a chaotic sparrow search algorithm combining the osprey optimization algorithm and Cauchy mutation (OCCSSA) to optimize the initial weights and thresholds of the proposed equalizer. The OCCSSA utilizes the Piecewise chaotic population initialization and combines the exploration strategy of the ospreywith the Cauchy mutation strategy to enhance both global and local search capabilities. RseultsSimulations were conducted using underwater multipath signals generated by the Bellhop Acoustic Toolbox. The results demonstrate that the performance of the SDFCFLNNE initialized by OCCSSA surpasses that of CFLNN-based and traditional nonlinear equalizers, with a notable improvement of 2-6 dB in terms of signal-to-noise ratio at a bit error rate (BER) of 10−4 and a reduced mean square error (MSE). Furthermore, the effectiveness of the proposed equalizer was validated using the lake experimental data, demonstrating lower BER and MSE with improved stability. DiscussionThis underscores thepromise of employing the SDFCFLNNE initialized by OCCSSA as a promising solution to enhance the robustness of underwater communication in challenging environments.

Analysis and Implementation of CNN in Real- time Classification and Translation of Kanji Characters

This research explores the training outcomes of the Convolutional Neural Network (CNN) algorithm applied to Kanji character recognition. It employs a CNN architecture with 10 layers for recognizing digital image of Kanji characters from N5 to N1 levels. The training of the CNN model reveals varying accuracies, influenced by factors such as architecture, training data size, and data quality. The lowest accuracy, occurring at the beginning of training, highlights challenges like poor random weight initialization and suboptimal architecture. Conversely, the highest accuracy demonstrates the optimal predictive ability of the model after multiple training iterations. The iterative training process refines the model's accuracy over time, with initial challenges paving the way for a better understanding of the data for future improvements. In the subsequent analysis and system development phase, two algorithms are compared to assess the effectiveness of the CNN algorithm in Kanji character recognition. Testing is conducted at various complexity levels to systematically evaluate accuracy. This testing involves complexity levels at each Kanji character level to assess system accuracy. The developed model shows potential for real-time classification of Kanji characters, with a focus on error rate and accuracy during training and testing. The built model can essentially be used to evaluate the ability to recognize Kanji characters effectively. The model's performance can be assessed based on the error rate and accuracy achieved during the training and testing processes.

Mechanical Performance Forecast for BP Neural Network Materials Optimized by Genetic Algorithm

The mechanical properties of steel materials play a crucial role in their design, selection, and application. In order to better predict the mechanical properties through chemical composition and process parameters, this paper uses genetic algorithm to optimize the BP neural network to establish a mechanical property prediction model for steel materials. The model can predict three mechanical properties, including yield strength, tensile strength, and elongation, through chemical composition and process parameters. After optimization by genetic algorithm, the problems of insufficient convergence effect, random initialization of weights and thresholds in the BP neural network model are improved, and the prediction error is significantly reduced. The experimental results show that the GA-BP algorithm model has excellent performance in predicting the mechanical properties of steel materials.

Training and meta-training an ensemble of binary neural networks with quantum computing

The design of NNs requires an expensive search in the space of the neural network architecture. This search performs heuristics called Neural Architecture Search (NAS), but an efficient algorithmic NAS remains an open problem. The development of quantum computing allows the use of quantum features, without a classical counterpart, as a proposal to solve some problems more efficiently. NAS through quantum algorithms is one such proposal, although none of the solutions presented so far is suitable for Noisy Intermediate Scale Quantum (NISQ) devices due to the complexity of the algorithms. This work reduces the use of quantum resources, which increases the possibility of running NAS on a NISQ device. The proposed method does not rely on random weights initialization and allows training a set of architectures simultaneously. The trained quantum model corresponds to an ensemble of classical ANNs, represented by a quantum superposition of an exponential number of ANNs.

A Deep Stacked Ensemble Model for Microarray Data Classification with Boosted Meta Classifier

Classification of microarray data is one of the major research interests in the biomedical field. It allows physicians for early detection of cancer through analysis of the Deoxyribonucleic acid (DNA). Classification of these sensitive data is still challenging due to the small sample and more feature size. In this paper, the authors have used an ensemble model for classifying two types of leukemia as Acute lymphocytic leukemia (ALL) and Acute myelocytic leukemia (AML). The work is carried out with other types of genetic data such as Leukemia, lung tumor, liver cancer, and liver Cirrhosis. The biomedical data are imbalanced. The ensemble classifier is based on a stacked approach where deep neural network (DNN) classifiers are used as the base classifier. The structure of each DNN is chosen as the homogenous type for the same training process for all classifiers. Because of the adaptive nature and random weight initialization, it provides different results for each classifier. The outputs of the base classifiers are again fed to a gradient boosting ensemble model termed a meta classifier. The meta classifier provides the final classification output. For comparison purposes, two types of meta-classifiers such as support vector machine (SVM) and ensemble gradient boosting are used in the proposed work. The performance of the model is verified well and the results are provided in the result section. From the experimental result, it is observed that the classification accuracy is 96% with SVM and 98% with boosted meta classifier for leukemia data, whereas 99.04% for lung tumor and 99.03% for liver Cirrhosis.

How neural networks learn to classify chaotic time series.

We tackle the outstanding issue of analyzing the inner workings of neural networks trained to classify regular-vs-chaotic time series. This setting, well-studied in dynamical systems, enables thorough formal analyses. We focus specifically on a family of networks dubbed large Kernel convolutional neural networks (LKCNNs), recently introduced by Boullé et al. [403, 132261 (2021)]. These non-recursive networks have been shown to outperform other established architectures (e.g., residual networks, shallow neural networks, and fully convolutional networks) at this classification task. Furthermore, they outperform "manual" classification approaches based on direct reconstruction of the Lyapunov exponent. We find that LKCNNs use qualitative properties of the input sequence. We show that LKCNN models trained from random weight initialization, end in two most common performance groups: one with relatively low performance (0.72 average classification accuracy) and one with high classification performance (0.94 average classification accuracy). Notably, the models in the low performance class display periodic activations that are qualitatively similar to those exhibited by LKCNNs with random weights. This could give very general criteria for identifying, a priori, trained weights that yield poor accuracy.

A transfer learning-based machine learning approach to predict mechanical properties of different material types fabricated by selective laser melting process

The necessity for a massive dataset has limited the desirability of the machine learning approaches for industrial applications, especially in the metal additive manufacturing processes, where, collecting a large dataset is expensive and virtually ineffective. Concerning this restriction, an effective machine learning technique should be developed to bridge the gap between the academia and the industry. Hence, in this research, a transfer learning-based artificial neural network (TL-ANN) model was developed to predict the mechanical properties of different metallic specimens fabricated by selective laser melting (SLM) process. This model was integrated with a Bayesian hyperparameters optimization algorithm to select the optimum training parameters of the model. The proposed model consists of a target network and a source network. The source network was trained based on the mechanical properties that were obtained experimentally for various materials, including pure and alloyed copper, steel, titanium, nickel, etc. The overall regression correlation coefficient ( R) of the TL-ANN model was about 0.99, with the mean square error of testing, validation, and training of datasets of about 2.031, 1.423, and 1.068, respectively, representing the successful execution of the source network in prediction of the mechanical properties of the SLMed parts. Using the achieved knowledge of the source network, the target network was trained to predict the mechanical properties of the target material (here SLMed pure and alloyed aluminum specimens). The obtained results revealed that with the help of the transfer learning, the hybrid neural network could predict the mechanical properties of SLM-fabricated aluminum parts with a high accuracy level, even with the small number of training dataset of the target material. To demonstrate the influence of transfer learning in the accuracy of the model, a separate network was developed from scratch, i.e. with random initial weights of the neurons. The R-values of the test dataset of the individual model for the output parameters of ultimate tensile strength, relative density, and yield strength of the fabricated aluminum samples were 0.787, 0.742, and 0.817, respectively, as compared with that of TL-ANN model of 0.966, 0.903, and 0.971, respectively, representing an average of 21% enhancement in the accuracy of the predictivity of the model by application of transfer learning algorithm.

Efficient real-time detection of electrical equipment images using a lightweight detector model

Infrared technology holds significant importance in the detection of electrical equipment, as it has the capability to swiftly and securely identify electrical apparatus. To simplify the implementation of proficient detection frameworks for electrical equipment within constrained settings (like embedded apparatus), this study presents an enhanced, lightweight model of the single-shot multibox detector (SSD). This model specifically addresses the detection of multiple equipment objects within infrared imagery. The model realized the lightweight of the model by using the network structure characteristics of squeezenet to modify the backbone network of SSD, and compensated for the impact of the lightweight model on the detection accuracy by adding multiple convolutional layers and connecting branches to enhance the propagation ability and extraction ability of features. To ensure a comprehensive evaluation of the model’s detection capabilities, all the models discussed in this study employed the technique of random weight initialization. This approach was utilized to validate the optimal structure of the model and its performance. The experimentation was conducted on both the PASCAL VOC 2007 benchmark dataset and an infrared image dataset encompassing five distinct categories of electrical equipment found within substations. The experimental outcomes indicate that this model offers an efficient approach for achieving lightweight, real-time detection of electrical apparatus.

Empowering Foot Health: Harnessing the Adaptive Weighted Sub-Gradient Convolutional Neural Network for Diabetic Foot Ulcer Classification.

In recent times, DFU (diabetic foot ulcer) has become a universal health problem that affects many diabetes patients severely. DFU requires immediate proper treatment to avert amputation. Clinical examination of DFU is a tedious process and complex in nature. Concurrently, DL (deep learning) methodologies can show prominent outcomes in the classification of DFU because of their efficient learning capacity. Though traditional systems have tried using DL-based models to procure better performance, there is room for enhancement in accuracy. Therefore, the present study uses the AWSg-CNN (Adaptive Weighted Sub-gradient Convolutional Neural Network) method to classify DFU. A DFUC dataset is considered, and several processes are involved in the present study. Initially, the proposed method starts with pre-processing, excluding inconsistent and missing data, to enhance dataset quality and accuracy. Further, for classification, the proposed method utilizes the process of RIW (random initialization of weights) and log softmax with the ASGO (Adaptive Sub-gradient Optimizer) for effective performance. In this process, RIW efficiently learns the shift of feature space between the convolutional layers. To evade the underflow of gradients, the log softmax function is used. When logging softmax with the ASGO is used for the activation function, the gradient steps are controlled. An adaptive modification of the proximal function simplifies the learning rate significantly, and optimal proximal functions are produced. Due to such merits, the proposed method can perform better classification. The predicted results are displayed on the webpage through the HTML, CSS, and Flask frameworks. The effectiveness of the proposed system is evaluated with accuracy, recall, F1-score, and precision to confirm its effectual performance.

Classification of Pulmonary Nodules in 2-[18F]FDG PET/CT Images with a 3D Convolutional Neural Network

Purpose2-[18F]FDG PET/CT plays an important role in the management of pulmonary nodules. Convolutional neural networks (CNNs) automatically learn features from images and have the potential to improve the discrimination between malignant and benign pulmonary nodules. The purpose of this study was to develop and validate a CNN model for classification of pulmonary nodules from 2-[18F]FDG PET images.MethodsOne hundred thirteen participants were retrospectively selected. One nodule per participant. The 2-[18F]FDG PET images were preprocessed and annotated with the reference standard. The deep learning experiment entailed random data splitting in five sets. A test set was held out for evaluation of the final model. Four-fold cross-validation was performed from the remaining sets for training and evaluating a set of candidate models and for selecting the final model. Models of three types of 3D CNNs architectures were trained from random weight initialization (Stacked 3D CNN, VGG-like and Inception-v2-like models) both in original and augmented datasets. Transfer learning, from ImageNet with ResNet-50, was also used.ResultsThe final model (Stacked 3D CNN model) obtained an area under the ROC curve of 0.8385 (95% CI: 0.6455–1.0000) in the test set. The model had a sensibility of 80.00%, a specificity of 69.23% and an accuracy of 73.91%, in the test set, for an optimised decision threshold that assigns a higher cost to false negatives.ConclusionA 3D CNN model was effective at distinguishing benign from malignant pulmonary nodules in 2-[18F]FDG PET images.

Autoencoder and restricted Boltzmann machine for transfer learning in functional magnetic resonance imaging task classification

Deep neural networks (DNNs) have been adopted widely as classifiers for functional magnetic resonance imaging (fMRI) data, advancing beyond traditional machine learning models. Consequently, transfer learning of the pre-trained DNN becomes crucial to enhance DNN classification performance, specifically by alleviating an overfitting issue that occurs when a substantial number of DNN parameters are fitted to a relatively small number of fMRI samples. In this study, we first systematically compared the two most popularly used, unsupervised pretraining models for resting-state fMRI (rfMRI) volume data to pre-train the DNNs, namely autoencoder (AE) and restricted Boltzmann machine (RBM). The group in-brain mask used when training AE and RBM displayed a sizable overlap ratio with Yeo's seven functional brain networks (FNs). The parcellated FNs obtained from the RBM were fine-grained compared to those from the AE. The pre-trained AE and RBM served as the weight parameters of the first of the two hidden DNN layers, and the DNN fulfilled the task classifier role for fMRI (tfMRI) data in the Human Connectome Project (HCP). We tested two transfer learning schemes: (1) fixing and (2) fine-tuning the DNN's pre-trained AE or RBM weights. The DNN with transfer learning was compared to a baseline DNN, trained using random initial weights. Overall, DNN classification performance from the transfer learning proved superior when the pre-trained RBM weights were fixed and when the pre-trained AE weights were fine-tuned (average error rates: 14.8% for fixed RBM, 15.1% fine-tuned AE, and 15.5% for the baseline model) compared to the alternative scenarios of DNN transfer learning schemes. Moreover, the optimal transfer learning scheme between the fixed RBM and fine-tuned AE varied according to seven task conditions in the HCP. Nonetheless, the computational load reduced substantially for the fixed-weight-based transfer learning compared to the fine-tuning-based transfer learning (e.g., the number of weight parameters for the fixed-weight-based DNN model reduced to 1.9% compared with a baseline/fine-tuned DNN model). Our findings suggest that weight initialization at the DNN's first layer using RBM-based pre-trained weights provides the most promising approach when the whole-brain fMRI volume supports associated task classification. We believe that our proposed scheme could be applied to a variety of task conditions to improve their classification performance and to utilize computational resources efficiently using our AE/RBM-based pre-trained weights compared to random initial weights for DNN training.

SCL-ST: Supervised Contrastive Learning With Semantic Transformations for Multiple Lead ECG Arrhythmia Classification.

The automatic classification of electrocardiogram (ECG) signals has played an important role in cardiovascular diseases diagnosis and prediction. With recent advancements in deep neural networks (DNNs), particularly Convolutional Neural Networks (CNNs), learning deep features automatically from the original data is becoming an effective and widespread approach in a variety of intelligent tasks including biomedical and health informatics. However, most of the existing approaches are trained on either 1D CNNs or 2D CNNs, and they suffer from the limitations of random phenomena (i.e. random initial weights). Furthermore, the ability to train such DNNs in a supervised manner in healthcare is often limited due to the scarcity of labeled training data. To address the problems of weight initialization and limited annotated data, in this work, we leverage recent self-supervised learning technique, namely, contrastive learning, and present supervised contrastive learning (sCL). Different from existing self-supervised contrastive learning approaches, which often generate false negatives because of random selection of negative anchors, our contrastive learning makes use of labeled data to pull the same class closer together and push different classes far apart to avoid potential false negatives. Furthermore, unlike other kinds of signals (e.g. speech, image, video), ECG signal is sensitive to changes, and inappropriate transformation could directly affect diagnosis results. To deal with this issue, we present two semantic transformations, i.e. semantic split-join and semantic weighted peaks noise smoothing. The proposed deep neural network sCL-ST with supervised contrastive learning and semantic transformations is trained as an end-to-end framework for the multi-label classification of 12-lead ECGs. Our sCL-ST network contains two sub-networks i.e. pre-text task and down-stream task. Our experimental results have been evaluated on 12-lead PhysioNet 2020 dataset and shown that our proposed network outperforms the state-of-the-art existing approaches.

ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification

Background and ObjectiveHistopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers. MethodsHere we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model. ResultsThe main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision. ConclusionsChoosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.

Structure and performance of fully connected neural networks: Emerging complex network properties

Understanding the behavior of Artificial Neural Networks is one of the main topics in the field recently, as black-box approaches have become usual since the widespread of deep learning. Such high-dimensional models may manifest instabilities and weird properties that resemble complex systems. Therefore, we propose Complex Network (CN) techniques to analyze the structure and performance of fully connected neural networks. For that, we build a dataset with 4 thousand models (varying the initialization seed) and their respective CN properties. This is the first work to explore the CN properties of an ample number of fully connected networks accounting for the variance caused by random weight initialization. The networks are trained in a supervised classification setup considering four vision benchmarks and then approached as a weighted and undirected graph of neurons and synapses (learned weights). Results show that neuronal centrality is highly correlated to network classification performance. We also propose the concept of Bag-Of-Neurons (BoN), a CN-based approach for finding topological signatures linking similar neurons. Results suggest that six neuronal types emerge in such networks, independently of the target domain, and are distributed differently according to classification accuracy. We also tackle specific CN properties related to performance, such as higher subgraph centrality on lower-performing models. Our findings suggest that CN properties play a critical role in the performance of fully connected neural networks, with topological patterns emerging independently on a wide range of models.