Proposed Training Scheme Research Articles

Transformative Deep Neural Network Approaches in Kidney Ultrasound Segmentation: Empirical Validation with an Annotated Dataset.

Kidney ultrasound (US) images are primarily employed for diagnosing different renal diseases. Among them, one is renal localization and detection, which can be carried out by segmenting the kidney US images. However, kidney segmentation from US images is challenging due to low contrast, speckle noise, fluid, variations in kidney shape, and modality artifacts. Moreover, well-annotated US datasets for renal segmentation and detection are scarce. This study aims to build a novel, well-annotated dataset containing 44,880 US images. In addition, we propose a novel training scheme that utilizes the encoder and decoder parts of a state-of-the-art segmentation algorithm. In the pre-processing step, pixel intensity normalization improves contrast and facilitates model convergence. The modified encoder-decoder architecture improves pyramid-shaped hole pooling, cascaded multiple-hole convolutions, and batch normalization. The pre-processing step gradually reconstructs spatial information, including the capture of complete object boundaries, and the post-processing module with a concave curvature reduces the false positive rate of the results. We present benchmark findings to validate the quality of the proposed training scheme and dataset. We applied six evaluation metrics and several baseline segmentation approaches to our novel kidney US dataset. Among the evaluated models, DeepLabv3+ performed well and achieved the highest dice, Hausdorff distance 95, accuracy, specificity, average symmetric surface distance, and recall scores of 89.76%, 9.91, 98.14%, 98.83%, 3.03, and 90.68%, respectively. The proposed training strategy aids state-of-the-art segmentation models, resulting in better-segmented predictions. Furthermore, the large, well-annotated kidney US public dataset will serve as a valuable baseline source for future medical image analysis research.

Uncertainty Injection: A Deep Learning Method for Robust Optimization

This paper proposes a paradigm of uncertainty injection for training deep learning model to solve robust optimization problems. The majority of existing studies on deep learning focus on the model learning capability, while assuming the quality and accuracy of the inputs data can be guaranteed. However, in realistic applications of deep learning for solving optimization problems, the accuracy of inputs, which are the problem parameters in this case, plays a large role. This is because, in many situations, it is often costly or sometime impossible to obtain the problem parameters accurately, and correspondingly, it is highly desirable to develop learning algorithms that can account for the uncertainties in the input and produce solutions that are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">robust</i> against these uncertainties. This paper presents a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uncertainty injection</i> scheme for training machine learning models that are capable of implicitly accounting for the uncertainties and producing statistically robust solutions. We further identify the wireless communications as an application field where uncertainties are prevalent in problem parameters such as the channel coefficients. We show the effectiveness of the proposed training scheme in two applications: the robust power loading for multiuser multiple-input-multiple-output (MIMO) downlink transmissions; and the robust power control for device-to-device (D2D) networks.

Development of a novel RBFNN‐trained nonlinear channel equalizer based on GDEBOA technique

AbstractThe equalization of digital channels is generally known to be a nonlinear classification problem. Applications such as these can benefit from networks that approximate nonlinear mappings. It gets good performance by adjusting only one coefficient and one center closest to the input vector of the radial basis function network (RBFN), which is a simplified version of stochastic gradient techniques. Artificial Neural Networks (ANNs) are suitable for channel equalization because they have the capability to map between variables. Having only one hidden layer in Radial Basis Function Neural Network (RBFNN) makes it the most preferred equalizer to mitigate distortions in the channels. The ability to equalize nonlinear channels is strength of radial RBFNN, which is a simplified version of stochastic gradient techniques. The conventional “hit and trial” approach poses the greatest difficulty when designing RBFNN Equalizers. As a solution to these limitations, this work suggests a training strategy based on Gaussian distribution estimation strategy (GDE) in hybrid butterfly optimization algorithm (GDEBOA) for RBFNN channel equalizer. The proposed RBFNN equalizer is being trained using a population‐based optimization algorithm. An algorithm, called GDEBOA, based on a GDE approach is developed in this paper. The proposed training scheme significantly outperforms existing metaheuristic algorithms in terms of mean square error (MSE) and bit error rate (BER). Furthermore, based on burst error scenarios and bit error probability (BEP), the proposed method has demonstrated greater robustness than other methods when faced with such scenarios. The effectiveness of the suggested scheme over a wide range of signal‐to‐noise ratios has been validated by several simulation studies. Additionally, statistical importance of the suggested technique is analyzed. It is visualized that GDEBOA performs better than existing algorithms for training RBFNN equalizer.

Deep Segmentation Networks for Segmenting Kidneys and Detecting Kidney Stones in Unenhanced Abdominal CT Images.

Recent breakthroughs of deep learning algorithms in medical imaging, automated detection, and segmentation techniques for renal (kidney) in abdominal computed tomography (CT) images have been limited. Radiomics and machine learning analyses of renal diseases rely on the automatic segmentation of kidneys in CT images. Inspired by this, our primary aim is to utilize deep semantic segmentation learning models with a proposed training scheme to achieve precise and accurate segmentation outcomes. Moreover, this work aims to provide the community with an open-source, unenhanced abdominal CT dataset for training and testing the deep learning segmentation networks to segment kidneys and detect kidney stones. Five variations of deep segmentation networks are trained and tested both dependently (based on the proposed training scheme) and independently. Upon comparison, the models trained with the proposed training scheme enable the highly accurate 2D and 3D segmentation of kidneys and kidney stones. We believe this work is a fundamental step toward AI-driven diagnostic strategies, which can be an essential component of personalized patient care and improved decision-making in treating kidney diseases.

Bridging the gap between patient-specific and patient-independent seizure prediction via knowledge distillation

Objective. Deep neural networks (DNNs) have shown unprecedented success in various brain-machine interface applications such as epileptic seizure prediction. However, existing approaches typically train models in a patient-specific fashion due to the highly personalized characteristics of epileptic signals. Therefore, only a limited number of labeled recordings from each subject can be used for training. As a consequence, current DNN based methods demonstrate poor generalization ability to some extent due to the insufficiency of training data. On the other hand, patient-independent models attempt to utilize more patient data to train a universal model for all patients by pooling patient data together. Despite different techniques applied, results show that patient-independent models perform worse than patient-specific models due to high individual variation across patients. A substantial gap thus exists between patient-specific and patient-independent models. Approach. In this paper, we propose a novel training scheme based on knowledge distillation which makes use of a large amount of data from multiple subjects. It first distills informative features from signals of all available subjects with a pre-trained general model. A patient-specific model can then be obtained with the help of distilled knowledge and additional personalized data. Main results. Four state-of-the-art seizure prediction methods are trained on the Children’s Hospital of Boston-MIT sEEG database with our proposed scheme. The resulting accuracy, sensitivity, and false prediction rate show that our proposed training scheme consistently improves the prediction performance of state-of-the-art methods by a large margin. Significance. The proposed training scheme significantly improves the performance of patient-specific seizure predictors and bridges the gap between patient-specific and patient-independent predictors.

Physiology-based augmented deep neural network frameworks for ECG biometrics with short ECG pulses considering varying heart rates

• ECG is a promising biometric method with high performance, liveness test capability, and wearable sensor availability. • Varying heartbeat rates is a major issue in ECG biometrics , leading to the performance degradation . • We propose a physiology-based augmented DNN framework for ECG biometrics with short pulses to encapsulate any DNN methods. • Our framework has enhanced DNN ECG biometrics; up to 11.7% improvement in accuracy over multiple sessions. Electrocardiogram (ECG) has been investigated as promising biometrics with high authentication accuracy, natural liveness test ability, and wearable sensor availability. There have been many algorithms developed for ECG biometric authentication or identification including recent state-of-the-art deep learning (DL) methods that usually yielded excellent performance with real ECG data in ideal conditions. However, one of the challenges against ideal conditions is the intra-personal variability of ECG pulses due to heart beat rate changes. Due to this variability, ECG based biometric methods have experienced significant performance degradation. It is especially challenging when a small number of ECG pulses must be used for biometrics with fast response authentication since there is not enough information available to correct for different heart rates. In this letter, we investigated DL based ECG biometrics with the input of a small number of ECG pulses considering varying heart rates. We propose physiology-based augmented deep neural network (DNN) frameworks for ECG biometric methods that are based on the Hodges’ QT interval correction. Unlike QT interval correction methods, our proposed framework does not require the estimated heart rate. Our proposed training and testing schemes were evaluated with representative DL based biometric methods using CNN and RNN with very short ECG pulses (1 or 3 pulses per authentication) from the public multi-session ECG-ID dataset (83 subjects). We exploited the ECG-ID dataset to simulate the challenging scenario including the enrollment and authentication happening over relatively long time duration so that heart rate variation is likely occurring. Our augmented DNN frameworks yielded significantly better performance than the original DL based biometrics; up to 11.7% improvement in accuracy and 8.6% improvement in sensitivity simultaneously with 99.9% specificity.

Deep Measurement Updates for Bayes Filters

Measurement update rules for Bayes filters often contain hand-crafted heuristics to compute observation probabilities for high-dimensional sensor data, like images. In this work, we propose the novel approach Deep Measurement Update (DMU) as a general update rule for a wide range of systems. DMU has a conditional encoder-decoder neural network structure to process depth images as raw inputs. Even though the network is trained only on synthetic data, the model shows good performance at evaluation time on real-world data. With our proposed training scheme primed data training , we demonstrate how the DMU models can be trained efficiently to be sensitive to condition variables without having to rely on a stochastic information bottleneck. We validate the proposed methods in multiple scenarios of increasing complexity, beginning with the pose estimation of a single object to the joint estimation of the pose and the internal state of an articulated system. Moreover, we provide a benchmark against Articulated Signed Distance Functions(A-SDF) on the RBO dataset as a baseline comparison for articulation state estimation.

Ear Recognition Based on Deep Unsupervised Active Learning

Cooperative machine learning has many applications, such as data annotation, where an initial model trained with partially labeled data is used to predict labels for unseen data continuously. Predicted labels with a low confidence value are manually revised to allow the model to be retrained with the predicted and revised data. In this paper, we propose an alternative to this approach: an initial training process called Deep Unsupervised Active Learning. Using the proposed training scheme, a classification model can incrementally acquire new knowledge during the testing phase without manual guidance or correction of decision making. The training process consists of two stages: the first stage of supervised training using a classification model, and an unsupervised active learning stage during the test phase. The labels predicted during the test phase, with high confidence, are continuously used to extend the knowledge base of the model. To optimize the proposed method, the model must have a high initial recognition rate. To this end, we exploited the Visual Geometric Group (VGG16) pre-trained model applied to three datasets: Mathematical Image Analysis (AMI), University of Science and Technology Beijing (USTB2), and Annotated Web Ears (AWE). This approach achieved impressive performance that shows a significant improvement in the recognition rate of the USTB2 dataset by coloring its images using a Generative Adversarial Network (GAN). The obtained performances are interesting compared to the current methods: the recognition rates are 100.00%, 98.33%, and 51.25% for the USTB2, AMI, and AWE datasets, respectively.

Deep Feature Extraction for Resting-State Functional MRI by Self-Supervised Learning and Application to Schizophrenia Diagnosis.

In this study, we propose a deep-learning technique for functional MRI analysis. We introduced a novel self-supervised learning scheme that is particularly useful for functional MRI wherein the subject identity is used as the teacher signal of a neural network. The neural network is trained solely based on functional MRI-scans, and the training does not require any explicit labels. The proposed method demonstrated that each temporal volume of resting state functional MRI contains enough information to identify the subject. The network learned a feature space in which the features were clustered per subject for the test data as well as for the training data; this is unlike the features extracted by conventional methods including region of interests (ROIs) pooling signals and principal component analysis. In addition, applying a simple linear classifier to the per-subject mean of the features (namely “identity feature”), we demonstrated that the extracted features could contribute to schizophrenia diagnosis. The classification accuracy of our identity features was comparable to that of the conventional functional connectivity. Our results suggested that our proposed training scheme of the neural network captured brain functioning related to the diagnosis of psychiatric disorders as well as the identity of the subject. Our results together highlight the validity of our proposed technique as a design for self-supervised learning.

Automatic skull defect restoration and cranial implant generation for cranioplasty

A fast and fully automatic design of 3D printed patient-specific cranial implants is highly desired in cranioplasty - the process to restore a defect on the skull. We formulate skull defect restoration as a 3D volumetric shape completion task, where a partial skull volume is completed automatically. The difference between the completed skull and the partial skull is the restored defect; in other words, the implant that can be used in cranioplasty. To fulfill the task of volumetric shape completion, a fully data-driven approach is proposed. Supervised skull shape learning is performed on a database containing 167 high-resolution healthy skulls. In these skulls, synthetic defects are injected to create training and evaluation data pairs. We propose a patch-based training scheme tailored for dealing with high-resolution and spatially sparse data, which overcomes the disadvantages of conventional patch-based training methods in high-resolution volumetric shape completion tasks. In particular, the conventional patch-based training is applied to images of high resolution and proves to be effective in tasks such as segmentation. However, we demonstrate the limitations of conventional patch-based training for shape completion tasks, where the overall shape distribution of the target has to be learnt, since it cannot be captured efficiently by a sub-volume cropped from the target. Additionally, the standard dense implementation of a convolutional neural network tends to perform poorly on sparse data, such as the skull, which has a low voxel occupancy rate. Our proposed training scheme encourages a convolutional neural network to learn from the high-resolution and spatially sparse data. In our study, we show that our deep learning models, trained on healthy skulls with synthetic defects, can be transferred directly to craniotomy skulls with real defects of greater irregularity, and the results show promise for clinical use. Project page: https://github.com/Jianningli/MIA.

Playing to distraction: towards a robust training of CNN classifiers through visual explanation techniques

The field of deep learning is evolving in different directions, with still the need for more efficient training strategies. In this work, we present a novel and robust training scheme that integrates visual explanation techniques in the learning process. Unlike the attention mechanisms that focus on the relevant parts of images, we aim to improve the robustness of the model by making it pay attention to other regions as well. Broadly speaking, the idea is to distract the classifier in the learning process to force it to focus not only on relevant regions but also on those that, a priori, are not so informative for the discrimination of the class. We tested the proposed approach by embedding it into the learning process of a convolutional neural network for the analysis and classification of two well-known datasets, namely Stanford cars and FGVC-Aircraft. Furthermore, we evaluated our model on a real-case scenario for the classification of egocentric images, allowing us to obtain relevant information about peoples' lifestyles. In particular, we work on the challenging EgoFoodPlaces dataset, achieving state-of-the-art results with a lower level of complexity. The obtained results indicate the suitability of our proposed training scheme for image classification, improving the robustness of the final model.

Estimating Calibrated Individualized Survival Curves with Deep Learning

In survival analysis, deep learning approaches have been proposed for estimating an individual's probability of survival over some time horizon. Such approaches can capture complex non-linear relationships, without relying on restrictive assumptions regarding the relationship between an individual's characteristics and their underlying survival process. To date, however, these methods have focused primarily on optimizing discriminative performance and have ignored model calibration. Well-calibrated survival curves present realistic and meaningful probabilistic estimates of the true underlying survival process for an individual. However, due to the lack of ground-truth regarding the underlying stochastic process of survival for an individual, optimizing and measuring calibration in survival analysis is an inherently difficult task. In this work, we i) highlight the shortcomings of existing approaches in terms of calibration and ii) propose a new training scheme for optimizing deep survival analysis models that maximizes discriminative performance, subject to good calibration. Compared to state-of-the-art approaches across two publicly available datasets, our proposed training scheme leads to significant improvements in calibration, while maintaining good discriminative performance.

Multi-DOA estimation based on the KR image tensor and improved estimation network

Deep neural networks have shown great performance for direction-of-arrival (DOA) estimation problem, but it is necessary to design some suitable networks to solve the multi-DOA estimation problem. In this paper, we use Khatri–Rao product to increase the degree of freedom of antenna array and obtain the image tensor of covariance matrix, then we propose an improved estimation network to process the tensor. We use the curriculum learning scheme and partial label strategy to develop a CurriculumNet training scheme. The training/validation results shows that the proposed training scheme can increase the generalization of the estimation network and improve the accuracy of network around 10%. The estimation performance of the proposed network shows high-resolution results, which can distinguish two adjacent signals with angle difference of 3^circ . Moreover, the proposed estimation network has root mean square estimation error lower than 1^circ when signal noise ratio equals -,10,{mathrm {dB}} and can estimate DOAs precisely by only 8 snapshots, which performs much better than prior deep neural network based estimation methods and can estimate multi-DOA results under hostile estimation environments.

Fooling intrusion detection systems using adversarially autoencoder

Due to the increasing cyber-attacks, various Intrusion Detection Systems (IDSs) have been proposed to identify network anomalies. Most existing machine learning-based IDSs learn patterns from the features extracted from network traffic flows, and the deep learning-based approaches can learn data distribution features from the raw data to differentiate normal and anomalous network flows. Although having been used in the real world widely, the above methods are vulnerable to some types of attacks. In this paper, we propose a novel attack framework, Anti-Intrusion Detection AutoEncoder (AIDAE), to generate features to disable the IDS. In the proposed framework, an encoder transforms features into a latent space, and multiple decoders reconstruct the continuous and discrete features, respectively. Additionally, a generative adversarial network is used to learn the flexible prior distribution of the latent space. The correlation between continuous and discrete features can be kept by using the proposed training scheme. Experiments conducted on NSL-KDD, UNSW-NB15, and CICIDS2017 datasets show that the generated features indeed degrade the detection performance of existing IDSs dramatically.

A new training scheme for neural network based non-linear channel equalizers in wireless communication system using Cuckoo Search Algorithm

Widespread use of the internet has resulted in a massive rise in the data rate of a wireless communication system. Eventually, to mitigate the effect of inter-symbol interference (ISI) in multipath wireless channels, designing a channel equalizer is becoming more demanding. For severe non-linear distortion, the neural network (NN) based non-linear channel equalizers provide superior performance than the adaptive filter based linear equalizers. The NN equalizers are generally trained with back-propagation algorithm which has limitations of slower convergence, local minima stagnation and sensitivity to initial parameters. To overcome these limitations, this paper proposes a training scheme using Cuckoo Search Algorithm (CSA) for functional link artificial NN (FLANN) based channel equalizers. The proposed training scheme has a better ability to escape from local minima, higher exploitation and exploration capabilities. To choose the optimum values of the parameters, the sensitivity analysis of the CSA based approach is performed with its key parameters. Furthermore, three non-linear channels have been simulated to demonstrate the equalization performance of the CSA based training scheme and the results have been compared with recent and well-established algorithms. The simulations confirm that the proposed training scheme performs substantially better than existing metaheuristic algorithms in terms of BER and MSE performance. To show the robustness of the CSA based method, the burst error scenario has been considered and results proved that the method is more successful in handling such scenarios when compared to other methods. The performance of the proposed scheme has been validated for a wide range of signal-to-noise ratio through simulation studies and it is observed that the scheme outperforms the other algorithms in poor SNR conditions as well. Also, to examine the statistical significance of the results provided by the proposed scheme, the Wilcoxon test is performed and the test reveals that the obtained results are statistically significant.

Acoustic plane-wave decomposition by means of multilayer perceptron neural networks

Acoustic mode decomposition is used for evaluating the damping of aircraft liners and, in general, to investigate acoustic scattering in flow ducts. Classical methods rely on analytical solutions of the wave properties and accept uncertainties due to simplified descriptions of the duct flow. In contrast, the current study provides a wave decomposition method that does not require explicit analytical knowledge of the wave properties and registers a wide range of flow-related acoustic phenomena. A multilayer perceptron artificial neural network is trained to learn acoustic wave decomposition for plane-wave-like duct modes. Training data are the numerical solutions of the Linearized Navier-Stokes Equations, from which the network not only learns the wave motion properties but also the dispersion of sound into the fluid flow. The network can account for flow-related effects, such as turbulent attenuation, refraction, convection, and thermo-viscous dissipation, which are only included in the classical models based on simplifications. The new method is validated for plane-waves against analytical data and experiments. It is demonstrated that the network can mimic the classical solutions accurately when trained under the same flow simplifications. In addition, it can cope with complex flow effects, such as turbulent attenuation, by including them in the training data. Therefore, the proposed wave decomposition complements the classical plane-wave decomposition when investigating in-duct sound with complex flow conditions. An important continuation of this work is to extend the new wave decomposition method to multi-modal sound fields. As the first step in this direction, it is demonstrated that the proposed training scheme also works for higher-order modes.

Accurate and robust neural networks for face morphing attack detection

Artificial neural networks tend to use only what they need for a task. For example, to recognize a rooster, a network might only considers the rooster’s red comb and wattle and ignores the rest of the animal. This makes them vulnerable to attacks on their decision making process and can worsen their generality. Thus, this phenomenon has to be considered during the training of networks, especially in safety and security related applications. In this paper, we propose neural network training schemes, which are based on different alternations of the training data, to increase robustness and generality. Precisely, we limit the amount and position of information available to the neural network for the decision making process and study their effects on the accuracy, generality, and robustness against semantic and black box attacks for the particular example of face morphing attacks. In addition, we exploit layer-wise relevance propagation (LRP) to analyze the differences in the decision making process of the differently trained neural networks. A face morphing attack is an attack on a biometric facial recognition system, where the system is fooled to match two different individuals with the same synthetic face image. Such a synthetic image can be created by aligning and blending images of the two individuals that should be matched with this image. We train neural networks for face morphing attack detection using our proposed training schemes and show that they lead to an improvement of robustness against attacks on neural networks. Using LRP, we show that the improved training forces the networks to develop and use reliable models for all regions of the analyzed image. This redundancy in representation is of crucial importance to security related applications.

HLHLp: Quantized Neural Networks Training for Reaching Flat Minima in Loss Surface

Quantization of deep neural networks is extremely essential for efficient implementations. Low-precision networks are typically designed to represent original floating-point counterparts with high fidelity, and several elaborate quantization algorithms have been developed. We propose a novel training scheme for quantized neural networks to reach flat minima in the loss surface with the aid of quantization noise. The proposed training scheme employs high-low-high-low precision in an alternating manner for network training. The learning rate is also abruptly changed at each stage for coarse- or fine-tuning. With the proposed training technique, we show quite good performance improvements for convolutional neural networks when compared to the previous fine-tuning based quantization scheme. We achieve the state-of-the-art results for recurrent neural network based language modeling with 2-bit weight and activation.

Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS

Tacotron-based end-to-end speech synthesis has shown remarkable voice quality. However, the rendering of prosody in the synthesized speech remains to be improved, especially for long sentences, where prosodic phrasing errors can occur frequently. In this paper, we extend the Tacotron-based speech synthesis framework to explicitly model the prosodic phrase breaks. We propose a multi-task learning scheme for Tacotron training, that optimizes the system to predict both Mel spectrum and phrase breaks. To our best knowledge, this is the first implementation of multi-task learning for Tacotron based TTS with a prosodic phrasing model. Experiments show that our proposed training scheme consistently improves the voice quality for both Chinese and Mongolian systems.

Skills and Training Requirements for the Future Transportation Sector of Europe

More than 10 million people are directly employed by the transport industry in Europe, accounting for 4.5 % of total employment and representing 4.6 % of Gross Domestic Product (GDP). This fact, combined with the rapid developments and changes of the sector, makes imperative the need to create, attract and retain appropriate staff. As the overall trend is to increase automation, the sector will depend more and more on specialised equipment and products. Future jobs will therefore require new and advanced skills in engineering as well as in back office operations, but at the same time, the growing interdisciplinary elements of transport activities will also require transport professionals with developed skills in safety, security, logistics, IT, behavioural sciences, marketing and economics. The European Research project SKILLFUL has developed a structured foresight into the vocational and academic qualifications in the Transportation sector of the future and has proposed training schemes and their supportive business models that could ideally be adopted European-wide, to enhance employability and sustainable industrial development in the transportation sector in Europe. The identification of future requirements constituted the basis of the project. The impact of new technologies and game changers, as well as emphasis on intermodality and interdisciplinarity on employability and future worker skills, have led to the development of relevant scenarios on future jobs knowledge and skills requirements, regarding the road transport in Europe. This has led to the identification and design of proper and specific curricula for training (with emphasis on middle-skilled professionals and lifelong learning), while also to the introduction of six novel concepts of business actors, expected to facilitate the training process and enhance the transport-education chain. The project goes a step beyond by addressing also critical issues towards a PanEuropean master curriculum on transport.