Training Deep Learning Networks Research Articles

An Automatic Path Planning Method of Pedicle Screw Placement Based on Preoperative CT Images

Due to good fixation and obvious clinical advantages in the treatment of spinal diseases, pedicle screw technology is widely employed in spinal surgery. However, most screw placement paths are manually selected, which is a complicated and time-consuming process. Therefore, various kinds of spine surgery robots have been invented to assist during surgeries. An automatic path planning method of pedicle screw placement is proposed in this article. First, based on preoperative CT scans, the spine of the patient can be segmented and each individual vertebra is classified by using a trained deep learning network model. Additionally, a local coordinate system in every single vertebra is established according to its anatomical characteristics. Second, after analyzing the images to recognize the feature points, the pedicle region and route of screw placement are identified, which renders completion of the surgical path planning automatic. The experimental results show that 93% of the planning results can be directly applied in surgeries after surgeon evaluation. In addition to the improvement in accuracy, the whole algorithm does not rely on any human assistance. This fully automatic process greatly reduces the time required for surgical planning.

High-Accuracy and Adaptive Fault Diagnosis of High-Speed Train Bogie Using Dense-Squeeze Network

As one of the most important systems of high-speed train (HST), bogie system matters when it comes to the safety and reliability of HST operation. In order to strengthen feature propagation and alleviate vanishing gradients problems in training deep-learning networks, a novel Dense-Squeeze Network based on one-dimensional convolutional neural networks (1D-CNN) is proposed for bogie fault diagnosis. On the one hand, dense blocks are capable of facilitating feature reuse and increasing the depth of feature propagation. On the other hand, the squeeze operation automatically measures the importance level of each feature channel, and then enhance the useful features and suppress the useless features for the current task. The experimental results tested by the HST model CRH380 A at different speeds verify the effectiveness of the proposed method, and the accuracy of fault diagnosis converges to 99.66%. Compared with other deep-learning-based methods, such as 1D-CNN, Long Short-Term Memory (LSTM) and DenseNet, the superiority of the proposed scheme is demonstrated clearly.

An empirical study of cyclical learning rate on neural machine translation

AbstractIn training deep learning networks, the optimizer and related learning rate are often used without much thought or with minimal tuning, even though it is crucial in ensuring a fast convergence to a good quality minimum of the loss function that can also generalize well on the test dataset. Drawing inspiration from the successful application of cyclical learning rate policy to computer vision tasks, we explore how cyclical learning rate can be applied to train transformer-based neural networks for neural machine translation. From our carefully designed experiments, we show that the choice of optimizers and the associated cyclical learning rate policy can have a significant impact on the performance. In addition, we establish guidelines when applying cyclical learning rates to neural machine translation tasks.

BPLight-CNN: A Photonics-Based Backpropagation Accelerator for Deep Learning

Training deep learning networks involves continuous weight updates across the various layers of the deep network while using a backpropagation (BP) algorithm. This results in expensive computation overheads during training. Consequently, most deep learning accelerators today employ pretrained weights and focus only on improving the design of the inference phase. The recent trend is to build a complete deep learning accelerator by incorporating the training module. Such efforts require an ultra-fast chip architecture for executing the BP algorithm. In this article, we propose a novel photonics-based backpropagation accelerator for high-performance deep learning training. We present the design for a convolutional neural network (CNN), BPLight-CNN , which incorporates the silicon photonics-based backpropagation accelerator. BPLight-CNN is a first-of-its-kind photonic and memristor-based CNN architecture for end-to-end training and prediction. We evaluate BPLight-CNN using a photonic CAD framework (IPKISS) on deep learning benchmark models, including LeNet and VGG-Net. The proposed design achieves (i) at least 34× speedup, 34× improvement in computational efficiency, and 38.5× energy savings during training; and (ii) 29× speedup, 31× improvement in computational efficiency, and 38.7× improvement in energy savings during inference compared with the state-of-the-art designs. All of these comparisons are done at a 16-bit resolution, and BPLight-CNN achieves these improvements at a cost of approximately 6% lower accuracy compared with the state-of-the-art.

AMERP: Adam moment estimation optimized mobility supported energy efficient routing protocol for wireless body area networks

The technological advancements have led to a revolutionized change in the field of the health care industry. One of these promising solutions is the Wireless Body Area Networks (WBAN) which has been providing smart healthcare solutions to the consumers. WBAN deals with the sensor nodes that are placed on or embedded into the human body. It is a very challenging task for the sensor nodes to transmit the medical information to the sink node with minimum energy consumption, minimum loss of packets and with longer network lifetime. The present paper proposes a routing protocol based on Adam Moment Estimation (Adam) optimizer trained deep learning network that is used to enhance the performance of both homogeneous and heterogeneous network configurations. The proposed protocol sustains the performance of the WBAN by providing improvement in the various parameters. The proposed routing protocol named as AMERP (Adam Moment Estimation Optimized Mobility Supported Energy Efficient Routing Protocol) supports mobility of nodes, normal traffic, multi hop communication, minimum energy consumption, enhanced average lifetime and lesser packet drop rate. The performance of the presented protocol has also been compared graphically with the routing protocols like SIMPLE (Nadeem et al., 2013), iM-SIMPLE (Javaid et al., 2015), M-ATTEMPT (Javaidet al., 2013) for both homogeneous and heterogeneous cases. It has been evaluated that the AMERP provides improvement of 39.8 % in number of packets received, 12.7 % improvement in average network lifetime and 49.2 % improvement in residual energy as compared to iM-SIMPLE protocol for homogeneous network. The percentage improvement of AMERP over iM-SIMPLE protocol for heterogeneous network is 15 % in number of packets received, 51 % in average network lifetime and 47.4 % in residual energy.

FindPorpoises: Deep learning for detection of harbor porpoise echolocation clicks

Tom Norris expressed conservation concern for the vaquita, a close relative of harbor porpoises; here we report on a harbor porpoise (Phocoena phocoena) conservation effort. Tidal energy devices are installed in high-flow estuaries that are also prime harbor porpoise habitat. To study tidal energy device impacts on porpoises, sound was recorded in Minas Passage, Bay of Fundy, Canada. Analysis aimed to distinguish harbor porpoises from noise sources. “Click candidate” sounds were detected using the ratio between the harbor porpoise frequency band and lower guard band, then reviewed by humans to label which were correct. Because more “correct” instances were needed, data were augmented by mixing porpoise clicks with known noise, producing 20,000 “click present” labeled instances. Additionally, 20 000 “non-click” instances were extracted from noise recordings. Labeled instances were made into a 0.5-s equalized spectrograms for training deep-learning networks. Of the network architectures tried, the best was a convolutional neural network with pooling and fully connected layers, achieving 99.1% accuracy. User-friendly “FindPorpoises” software was made to pre-process raw files, detect candidate clicks, use the trained network to sort candidate clicks into correct and incorrect instances, and plot and tabulate the results by time of day, tide cycle, and month. [Work supported by OERA.]

Deep learning virtual colorization overcoming chromatic aberrations in singlet lens microscopy

Singlet lenses are free from precise assembling, aligning, and testing, which are helpful for the development of portable and low-cost microscopes. However, balancing the spectrum dispersion or chromatic aberrations using a singlet lens made of one material is difficult. Here, a novel method combining singlet lens microscopy and computational imaging, which is based on deep learning image-style-transfer algorithms, is proposed to overcome this problem in clinical pathological slide microscopy. In this manuscript, a singlet aspheric lens is used, which has a high cut-off frequency and linear signal properties. Enhanced by a trained deep learning network, it is easy to transfer the monochromatic gray-scale microscopy picture to a colorful microscopy picture, with only one single-shot recording by a monochromatic CMOS image sensor. By experiments, data analysis, and discussions, it is proved that our proposed virtual colorization microscope imaging method is effective for H&E stained tumor tissue slides in singlet microscopy. It is believable that the computational virtual colorization method for singlet microscopes would promote the low-cost and portable singlet microscopy development in medical pathological label staining observing (e.g., H&E staining, Gram staining, and fluorescent labeling) biomedical research.

An Efficient Method for Training Deep Learning Networks Distributed

An Efficient Method for Training Deep Learning Networks Distributed

Intelligent Signal Classification in Industrial Distributed Wireless Sensor Networks Based Industrial Internet of Things

In industrial sensor networks, complex industrial environments may be encountered leading to a mix of signals of different types. Complicated interference caused by mixed signals on industrial equipments may significantly degrade the classification rate of signals, which may result in a long training time in order to extract features. In addition, with limited channel resources, it is difficult to make the global optimal decision in industrial distributed wireless sensor networks. To address this problem, a signal classification method using feature fusion is proposed for industrial Internet of Things in this article. In the proposed method, the received signals of nodes are processed by frequency reduction and sampling pretreatment, based on which intelligent representations of signals are obtained. Using federated learning, the data samples are trained with the feature fusion network. Moreover, the trained deep learning network is used on each sensor node to classify signals, the results of which will be transmitted to aggregation center. In the aggregation center, the improved evidence theory method is used to aggregate the recognition results of each sensor node to achieve the final classification. Simulation shows that the proposed method has excellent classification performances. Notably, it is not required for the proposed method to transmit signals from nodes to the aggregation center, which could effectively protect the privacy of industrial information.

Microwave Imaging by Deep Learning Network: Feasibility and Training Method.

Microwave image reconstruction based on a deep-learning method is investigated in this paper. The neural network is capable of converting measured microwave signals acquired from a 24×24 antenna array at 4 GHz into a 128×128 image. To reduce the training difficulty, we first developed an autoencoder by which high-resolution images (128×128) were represented with 256×1 vectors; then we developed the second neural network which aimed to map microwave signals to the compressed features (256×1 vector). Two neural networks can be combined to a full network to make reconstructions, when both are successfully developed. The present two-stage training method reduces the difficulty in training deep learning networks (DLN) for inverse reconstruction. The developed neural network is validated by simulation examples and experimental data with objects in different shapes/sizes, placed in different locations, and with dielectric constant ranging from 2~6. Comparisons between the imaging results achieved by the present method and two conventional approaches: distorted Born iterative method (DBIM) and phase confocal method (PCM) are also provided.

CORN: An Alternative Way to Utilize Time-Series Data of SAR Images in Newly Built Construction Detection

The limitations in obtaining sufficient datasets for training deep learning networks is preventing many applications from achieving accurate results, especially when detecting new constructions using time-series satellite imagery, since this requires at least two images of the same scene and it must contain new constructions in it. To tackle this problem, we introduce Chronological Order Reverse Network (CORN)—an architecture for detecting newly built constructions in time-series SAR images that does not require a large quantity of training data. The network uses two U-net adaptations to learn the changes between images from both Time 1–Time 2 and Time 2–Time 1 formats, which allows it to learn double the amount of changes in different perspectives. We trained the network with 2028 pairs of 256 × 256 pixel SAR images from ALOS-PALSAR, totaling 4056 pairs for the network to learn from, since it learns from both Time 1–Time 2 and Time 2–Time 1. As a result, the network can detect new constructions more accurately, especially at the building boundary, compared to the original U-net trained by the same amount of training data. The experiment also shows that the model trained with CORN can be used with images from Sentinel-1. The source code is available at https://github.com/Raveerat-titech/CORN.

Space Debris Detection Using Feature Learning of Candidate Regions in Optical Image Sequences

Space debris detection is important in space situation awareness and space asset protection. In this article, we propose a method to detect space debris using feature learning of candidate regions. The acquired optical image sequences are first processed to remove hot pixels and flicker noise, and the nonuniform background information is removed by the proposed one dimensional mean iteration method. Then, the feature learning of candidate regions (FLCR) method is proposed to extract the candidate regions and to detect space debris. The candidate regions of space debris are precisely extracted, and then classified by a trained deep learning network. The feature learning model is trained using a large number of simulated space debris with different signal to noise ratios (SNRs) and motion parameters, instead of using real space debris, which make it difficult to extract a sufficient number of real space debris with diverse parameters in optical image sequences. Finally, the candidate regions are precisely placed in the optical image sequences. The experiment is performed using the simulated data and acquired image sequences. The results show that the proposed method has good performance when estimating and removing background, and it can detect low SNR space debris with high detection probability.

Adiabatic Quantum Computation Applied to Deep Learning Networks.

Training deep learning networks is a difficult task due to computational complexity, and this is traditionally handled by simplifying network topology to enable parallel computation on graphical processing units (GPUs). However, the emergence of quantum devices allows reconsideration of complex topologies. We illustrate a particular network topology that can be trained to classify MNIST data (an image dataset of handwritten digits) and neutrino detection data using a restricted form of adiabatic quantum computation known as quantum annealing performed by a D-Wave processor. We provide a brief description of the hardware and how it solves Ising models, how we translate our data into the corresponding Ising models, and how we use available expanded topology options to explore potential performance improvements. Although we focus on the application of quantum annealing in this article, the work discussed here is just one of three approaches we explored as part of a larger project that considers alternative means for training deep learning networks. The other approaches involve using a high performance computing (HPC) environment to automatically find network topologies with good performance and using neuromorphic computing to find a low-power solution for training deep learning networks. Our results show that our quantum approach can find good network parameters in a reasonable time despite increased network topology complexity; that HPC can find good parameters for traditional, simplified network topologies; and that neuromorphic computers can use low power memristive hardware to represent complex topologies and parameters derived from other architecture choices.

A comprehensive non-invasive framework for diagnosing prostate cancer

Early detection of prostate cancer increases chances of patients' survival. Our automated non-invasive system for computer-aided diagnosis (CAD) of prostate cancer segments the prostate on diffusion-weighted magnetic resonance images (DW-MRI) acquired at different b-values, estimates its apparent diffusion coefficients (ADC), and classifies their descriptors – empirical cumulative distribution functions (CDF) – with a trained deep learning network. To segment the prostate, an evolving geometric (level-set-based) deformable model is guided by a speed function depending on intensity attributes extracted from the DW-MRI with nonnegative matrix factorization (NMF). For a more robust evolution, the attributes are fused with a probabilistic shape prior and estimated spatial dependencies between prostate voxels. To preserve continuity, the ADCs of the segmented prostate volume at different b-values are normalized and refined using a generalized Gauss-Markov random field image model. The CDFs of the refined ADCs at different b-values are considered global water diffusion features and used to distinguish between benign and malignant prostates. A deep learning network of stacked non-negativity-constrained auto-encoders (SNCAE) is trained to classify the benign or malignant prostates on the basis of the constructed CDFs. Our experiments on 53 clinical DW-MRI data sets resulted in 92.3% accuracy, 83.3% sensitivity, and 100% specificity, indicating that the proposed CAD system could be used as a reliable non-invasive diagnostic tool.