Multi-resolution Convolutional Neural Networks Research Articles

3D-MRI super-resolution reconstruction using multi-modality based on multi-resolution CNN

Background and ObjectiveHigh-resolution (HR) MR images provide rich structural detail to assist physicians in clinical diagnosis and treatment plan. However, it is arduous to acquire HR MRI due to equipment limitations, scanning time or patient comfort. Instead, HR MRI could be obtained through a number of computer assisted post-processing methods that have proven to be effective and reliable. This paper aims to develop a convolutional neural network (CNN) based super-resolution reconstruction framework for low-resolution (LR) T2w images. MethodIn this paper, we propose a novel multi-modal HR MRI generation framework based on deep learning techniques. Specifically, we construct a CNN based on multi-resolution analysis to learn an end-to-end mapping between LR T2w and HR T2w, where HR T1w is fed into the network to offer detailed a priori information to help generate HR T2w. Furthermore, a low-frequency filtering module is introduced to filter out the interference from HR-T1w during high-frequency information extraction. Based on the idea of multi-resolution analysis, detailed features extracted from HR T1w and LR T2w are fused at two scales in the network and then HR T2w is reconstructed by upsampling and dense connectivity module. ResultsExtensive quantitative and qualitative evaluations demonstrate that the proposed method enhances the recovered HR T2w details and outperforms other state-of-the-art methods. In addition, the experimental results also suggest that our network has a lightweight structure and favorable generalization performance. ConclusionThe results show that the proposed method is capable of reconstructing HR T2w with higher accuracy. Meanwhile, the super-resolution reconstruction results on other dataset illustrate the excellent generalization ability of the method.

An attention-based deep learning method for the detection of electrical status epilepticus during sleep from electroencephalogram waveform analysis in children

Electrical status epilepticus during sleep (ESES) is a special interictal electroencephalographic phenomenon observed in children with epilepsy. ESES refers to continuous or nearly continuous epileptiform discharges of bilateral (or unilateral) spikes and waves induced during non-rapid eye movement (NREM) sleep. It mainly manifests in a series of childhood epileptic syndromes characterized by seizures, ESES, and cognitive impairment, which seriously affect the health of children. Typically, the spike-wave index (SWI), which is the percentage of spikes and waves duration in the total NREM time, is an important criterion for diagnosing ESES. The SWI measures the proportion of the total duration of spikes and waves to the NREM sleep time within several hours. Currently, SWI is obtained primarily by manual measurement and calculation, which has a certain subjective bias, low accuracy, time and labor consumption, and is a waste of medical resources. Therefore, we designed a novel data segmentation model based on a multiresolution convolutional neural network and a self-attention mechanism to calculate the SWI quickly and accurately. This can assist clinicians in making more accurate judgments and providing a better guide for the treatment of children with epilepsy. A large amount of clinical patient data was used to test the proposed model. The results confirmed that this model has an acceptable generalization ability for data from different patients and the same patient at different times. Each performance index of the model was superior to that of current common models, thus fulfilling the requirements for clinical testing.

Multi-Resolution Convolutional Residual Neural Networks for Monaural Speech Dereverberation

Multi-Resolution Convolutional Residual Neural Networks for Monaural Speech Dereverberation

Lung nodule segmentation via semi-residual multi-resolution neural networks.

The integration of deep neural networks and cloud computing has become increasingly prevalent within the domain of medical image processing, facilitated by the recent strides in neural network theory and the advent of the internet of things (IoTs). This juncture has led to the emergence of numerous image segmentation networks and innovative solutions that facilitate medical practitioners in diagnosing lung cancer. Within the contours of this study, we present an end-to-end neural network model, christened as the "semi-residual Multi-resolution Convolutional Neural Network" (semi-residual MCNN), devised to engender precise lung nodule segmentation maps within the milieu of cloud computing. Central to the architecture are three pivotal features, each coalescing to effectuate a notable enhancement in predictive accuracy: the incorporation of semi-residual building blocks, the deployment of group normalization techniques, and the orchestration of multi-resolution output heads. This innovative model is systematically subjected to rigorous training and testing regimes, using the LIDC-IDRI dataset - a widely embraced and accessible repository - comprising a diverse ensemble of 1,018 distinct lung CT images tailored to the realm of lung nodule segmentation.

Depressive Disorder Recognition Based on Frontal EEG Signals and Deep Learning.

Depressive disorder (DD) has become one of the most common mental diseases, seriously endangering both the affected person's psychological and physical health. Nowadays, a DD diagnosis mainly relies on the experience of clinical psychiatrists and subjective scales, lacking objective, accurate, practical, and automatic diagnosis technologies. Recently, electroencephalogram (EEG) signals have been widely applied for DD diagnosis, but mainly with high-density EEG, which can severely limit the efficiency of the EEG data acquisition and reduce the practicability of diagnostic techniques. The current study attempts to achieve accurate and practical DD diagnoses based on combining frontal six-channel electroencephalogram (EEG) signals and deep learning models. To this end, 10 min clinical resting-state EEG signals were collected from 41 DD patients and 34 healthy controls (HCs). Two deep learning models, multi-resolution convolutional neural network (MRCNN) combined with long short-term memory (LSTM) (named MRCNN-LSTM) and MRCNN combined with residual squeeze and excitation (RSE) (named MRCNN-RSE), were proposed for DD recognition. The results of this study showed that the higher EEG frequency band obtained the better classification performance for DD diagnosis. The MRCNN-RSE model achieved the highest classification accuracy of 98.48 ± 0.22% with 8-30 Hz EEG signals. These findings indicated that the proposed analytical framework can provide an accurate and practical strategy for DD diagnosis, as well as essential theoretical and technical support for the treatment and efficacy evaluation of DD.

Soccer, Sleep, Repeat: Effects of Training Characteristics on Sleep Quantity and Sleep Architecture.

Due to the high demands of competitive sports, the sleep architecture of adolescent athletes may be influenced by their regular training. To date, there is no clear evidence on how training characteristics (intensity, time of day, number of sessions) influence sleep quality and quantity. 53 male soccer players (M = 14.36 years, SD = 0.55) of Austrian U15 (n = 45) and U16 elite teams (n = 8) were tested on at least three consecutive days following their habitual training schedules. Participants completed daily sleep protocols (7 a.m., 8 p.m.) and questionnaires assessing sleep quality (PSQI), chronotype (D-MEQ), competition anxiety (WAI-T), and stress/recovery (RESTQ). Electrocardiography (ECG) and actigraphy devices measured sleep. Using sleep protocols and an ECG-based multi-resolution convolutional neural network (MCNN), we found that higher training intensity leads to more wake time, that later training causes longer sleep duration, and that one training session per day was most advantageous for sleep quality. In addition, somatic complaints assessed by the WAI-T negatively affected adolescent athletes' sleep. Individual training loads and longer recovery times after late training sessions during the day should be considered in training schedules, especially for adolescent athletes. MCNN modeling based on ECG data seems promising for efficient sleep analysis in athletes.

Shear-Wave Particle-Velocity Estimation and Enhancement Using a Multi-Resolution Convolutional Neural Network.

Tissue mechanical properties are valuable markers for tissue characterization, aiding in the detection and staging of pathologies. Shear wave elastography (SWE) offers a quantitative assessment of tissue mechanical characteristics based on the SW propagation profile, which is derived from the SW particle motion. Improving the signal-to-noise ratio (SNR) of the SW particle motion would directly enhance the accuracy of the material property estimates such as elasticity or viscosity. In this paper, we present a 3-D multi-resolution convolutional neural network (MRCNN) to perform improved estimation of the SW particle velocity Vz. Additionally, we propose a novel approach to generate training data from real acquisitions, providing high SNR ground truth target data, one-to-one paired to inputs that are corrupted with real-world noise and disturbances. By testing the network on in vitro data acquired from a commercial breast elastography phantom, we show that the MRCNN outperforms Loupas' autocorrelation algorithm with an improved SNR of 4.47dB for the Vz signals, a two-fold decrease in the standard deviation of the downstream elasticity estimates, and a two-fold increase in the contrast-to-noise ratio of the elasticity maps. The generalizability of the network was further demonstrated with a set of ex vivo porcine liver data. The proposed MRCNN outperforms the standard autocorrelation method, in particular in low SNR regimes.

Epileptic seizure prediction based on multiresolution convolutional neural networks

Epilepsy withholds patients’ control of their body or consciousness and puts them at risk in the course of their daily life. This article pursues the development of a smart neurocomputational technology to alert epileptic patients wearing EEG sensors of an impending seizure. An innovative approach for epileptic seizure prediction has been proposed to improve prediction accuracy and reduce the false alarm rate in comparison with state-of-the-art benchmarks. Maximal overlap discrete wavelet transform was used to decompose EEG signals into different frequency resolutions, and a multiresolution convolutional neural network is designed to extract discriminative features from each frequency band. The algorithm automatically generates patient-specific features to best classify preictal and interictal segments of the subject. The method can be applied to any patient case from any dataset without the need for a handcrafted feature extraction procedure. The proposed approach was tested with two popular epilepsy patient datasets. It achieved a sensitivity of 82% and a false prediction rate of 0.058 with the Children’s Hospital Boston-MIT scalp EEG dataset and a sensitivity of 85% and a false prediction rate of 0.19 with the American Epilepsy Society Seizure Prediction Challenge dataset. This technology provides a personalized solution for the patient that has improved sensitivity and specificity, yet because of the algorithm’s intrinsic ability for generalization, it emancipates from the reliance on epileptologists’ expertise to tune a wearable technological aid, which will ultimately help to deploy it broadly, including in medically underserved locations across the globe.

The Virtual Sleep Lab—A Novel Method for Accurate Four-Class Sleep Staging Using Heart-Rate Variability from Low-Cost Wearables

Sleep staging based on polysomnography (PSG) performed by human experts is the de facto "gold standard" for the objective measurement of sleep. PSG and manual sleep staging is, however, personnel-intensive and time-consuming and it is thus impractical to monitor a person's sleep architecture over extended periods. Here, we present a novel, low-cost, automatized, deep learning alternative to PSG sleep staging that provides a reliable epoch-by-epoch four-class sleep staging approach (Wake, Light [N1 + N2], Deep, REM) based solely on inter-beat-interval (IBI) data. Having trained a multi-resolution convolutional neural network (MCNN) on the IBIs of 8898 full-night manually sleep-staged recordings, we tested the MCNN on sleep classification using the IBIs of two low-cost (<EUR 100) consumer wearables: an optical heart rate sensor (VS) and a breast belt (H10), both produced by POLAR®. The overall classification accuracy reached levels comparable to expert inter-rater reliability for both devices (VS: 81%, κ = 0.69; H10: 80.3%, κ = 0.69). In addition, we used the H10 and recorded daily ECG data from 49 participants with sleep complaints over the course of a digital CBT-I-based sleep training program implemented in the App NUKKUAA™. As proof of principle, we classified the IBIs extracted from H10 using the MCNN over the course of the training program and captured sleep-related changes. At the end of the program, participants reported significant improvements in subjective sleep quality and sleep onset latency. Similarly, objective sleep onset latency showed a trend toward improvement. Weekly sleep onset latency, wake time during sleep, and total sleep time also correlated significantly with the subjective reports. The combination of state-of-the-art machine learning with suitable wearables allows continuous and accurate monitoring of sleep in naturalistic settings with profound implications for answering basic and clinical research questions.

LWSleepNet: A lightweight attention-based deep learning model for sleep staging with singlechannel EEG.

Sleep is vital to human health, and sleep staging is an essential process in sleep assessment. However, manual classification is an inefficient task. Along with the increased demand for portable sleep quality detection devices, lightweight automatic sleep staging needs to be developed. This study proposes a novel attention-based lightweight deep learning model called LWSleepNet. A depthwise separable multi-resolution convolutional neural network is introduced to analyze the input feature map and captures features at multiple frequencies using two different sized convolutional kernels. The temporal feature extraction module divides the input into patches and feeds them into a multi-head attention block to extract time-dependent information from sleep recordings. The model's convolution operations are replaced with depthwise separable convolutions to minimize its number of parameters and computational cost. The model's performance on two public datasets (Sleep-EDF-20 and Sleep-EDF-78) was evaluated and compared with those of previous studies. Then, an ablation study and sensitivity analysis were performed to evaluate further each module. LWSleepNet achieves an accuracy of 86.6% and Macro-F1 score of 79.2% for the Sleep-EDF-20 dataset and an accuracy of 81.5% and Macro-F1 score of 74.3% for the Sleep-EDF-78 dataset with only 55.3 million floating-point operations per second and 180 K parameters. On two public datasets, LWSleepNet maintains excellent prediction performance while substantially reducing the number of parameters, demonstrating that our proposed Light multiresolution convolutional neural network and temporal feature extraction modules can provide excellent portability and accuracy and can be easily integrated into portable sleep monitoring devices.

GMCNet: A Generative Multi-Resolution Framework for Cardiac Registration

Deformable image registration plays a crucial role in estimating cardiac deformation from a sequence of images. However, existing registration methods primarily process images as pairs instead of processing all images in a sequence together. This study proposes a novel end-to-end learning-free generative multi-resolution convolutional neural network (GMCNet) with the primary focus of registering images in a sequence. Even though learning-based methods have yielded high performance for image registration, their performance depends on their ability to learn information from a large number of samples which are difficult to obtain and might bias the framework to the specific domain of data. The proposed learning-free method eliminates the need for a dedicated training set while exploiting the capabilities of neural networks to achieve accurate deformation fields. Due to its capability of parameter sharing through the architecture, the GMCNet can be used as a groupwise registration as well as pairwise registration. The proposed method was evaluated on three different clinical cardiac magnetic resonance imaging datasets and compared quantitatively against nine other state-of-the-art learning and optimization-based algorithms. The proposed method outperformed other methods in all comparisons and yielded average Dice metric values ranging from 0.85 to 0.88 for the datasets. Different aspects of the GMCNet are also explored by assessing 1) the robustness; 2) performance on pairwise registration; 3) the influence of spatial transformation in a controlled environment; and 4) the impact of different multi-resolution structures. The results demonstrate that using temporal information to estimate the deformation fields leads to more accurate registration results and improved robustness under different noise levels. Moreover, the proposed method does not need images for training, and therefore, its prediction is not domain-specific and can be applied to any sequence of images.

COVID-19 Diagnosis from Chest X-ray Images Using a Robust Multi-Resolution Analysis Siamese Neural Network with Super-Resolution Convolutional Neural Network.

Chest X-ray (CXR) is becoming a useful method in the evaluation of coronavirus disease 19 (COVID-19). Despite the global spread of COVID-19, utilizing a computer-aided diagnosis approach for COVID-19 classification based on CXR images could significantly reduce the clinician burden. There is no doubt that low resolution, noise and irrelevant annotations in chest X-ray images are a major constraint to the performance of AI-based COVID-19 diagnosis. While a few studies have made huge progress, they underestimate these bottlenecks. In this study, we propose a super-resolution-based Siamese wavelet multi-resolution convolutional neural network called COVID-SRWCNN for COVID-19 classification using chest X-ray images. Concretely, we first reconstruct high-resolution (HR) counterparts from low-resolution (LR) CXR images in order to enhance the quality of the dataset for improved performance of our model by proposing a novel enhanced fast super-resolution convolutional neural network (EFSRCNN) to capture texture details in each given chest X-ray image. Exploiting a mutual learning approach, the HR images are passed to the proposed Siamese wavelet multi-resolution convolutional neural network to learn the high-level features for COVID-19 classification. We validate the proposed COVID-SRWCNN model on public-source datasets, achieving accuracy of 98.98%. Our screening technique achieves 98.96% AUC, 99.78% sensitivity, 98.53% precision, and 98.86% specificity. Owing to the fact that COVID-19 chest X-ray datasets are low in quality, experimental results show that our proposed algorithm obtains up-to-date performance that is useful for COVID-19 screening.

MCG-Net: End-to-End Fine-Grained Delineation and Diagnostic Classification of Cardiac Events From Magnetocardiographs.

In this paper, we propose an end-to-end deep learning architecture, referred as MCG-Net, integrating convolutional neural network (CNN) with transformer-based global context block for fine-grained delineation and diagnostic classification of four cardiac events from magnetocardiogram (MCG) data, namely Q-, R-, S- and T-waves. MCG-Net takes advantage of a multi-resolution CNN backbone as well as the state-of-the-art (SOTA) transformer encoders that facilitate global temporal feature aggregation. Besides the novel network architecture, we introduce a multi-task learning scheme to achieve simultaneous delineation and classification. Specifically, the problem of MCG delineation is formulated as multi-class heatmap regression. Meanwhile, a binary diagnostic classification label as well as a duration are jointly estimated for each cardiac event using features that are temporally aligned by event heatmaps. The framework is evaluated on a clinical MCG dataset, containing data collected from 270 subjects with cardiac anomalies and 108 control subjects. We designed and conducted a two-fold cross-validation study to validate the proposed method and to compare its performance with the SOTA methods. Experimental results demonstrated that our method outperformed counterparts on both event delineation and diagnostic classification tasks, achieving respectively an average ECG-F1 of 0.987 and an average Event-F1 of 0.975 for MCG delineation, and an average accuracy of 0.870, an average sensitivity of 0.732, an average specificity of 0.914 and an average AUC of 0.903 for diagnostic classification. Comprehensive ablation experiments are additionally performed to investigate effectiveness of different network components.

Real-time 3D motion estimation from undersampled MRI using multi-resolution neural networks.

Purpose: To enable real‐time adaptive magnetic resonance imaging–guided radiotherapy (MRIgRT) by obtaining time‐resolved three‐dimensional (3D) deformation vector fields (DVFs) with high spatiotemporal resolution and low latency (<500 ms). Theory and Methods: Respiratory‐resolved T1‐weighted 4D‐MRI of 27 patients with lung cancer were acquired using a golden‐angle radial stack‐of‐stars readout. A multiresolution convolutional neural network (CNN) called TEMPEST was trained on up to 32× retrospectively undersampled MRI of 17 patients, reconstructed with a nonuniform fast Fourier transform, to learn optical flow DVFs. TEMPEST was validated using 4D respiratory‐resolved MRI, a digital phantom, and a physical motion phantom. The time‐resolved motion estimation was evaluated in‐vivo using two volunteer scans, acquired on a hybrid MR‐scanner with integrated linear accelerator. Finally, we evaluated the model robustness on a publicly‐available four‐dimensional computed tomography (4D‐CT) dataset. Results: TEMPEST produced accurate DVFs on respiratory‐resolved MRI at 20‐fold acceleration, with the average end‐point‐error <2 mm, both on respiratory‐sorted MRI and on a digital phantom. TEMPEST estimated accurate time‐resolved DVFs on MRI of a motion phantom, with an error <2 mm at 28× undersampling. On two volunteer scans, TEMPEST accurately estimated motion compared to the self‐navigation signal using 50 spokes per dynamic (366× undersampling). At this undersampling factor, DVFs were estimated within 200 ms, including MRI acquisition. On fully sampled CT data, we achieved a target registration error of 1.87±1.65 mm without retraining the model. Conclusion: A CNN trained on undersampled MRI produced accurate 3D DVFs with high spatiotemporal resolution for MRIgRT.

Pancreatic cancer tumor analysis in CT images using patch-based multi-resolution convolutional neural network

In this paper we proposed a cascaded structure for extraction of the volumetric shape of the pancreas and tumor in adenocarcinoma patients. This process is a combination of an elastic atlas which is capable of fitting on 3D volumetric shape extracted from CT slices, a convolutional neural network with three forward paths to label the patches of images with coarse to fine resolutions using a multi-resolution structure, a region growing edge detection and finally a wavelet based multi-resolution rendering to extract the volumetric shape of the pancreas and tumor from planar CT slices. The atlas organs were weighted by geometrical parameters which were adjusted at global and organ levels. The classification of image patches was performed using a multiresolution convolutional neural network. The final segmentation of the pancreas and tumor was obtained by applying an edge detection method. A multiresolution wavelet is used for extraction of the volumetric shapes from segmented images. The subjects were 53–86 years of age with a mean and SD of 66 ± 10.24 years. K-fold cross validation is used for all the experiments. The results were compared with those of previously reported methods including Residual Network. The performance of the proposed algorithm was evaluated by Dice Similarity Coefficient (DSC), Jaccard index (JI), precision and recall yielded 89.67, 80.12, 91.37 and 93.63 for pancreas and 81.42, 68.66, 84.97 and 88.24 for tumor, respectively. The results of the proposed fully cascaded method overperformed all the other methods being compared.

Extraction of High-level and Low-level feature for classification of Image using Ridgelet and CNN based Image Classification

Remote Sensing image classification is an important research area for the recent time, because of its various application areas. Among the many available feature extraction methods, this paper uses the ridgelet based feature extraction method and those obtained features are combined with deep features obtained from the Convolutional Neural Network (CNN). Here, the Ridgelet’s are used to obtain the low-level features and CNN is used to obtain high-level feature. The system tries to construct the ridgelet filter for obtaining the low-level feature. The multi-resolution CNN is introduced using the concept of fusing high-level and low-level features via ridgelet and CNNs. Then, fused features are then classified and the output classified image is obtained. Experimental verifications are conducted on NWPU-RESISC45 dataset and the output results are provided to prove the best classification accuracies compared with the other proposed systems.

CT image segmentation for inflamed and fibrotic lungs using a multi-resolution convolutional neural network

The purpose of this study was to develop a fully-automated segmentation algorithm, robust to various density enhancing lung abnormalities, to facilitate rapid quantitative analysis of computed tomography images. A polymorphic training approach is proposed, in which both specifically labeled left and right lungs of humans with COPD, and nonspecifically labeled lungs of animals with acute lung injury, were incorporated into training a single neural network. The resulting network is intended for predicting left and right lung regions in humans with or without diffuse opacification and consolidation. Performance of the proposed lung segmentation algorithm was extensively evaluated on CT scans of subjects with COPD, confirmed COVID-19, lung cancer, and IPF, despite no labeled training data of the latter three diseases. Lobar segmentations were obtained using the left and right lung segmentation as input to the LobeNet algorithm. Regional lobar analysis was performed using hierarchical clustering to identify radiographic subtypes of COVID-19. The proposed lung segmentation algorithm was quantitatively evaluated using semi-automated and manually-corrected segmentations in 87 COVID-19 CT images, achieving an average symmetric surface distance of 0.495pm 0.309 mm and Dice coefficient of 0.985pm 0.011. Hierarchical clustering identified four radiographical phenotypes of COVID-19 based on lobar fractions of consolidated and poorly aerated tissue. Lower left and lower right lobes were consistently more afflicted with poor aeration and consolidation. However, the most severe cases demonstrated involvement of all lobes. The polymorphic training approach was able to accurately segment COVID-19 cases with diffuse consolidation without requiring COVID-19 cases for training.

An Attention-Based Deep Learning Approach for Sleep Stage Classification With Single-Channel EEG.

Automatic sleep stage mymargin classification is of great importance to measure sleep quality. In this paper, we propose a novel attention-based deep learning architecture called AttnSleep to classify sleep stages using single channel EEG signals. This architecture starts with the feature extraction module based on multi-resolution convolutional neural network (MRCNN) and adaptive feature recalibration (AFR). The MRCNN can extract low and high frequency features and the AFR is able to improve the quality of the extracted features by modeling the inter-dependencies between the features. The second module is the temporal context encoder (TCE) that leverages a multi-head attention mechanism to capture the temporal dependencies among the extracted features. Particularly, the multi-head attention deploys causal convolutions to model the temporal relations in the input features. We evaluate the performance of our proposed AttnSleep model using three public datasets. The results show that our AttnSleep outperforms state-of-the-art techniques in terms of different evaluation metrics. Our source codes, experimental data, and supplementary materials are available at https://github.com/emadeldeen24/AttnSleep.

Evolving Multi-Resolution Pooling CNN for Monaural Singing Voice Separation

Monaural singing voice separation (MSVS) is a challenging task and has been extensively studied. Deep neural networks (DNNs) are current state-of-the-art methods for MSVS. However, they are often designed manually, which is time-consuming and error-prone. They are also pre-defined, thus cannot adapt their structures to the training data. To address these issues, we first designed a multi-resolution convolutional neural network (CNN) for MSVS called multi-resolution pooling CNN (MRP-CNN), which uses various-sized pooling operators to extract multi-resolution features. We then introduced Neural Architecture Search (NAS) to extend the MRP-CNN to the evolving MRP-CNN (E-MRP-CNN) to automatically search for effective MRP-CNN structures using genetic algorithms optimized in terms of a single objective taking into account only separation performance and multiple objectives taking into account both separation performance and model complexity. The E-MRP-CNN using the multi-objective algorithm gives a set of Pareto-optimal solutions, each providing a trade-off between separation performance and model complexity. Evaluations on the MIR-1 K, DSD100, and MUSDB18 datasets were used to demonstrate the advantages of the E-MRP-CNN over several recent baselines.

HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images.

We propose HookNet, a semantic segmentation model for histopathology whole-slide images, which combines context and details via multiple branches of encoder-decoder convolutional neural networks. Concentric patches at multiple resolutions with different fields of view, feed different branches of HookNet, and intermediate representations are combined via a hooking mechanism. We describe a framework to design and train HookNet for achieving high-resolution semantic segmentation and introduce constraints to guarantee pixel-wise alignment in feature maps during hooking. We show the advantages of using HookNet in two histopathology image segmentation tasks where tissue type prediction accuracy strongly depends on contextual information, namely (1) multi-class tissue segmentation in breast cancer and, (2) segmentation of tertiary lymphoid structures and germinal centers in lung cancer. We show the superiority of HookNet when compared with single-resolution U-Net models working at different resolutions as well as with a recently published multi-resolution model for histopathology image segmentation. We have made HookNet publicly available by releasing the source code1 as well as in the form of web-based applications2,3 based on the grand-challenge.org platform.