Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation.

Three-dimensional convolutional neural networks (3D CNNs) are used efficiently in many computer vision applications. Most previous work in this area has concentrated only on designing and optimizing accelerators for 2D CNN, with few attempts made to accelerate 3D CNN on FPGA. We find accelerating 3D CNNs on FPGA to be challenge due to their high computational complexity and storage demands. More importantly, although the computation patterns of 2D and 3D CNNs are analogous, the conventional approaches adopted for accelerating 2D CNNs may be unfit for 3D CNN acceleration. In this paper, in order to accelerate 2D and 3D CNNs using a uniform framework, we propose a uniform template-based architecture that uses templates based on the Winograd algorithm to ensure fast development of 2D and 3D CNN accelerators. Furthermore, we also develop a uniform analytical model to facilitate efficient design space explorations of 2D and 3D CNN accelerators based on our architecture. Finally, we demonstrate the effectiveness of the template-based architecture by implementing accelerators for real-life 2D and 3D CNNs (VGG16 and C3D) on multiple FPGA platforms. On S2C VUS440, we achieve up to 1.13 TOPS and 1.11 TOPS under low resource utilization for VGG16 and C3D, respectively. End-to-end comparisons with CPU and GPU solutions demonstrate that our implementation of C3D achieves gains of up to 13x and 60x in performance and energy relative to a CPU solution, and a 6.4x energy efficiency gain over a GPU solution.

Prior studies demonstrated encouraging results in the application of convolution neural network models (CNN), one-dimensional (1D CNN), or three-dimensional (3D CNN) convolutional layers to decode the neural drive to muscles from highdensity surface electromyography (HD-sEMG) signals. However, the impact of the dimensionality (1D or 3D) of the convolutional layers on the performance of the deep CNN models using the same dataset has yet to be investigated. This study assesses the performance of 3D CNNs and 1D CNNs in extracting the neural drive as a cumulative spike train (CST) under various window sizes and step sizes that are critical parameters in decoding neural drives. Experimental HD-sEMG dataset sourced from the gastrocnemius medialis muscle of three participants, alongside the corresponding neural drive decoded using the convolution kernel compensation (CKC) algorithm, was employed to train and validate the 1D and 3D CNN models. We compared the F1 score and correlation coefficient between the CST from CKC and those from both 1D and 3D CNN models, revealing that 1D CNN performs more effectively with larger sliding window sizes (80 or 120 samples) with a peak F1 score of 0.84 and a correlation of 0.94. In contrast, 3D CNN achieves peak F1 score (0.83) and correlation (0.92) with smaller sliding window sizes (20 or 40 samples), indicating reduced latency in using 3D CNN to decode neural drives. Both models experience a performance decline as the step size increases. Furthermore, this research evaluates the computational cost of 1D and 3D CNN models, finding that the 3D CNN model requires significantly more computational resources (938G FLOPs) than the 1D CNN model (60G FLOPs). The results elucidate significant distinctions between CNN architectures and identify optimal parameters and model selection for precise and real-time neural drive decoding.

Three-dimensional convolutional neural networks (3D CNNs) have been widely applied to analyze brain tumour images (BT) to understand the disease's progress better. It is well-known that training 3D-CNN is computationally expensive and has the potential of overfitting due to the small sample size available in the medical imaging field. Here, we proposed a novel 2D-3D approach by converting a 2D brain image to a 3D fused image using a gradient of the image Learnable Weighted. By the 2D-to-3D conversion, the proposed model can easily forward the fused 3D image through a pre-trained 3D model while achieving better performance over different 3D baselines. We used VGG16 for feature extraction in the implementation as it outperformed other 3D CNN backbones. We further showed that the weights of the slices are location-dependent, and the model performance relies on the 3D-to-2D fusion view, with the best outcomes from the coronal view. With the new approach, we increased the accuracy to 0.88, compared with conventional 3D CNNs, for classifying brain tumour images. The novel 2D-3D model may have profound implications for future timely BT diagnosis in clinical settings.

3D Convolutional Neural Networks (3D CNNs) can outperform 2D CNNs on several tasks, including action recognition, video captioning, abnormal event detection, and medical image interpretation. Compared to 2D CNNs, 3D CNNs have a larger number of parameters and higher computational complexity. For this reason, researchers have focused on designing efficient 3D CNN architectures and hardware accelerators. The purpose of this survey is to serve as a guide to recent work on 3D CNNs for action recognition with a focus on FPGA-based accelerators and hardware-aware algorithmic optimizations. We provide an overview of the state-of-the-art in 3D CNN architectures as well as the action recognition datasets used for training and testing the architectures. A performance comparison of 2D versus 3D CNNs on two datasets (Sports-1M and UCF101) is included. We explore the designs of FPGA-based accelerators and compare them in terms of achieved throughput, resource utilization, and power. We survey current methods of optimization for 3D CNN architectures, which are meant to reduce the number of parameters and the memory requirements and to facilitate their deployment on FPGAs. Finally, we highlight current challenges and potential areas of improvement in acceleration of 3D CNNs on FPGA platforms.

Human beings learn efficiently from limited information, whereas supervised machine learning requires vast data. One of the reasons is that humans have acquired prior knowledge about the world, which provides the person with a head-start to learn quickly. In machine learning, we are also faced with the problem of having limited data in the medical domain. Annotated medical data is scarce and costly to acquire. We address data insufficiency in a specific domain for medical imaging, such as brain imaging, by relying on the strength of unsupervised deep learning. This thesis applies brain tumour and lesion segmentation obtained from various brain imaging sources. It is a crucial task in neuroimaging analysis, but supervised methods require large, annotated datasets, which are often limited in acquisition. This work established several approaches to address this limitation. Firstly, we introduced a novel Hybrid UNet Transformer (HUT) network that combines the UNet and the Transformer models. The Convolutional Neural Network (CNN) in the UNet has distinct inductive biases such as local correlation, spatial relationship, and invariance to translation. The Transformer, on the other hand, was originally applied to natural language processing (NLP). It is primarily constructed for sequence-to-sequence tasks. It uses self-attention mechanisms to correlate the importance of the embeddings in the sequence. In addition, it is designed to handle long-range dependencies between the embeddings. In the Vision Transformer, the images are sliced into smaller patches represented as the embeddings via an encoder. The ViT is not data efficient and, therefore, relies on the efficacy of the UNet for more efficient training. Both architectures in the HUT system run in tandem. The Transformer produces the output of different resolutions and fuses it with the UNet at part of its decoder stage. We established a data-efficient system that helps with training when the dataset is smaller. Experimental results on benchmark datasets demonstrate the superiority of HUT over the state-of-the-art methods, which achieves a 0.96% improvement in the Dice score and a 4.1% reduction in the Hausdorff Distance score on the BraTS20 dataset for the brain tumour segmentation. It attains a 4.84% improvement in Dice score and 40.7% reduction in Hausdorff Distance score on the ATLAS dataset for the brain lesion segmentation. HUT gains a 3.3% improvement in Dice score and a 12.5% reduction in Hausdorff Distance score on the ISLES2018 dataset. Secondly, we extend the HUT system with a novel Noise-induced Self-Supervised (NSS) method, also known as an unsupervised approach, introduced in this work using a noise anchor and assumes the model is an energy spectral density model. The method is applied in the pre-training process of the HUT system to address the challenge of small datasets. With the proposed method, the pre-trained network achieves better separation of clusters, as shown in a simple MNIST experiment. This approach improves the network's ability to learn from limited data and increases the system's robustness by preventing overfitting during the training when the data is scarce. The pre-trained network can be deployed on downstream tasks like brain lesion segmentation. We experimented on the smaller ISLES2018 dataset to demonstrate the method's effectiveness. The self-supervised HUT-NSS version outperforms its supervised counterpart and state-of-the-art networks, even with limited annotated data, and achieves a 2.4% improvement in Dice score on average. It produces a 7.2% improvement in Dice score and 28.1% reduction in Hausdorff Distance score on the ISLES2018 dataset with 50% annotated data. The method acquires a 7.87% improvement in Dice score with 10% annotated data and 5.34% improvement in Dice score with 1% annotated data. To put it concisely, the work paves the way for improving learning efficiency on brain image segmentation when the dataset is limited.

Three-dimensional convolutional neural networks (3D CNNs) have demonstrated their outstanding classification accuracy for human action recognition (HAR). However, the large number of computations and parameters in 3D CNNs limits their deployability in real-life applications. To address this challenge, this paper adopts an algorithm-hardware co-design method by proposing an efficient 3D CNN building unit called 3D-1 bottleneck residual block (3D-1 BRB) at the algorithm level, and a corresponding FPGA-based hardware architecture called F-E3D at the hardware level. Based on 3D-1 BRB, a novel 3D CNN model called E3DNet is developed, which achieves nearly 37 times reduction in model size and 5% improvement in accuracy compared to standard 3D CNNs on the UCF101 dataset. Together with several hardware optimizations, including 3D fused BRB, online blocking and kernel reuse, the proposed F-E3D is nearly 13 times faster than a previous FPGA design for 3D CNNs, with performance and accuracy comparable to other state-of-the-art 3D CNN models on GPU platforms while requiring only 7% of their energy consumption.

Brain tissue segmentation from multimodal MRI is a key building block of many neuroimaging analysis pipelines. Established tissue segmentation approaches have, however, not been developed to cope with large anatomical changes resulting from pathology, such as white matter lesions or tumours, and often fail in these cases. In the meantime, with the advent of deep neural networks (DNNs), segmentation of brain lesions has matured significantly. However, few existing approaches allow for the joint segmentation of normal tissue and brain lesions. Developing a DNN for such a joint task is currently hampered by the fact that annotated datasets typically address only one specific task and rely on task-specific imaging protocols including a task-specific set of imaging modalities. In this work, we propose a novel approach to build a joint tissue and lesion segmentation model from aggregated task-specific hetero-modal domain-shifted and partially-annotated datasets. Starting from a variational formulation of the joint problem, we show how the expected risk can be decomposed and optimised empirically. We exploit an upper bound of the risk to deal with heterogeneous imaging modalities across datasets. To deal with potential domain shift, we integrated and tested three conventional techniques based on data augmentation, adversarial learning and pseudo-healthy generation. For each individual task, our joint approach reaches comparable performance to task-specific and fully-supervised models. The proposed framework is assessed on two different types of brain lesions: White matter lesions and gliomas. In the latter case, lacking a joint ground-truth for quantitative assessment purposes, we propose and use a novel clinically-relevant qualitative assessment methodology.

Abstract. This paper presents an automated workflow for pixel-wise land cover (LC) classification from multispectral airborne laser scanning (ALS) data using deep learning methods. It mainly contains three procedures: data pre-processing, land cover classification, and accuracy assessment. First, a total of nine raster images with different information were generated from the pre-processed point clouds. These images were assembled into six input data combinations. Meanwhile, the labelled dataset was created using the orthophotos as the ground truth. Also, three deep learning networks were established. Then, each input data combination was used to train and validate each network, which developed eighteen LC classification models with different parameters to predict LC types for pixels. Finally, accuracy assessments and comparisons were done for the eighteen classification results to determine an optimal scheme. The proposed method was tested on six input datasets with three deep learning classification networks (i.e., 1D CNN, 2D CNN, and 3D CNN). The highest overall classification accuracy of 97.2% has been achieved using the proposed 3D CNN. The overall accuracy (OA) of the 2D and 3D CNNs was, on average, 8.4% higher than that of the 1D CNN. Although the OA of the 2D CNN was at most 0.3% lower than that of the 3D CNN, the runtime of the 3D CNN was five times longer than the 2D CNN. Thus, the 2D CNN was the best choice for the multispectral ALS LC classification when considering efficiency. The results demonstrated the proposed methods can successfully classify land covers from multispectral ALS data.

RFDCR: Automated brain lesion segmentation using cascaded random forests with dense conditional random fields

Three-dimensional convolutional neural networks (3D CNNs) have gained popularity in many complicated computer vision applications. Many customized accelerators based on FPGAs are proposed for 2D CNNs, while very few are for 3D CNNs. Three-D CNNs are far more computationally intensive and the design space for 3D CNN acceleration has been further expanded since one more dimension is introduced, making it a big challenge to accelerate 3D CNNs on FPGAs. Motivated by the finding that the computation patterns of 2D and 3D CNNs are very similar, we propose a uniform architecture design for accelerating both 2D and 3D CNNs in this paper. The uniform architecture is based on the idea of mapping convolutions to matrix multiplications. A customized mapping module is developed to generate the feature matrix tilings with no need to store the entire enlarged feature matrix on-chip or off-chip, a splitting strategy is adopted to reconstruct a convolutional layer to adapt to the on-chip memory capacity, and a 2D multiply-and-accumulate (MAC) array is adopted to compute matrix multiplications efficiently. For demonstration, we implement an accelerator prototype with a high-level synthesis (HLS) methodology on a Xilinx VC709 board and test the accelerator on three typical CNN models: AlexNet, VGG16, and C3D. Experimental results show that the accelerator achieves state-of-the-art throughput performance on both 2D and 3D CNNs, with much better energy efficiency than the CPU and GPU.

The advent of three-dimensional convolutional neural networks (3D CNNs) has revolutionized the detection and analysis of COVID-19 cases. As imaging technologies have advanced, 3D CNNs have emerged as a powerful tool for segmenting and classifying COVID-19 in medical images. These networks have demonstrated both high accuracy and rapid detection capabilities, making them crucial for effective COVID-19 diagnostics. This study offers a thorough review of various 3D CNN algorithms, evaluating their efficacy in segmenting and classifying COVID-19 across a range of medical imaging modalities. This review systematically examines recent advancements in 3D CNN methodologies. The process involved a comprehensive screening of abstracts and titles to ensure relevance, followed by a meticulous selection and analysis of research papers from academic repositories. The study evaluates these papers based on specific criteria and provides detailed insights into the network architectures and algorithms used for COVID-19 detection. The review reveals significant trends in the use of 3D CNNs for COVID-19 segmentation and classification. It highlights key findings, including the diverse range of networks employed for COVID-19 detection compared to other diseases, which predominantly utilize encoder/decoder frameworks. The study provides an in-depth analysis of these methods, discussing their strengths, limitations, and potential areas for future research. The study reviewed a total of 60 papers published across various repositories, including Springer and Elsevier. The insights from this study have implications for clinical diagnosis and treatment strategies. Despite some limitations, the accuracy and efficiency of 3D CNN algorithms underscore their potential for advancing medical image segmentation and classification. The findings suggest that 3D CNNs could significantly enhance the detection and management of COVID-19, contributing to improved healthcare outcomes.

High-density surface electromyography (HD-sEMG) and deep learning technology are becoming increasingly used in gesture recognition. Based on electrode grid data, information can be extracted in the form of images that are generated with instant values of multi-channel sEMG signals. In previous studies, image-based, two-dimensional convolutional neural networks (2D CNNs) have been applied in order to recognize patterns in the electrical activity of muscles from an instantaneous image. However, 2D CNNs with 2D kernels are unable to handle a sequence of images that carry information concerning how the instantaneous image evolves with time. This paper presents a 3D CNN with 3D kernels to capture both spatial and temporal structures from sequential sEMG images and investigates its performance on HD-sEMG-based gesture recognition in comparison to the 2D CNN. Extensive experiments were carried out on two benchmark datasets (i.e., CapgMyo DB-a and CSL-HDEMG). The results show that, where the same network architecture is used, 3D CNN can achieve a better performance than 2D CNN, especially for CSL-HDEMG, which contains the dynamic part of finger movement. For CapgMyo DB-a, the accuracy of 3D CNN was 1% higher than 2D CNN when the recognition window length was equal to 40 ms, and was 1.5% higher when equal to 150 ms. For CSL-HDEMG, the accuracies of 3D CNN were 15.3% and 18.6% higher than 2D CNN when the window length was equal to 40 ms and 150 ms, respectively. Furthermore, 3D CNN achieves a competitive performance in comparison to the baseline methods.

Despite considerable enhances in recognizing hand gestures from still images, there are still many challenges in the classification of hand gestures in videos. The latter comes with more challenges, including higher computational complexity and arduous task of representing temporal features. Hand movement dynamics, represented by temporal features, have to be extracted by analyzing the total frames of a video. So far, both 2D and 3D convolutional neural networks have been used to manipulate the temporal dynamics of the video frames. 3D CNNs can extract the changes in the consecutive frames and tend to be more suitable for the video classification task, however, they usually need more time. On the other hand, by using techniques like tiling it is possible to aggregate all the frames in a single matrix and preserve the temporal and spatial features. This way, using 2D CNNs, which are inherently simpler than 3D CNNs can be used to classify the video instances. In this paper, we compared the application of 2D and 3D CNNs for representing temporal features and classifying hand gesture sequences. Additionally, providing a two-stage two-stream architecture, we efficiently combined color and depth modalities and 2D and 3D CNN predictions. The effect of different types of augmentation techniques is also investigated. Our results confirm that appropriate usage of 2D CNNs outperforms a 3D CNN implementation in this task.

MR images (MRIs) accurate segmentation of brain lesions is important for improving cancer diagnosis, surgical planning, and prediction of outcome. However, manual and accurate segmentation of brain lesions from 3D MRIs is highly expensive, time-consuming, and prone to user biases. We present an efficient yet conceptually simple brain segmentation network (referred as Brain SegNet), which is a 3D residual framework for automatic voxel-wise segmentation of brain lesion. Our model is able to directly predict dense voxel segmentation of brain tumor or ischemic stroke regions in 3D brain MRIs. The proposed 3D segmentation network can run at about 0.5s per MRIs - about 50 times faster than previous approaches Med Image Anal 43: 98–111, 2018, Med Image Anal 36:61–78, 2017. Our model is evaluated on the BRATS 2015 benchmark for brain tumor segmentation, where it obtains state-of-the-art results, by surpassing recently published results reported in Med Image Anal 43: 98–111, 2018, Med Image Anal 36:61–78, 2017. We further applied the proposed Brain SegNet for ischemic stroke lesion outcome prediction, with impressive results achieved on the Ischemic Stroke Lesion Segmentation (ISLES) 2017 database.

Hyperspectral Image (HSI) classification started with the machine learning classifiers like SVM (Support Vector Machine). It transitioned towards deep neural networks, which perform better than their classifier counterparts.CNN (Convolution Neural Networks) approaches are widely found in the current literature, including architectures like 2D CNN,3D CNN, and various other models using a combination of multiple CNN architectures. The correctness of HSI classification depends on both spectral and spatial features. 2D CNN cannot capture spatial features six, resulting in poor performance. On the other hand, 3D CNNs are computationally heavy and are not widely deployed. To overcome these bottlenecks, hybrid models were proposed where both 3D and 2D CNN are used together to exploit both spatial and spectral features. These models perform considerably well and are more computationally efficient when compared to 3D CNN. This paper proposes a custom hybrid model composed of 3D CNN with Max-Pooling coupled with depth-wise 2D CNN for reduced computational cost. It has considerably fewer learning parameters compared to other proposed CNN models. The performance of the proposed model was evaluated on Indian Pines, University of Pavia, and Salinas Scenes remote sensing datasets and results of the same were compared with state of the art and various other CNN models to validate the performance of the proposed model.