Large-scale Convolutional Neural Networks Research Articles

LRFE: A NOVEL LOCAL RESPONSE FEATURE ELIMINATION PROCESS FOR IDENTIFICATION OF LUNG CANCER CELLS

One of the main causes of cancer-related mortality across the globe is lung cancer. Early-stage lung cancer frequently exhibits no symptoms, which delays diagnosis until the illness has progressed. Before symptoms manifest, screening and early detection techniques can aid in the early diagnosis and treatment of lung cancer. Many olden research papers have implemented image processing and few latest papers have implemented computer vision techniques to detect lung cancer. Particularly when dealing with minor or subtle anomalies, image processing algorithms may not be able to detect lung cancer lesions with sufficient sensitivity and specificity. It is still difficult to increase the detection algorithms' accuracy and dependability, especially when dealing with early-stage lesions or situations where attributes overlap. It takes a lot of processing power, such as high-performance GPUs and enormous memory capacities, to train deep learning models, particularly large-scale convolutional neural networks (CNNs). In this proposed research, the model pre-processes the images using the ostu and sober filter mechanisms because Otsu's approach adjusts to the features of the input image, including noise, contrast, and lighting fluctuations. It is capable of handling images with varying dynamic ranges and intensity distributions without depending on pre-established threshold settings. When it comes to image noise, the Sobel filter is more resilient than other edge detection methods. It produces clearer edge maps and fewer false detections by determining the gradient magnitude, which amplifies edge information while suppressing noise. The features are extracted using the tuned AlexNet pre-trained model, in AlexNet there is a layer known as “Layer-wise Relevance Propagation”. By giving each pixel or feature in the input image a relevance score, the LRP layer offers fine-grained feature attribution. This makes it possible to analyze in great depth which particular elements or areas of the input image are most important for the network to forecast, which helps to clarify the underlying patterns that the network has learned. At last, the extracted features are further reduced using the enhanced feature elimination method. By iteratively selecting subsets of features based on their importance, RFE helps to identify the most relevant features for the classification task. Integrating RFE with SVM classifiers can lead to improved model performance by focusing on the subset of features that are most discriminative and informative for the classification problem.

FEDAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning.

Federated distillation (FD) is a popular novel algorithmic paradigm for Federated learning (FL), which achieves training performance competitive to prior parameter averaging-based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work, we propose FEDAUX, an extension to FD, which, under the same set of assumptions, drastically improves the performance by deriving maximum utility from the unlabeled auxiliary data. FEDAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a suitable model initialization for the distributed training. Second, ε, δ)-differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks (CNNs) and transformer models demonstrate that our proposed method achieves remarkable performance improvements over state-of-the-art FL methods, without adding appreciable computation, communication, or privacy cost. For instance, when training ResNet8 on non-independent identically distributed (i.i.d.) subsets of CIFAR10, FEDAUX raises the maximum achieved validation accuracy from 30.4% to 78.1%, further closing the gap to centralized training performance. Code is available at https://github.com/fedl-repo/fedaux.

A surrogate-assisted highly cooperative coevolutionary algorithm for hyperparameter optimization in deep convolutional neural networks

Hyperparameter optimization in convolutional neural networks (CNNs) plays a vital role in ensuring the effectiveness of the models. However, with the depth of the existing CNN expanding, this task becomes very challenging due to the high-dimensional and computationally expensive characteristics. Given these difficulties, this study proposes a surrogate-assisted highly cooperative hyperparameter optimization (SHCHO) algorithm for large-scale chain-styled CNNs. SHCHO tackles the original hyperparameter optimization problem by cooperatively optimizing its subproblems. Specifically, it first decomposes the whole CNN into several overlapping sub-CNNs, which conforms to the inherent overlapping interaction structure among hyperparameters and significantly reduces the search space. The resulting hyperparameter optimization subproblems on these sub-CNNs are then coevolved collaboratively and competitively, facilitating a proper hyperparameter configuration for the whole CNN. Besides, SHCHO also employs the computationally efficient surrogate technique to assist in each subproblem optimization. As a result, the expensive computational cost can be greatly reduced. Extensive experimental results on three widely-used image classification datasets indicate that SHCHO shows competitive performance when compared with several state-of-the-art algorithms in terms of both hyperparameter superiority and computational cost.

Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy.

Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs' generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi's entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

Implementation of Fully-Pipelined CNN Inference Accelerator on FPGA and HBM2 Platform

Many deep convolutional neural network (CNN) inference accelerators on the field-programmable gate array (FPGA) platform have been widely adopted due to their low power consumption and high performance. In this paper, we develop the following to improve performance and power efficiency. First, we use a high bandwidth memory (HBM) to expand the bandwidth of data transmission between the off-chip memory and the accelerator. Second, a fully-pipelined manner, which consists of pipelined inter-layer computation and a pipelined computation engine, is implemented to decrease idle time among layers. Third, a multi-core architecture with shared-dual buffers is designed to reduce off-chip memory access and maximize the throughput. We designed the proposed accelerator on the Xilinx Alveo U280 platform with in-depth Verilog HDL instead of high-level synthesis as the previous works and explored the VGG-16 model to verify the system during our experiment. With a similar accelerator architecture, the experimental results demonstrate that the memory bandwidth of HBM is 13.2× better than DDR4. Compared with other accelerators in terms of throughput, our accelerator is 1.9×/1.65×/11.9× better than FPGA+HBM2 based/low batch size (4) GPGPU/low batch size (4) CPU. Compared with the previous DDR+FPGA/DDR+GPGPU/DDR+CPU based accelerators in terms of power efficiency, our proposed system provides 1.4-1.7×/1.7-12.6×/6.6-37.1× improvement with the large-scale CNN model.

A walk in the black-box: 3D visualization of large neural networks in virtual reality

Within the last decade Deep Learning has become a tool for solving challenging problems like image recognition. Still, Convolutional Neural Networks (CNNs) are considered black-boxes, which are difficult to understand by humans. Hence, there is an urge to visualize CNN architectures, their internal processes and what they actually learn. Previously, virtual realityhas been successfully applied to display small CNNs in immersive 3D environments. In this work, we address the problem how to feasibly render large-scale CNNs, thereby enabling the visualization of popular architectures with ten thousands of feature maps and branches in the computational graph in 3D. Our software ”DeepVisionVR” enables the user to freely walk through the layered network, pick up and place images, move/scale layers for better readability, perform feature visualization and export the results. We also provide a novel Pytorch module to dynamically link PyTorch with Unity, which gives developers and researchers a convenient interface to visualize their own architectures. The visualization is directly created from the PyTorch class that defines the Pytorch model used for training and testing. This approach allows full access to the network’s internals and direct control over what exactly is visualized. In a use-case study, we apply the module to analyze models with different generalization abilities in order to understand how networks memorize images. We train two recent architectures, CovidResNet and CovidDenseNet on the Caltech101 and the SARS-CoV-2 datasets and find that bad generalization is driven by high-frequency features and the susceptibility to specific pixel arrangements, leading to implications for the practical application of CNNs. The code is available on Github https://github.com/Criscraft/DeepVisionVR.

Android Malware Detection Technology Based on Lightweight Convolutional Neural Networks

With the rapid development of Android, a major mobile Internet platform, Android malware attacks have become the number one threat to mobile Internet security. Traditional malware detection methods have low precision and greater time complexity. At present, image detection methods based on deep learning are used in malware detection. However, most of these methods are based on the largescale convolutional neural network model (such as VGG16). The computation and weight files of these models are very large, so they are not suitable for mobile Internet platforms with limited computation. A novel detection method based on a lightweight convolutional neural network is presented in this study. It transforms Android malware classes.dex, Androidmanifest.xml, and resource.arsc into RGB images and uses the lightweight convolutional neural network to extract the features of RGB images automatically. The experimental results of this study indicate that the method performs well in terms of precision and speed of detection.

A Flexible and Efficient FPGA Accelerator for Various Large-Scale and Lightweight CNNs

To enable efficient deployment of convolutional neural networks (CNNs) on embedded platforms for different computer vision applications, several convolution variants have been introduced, such as depthwise convolution (DWCV), transposed convolution (TPCV), and dilated convolution (DLCV). To address the utilization degradation issue occurred in a general convolution engine for these emerging operators, a highly flexible and reconfigurable hardware accelerator is proposed to efficiently support various CNN-based vision tasks. Firstly, to avoid workload imbalance of TPCV, a zero transfer and skipping (ZTS) method is proposed to reorganize the computation process. To eliminate the redundant zero calculations of TPCV and DLCV, a sparsity-alike processing (SAP) method is proposed based on weight-oriented dataflow. Secondly, the DWCV or pooling layers are configured to be directly executed after standard convolutions without external memory accesses. Furthermore, a programmable execution schedule is introduced to gain better flexibility. Finally, the proposed accelerator is evaluated on Intel Arria 10 SoC FPGA. Experimental results show state-of-the-art performance on both large-scale and lightweight CNNs for image segmentation or classification. Specifically, the accelerator can achieve a processing speed up to 339.9 FPS and computational efficiency up to 0.58 GOPS/DSP, which is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.3\times $ </tex-math></inline-formula> better than the prior art evaluated on the same network.

A Scatter-and-Gather Spiking Convolutional Neural Network on a Reconfigurable Neuromorphic Hardware.

Artificial neural networks (ANNs), like convolutional neural networks (CNNs), have achieved the state-of-the-art results for many machine learning tasks. However, inference with large-scale full-precision CNNs must cause substantial energy consumption and memory occupation, which seriously hinders their deployment on mobile and embedded systems. Highly inspired from biological brain, spiking neural networks (SNNs) are emerging as new solutions because of natural superiority in brain-like learning and great energy efficiency with event-driven communication and computation. Nevertheless, training a deep SNN remains a main challenge and there is usually a big accuracy gap between ANNs and SNNs. In this paper, we introduce a hardware-friendly conversion algorithm called “scatter-and-gather” to convert quantized ANNs to lossless SNNs, where neurons are connected with ternary {−1,0,1} synaptic weights. Each spiking neuron is stateless and more like original McCulloch and Pitts model, because it fires at most one spike and need be reset at each time step. Furthermore, we develop an incremental mapping framework to demonstrate efficient network deployments on a reconfigurable neuromorphic chip. Experimental results show our spiking LeNet on MNIST and VGG-Net on CIFAR-10 datasetobtain 99.37% and 91.91% classification accuracy, respectively. Besides, the presented mapping algorithm manages network deployment on our neuromorphic chip with maximum resource efficiency and excellent flexibility. Our four-spike LeNet and VGG-Net on chip can achieve respective real-time inference speed of 0.38 ms/image, 3.24 ms/image, and an average power consumption of 0.28 mJ/image and 2.3 mJ/image at 0.9 V, 252 MHz, which is nearly two orders of magnitude more efficient than traditional GPUs.

QTTNet: Quantized tensor train neural networks for 3D object and video recognition

Relying on the rapidly increasing capacity of computing clusters and hardware, convolutional neural networks (CNNs) have been successfully applied in various fields and achieved state-of-the-art results. Despite these exciting developments, the huge memory cost is still involved in training and inferring a large-scale CNN model and makes it hard to be widely used in resource-limited portable devices. To address this problem, we establish a training framework for three-dimensional convolutional neural networks (3DCNNs) named QTTNet that combines tensor train (TT) decomposition and data quantization together for further shrinking the model size and decreasing the memory and time cost. Through this framework, we can fully explore the superiority of TT in reducing the number of trainable parameters and the advantage of quantization in decreasing the bit-width of data, particularly compressing 3DCNN model greatly with little accuracy degradation. In addition, due to the low bit quantization to all parameters during the inference process including TT-cores, activations, and batch normalizations, the proposed method naturally takes advantage in memory and time cost. Experimental results of compressing 3DCNNs for 3D object and video recognition on ModelNet40, UCF11, and UCF50 datasets verify the effectiveness of the proposed method. The best compression ratio we have obtained is up to nearly 180× with competitive performance compared with other state-of-the-art researches. Moreover, the total bytes of our QTTNet models on ModelNet40 and UCF11 datasets can be 1000× lower than some typical practices such as MVCNN.

Real-time traffic sign detection network using DS-DetNet and lite fusion FPN

Traffic sign detection (TSD) using convolutional neural networks (CNN) is promising and intriguing for autonomous driving. Especially, with sophisticated large-scale CNN models, TSD can be performed with high accuracy. However, the conventional CNN models suffer the drawbacks of being time-consuming and resource-hungry, which limit their application and deployments in various platforms of limited resources. In this paper, we propose a novel real-time traffic sign detection system with a lightweight backbone network named Depth Separable DetNet (DS-DetNet) and a lite fusion feature pyramid network (LFFPN) for efficient feature fusion. The new model can achieve a performance trade-off between speed and accuracy using a depthwise separable bottleneck block, a lite fusion module, and an improved SSD detection front-end. The testing results on the MS COCO and the GTSDB datasets reveal that 23.1% mAP with 6.39 M parameters and only 1.08B FLOPs on MSCOCO, 81.35% mAP with 5.78 M parameters on GTSDB. With our model, the run speed is 61 frames per second (fps) on GTX 1080ti, 12 fps on Nvidia Jetson Nano and 16 fps on Nvidia Jetson Xavier NX.

Efficient Scheduling of Irregular Network Structures on CNN Accelerators

The state-of-the-art convolutional neural network (CNN) structures present growing irregularity in the sense of layer connections, which derives from the innovative manual designs and the recently proposed neural architecture searching approaches. Such irregular structures improve recognition accuracy, but also bring challenges for hardware deployment, especially on CNN accelerators with regular architectures: 1) the complicated data dependency makes it nontrivial to decide the data reuse strategy between layers and 2) since the execution order of each network is not unique, the choice of layer scheduling, memory allocating, and loop tiling strategies greatly impact the hardware performance. These challenges cannot be solved by the existing CNN schedulers, which mainly focuses on the dataflow of a single layer. In this work, we propose a comprehensive framework to analyze and solve the mapping of an arbitrarily connected CNN network to specific hardware accelerators. We propose: 1) a dynamic programming and node-clustering-based DAG partitioning approach to efficiently exploit interlayer data reuse and 2) a subgraph scheduling and on-chip memory allocating strategy to find the optimal execution order. With the modeling of CNN accelerators, we also propose a loop tiling approach for fused layers. An automated framework is established to generate binary machine codes from original CNN models produced by mainstream deep learning frameworks, which can process large-scale CNNs with more than 1000 layers in only a few minutes. Experiments based on state-of-the-art accelerators (e.g., NVDLA) show that our techniques greatly reduce the external data transfer of interlayer dependencies and bring significant performance improvement over existing approaches.

An Efficient Task Assignment Framework to Accelerate DPU-Based Convolutional Neural Network Inference on FPGAs

Field Programmable Gate Array (FPGA) has become an efficient accelerator for convolutional neural network (CNN) inference due to its high performance and flexibility. To further improve the performance of CNN inference on FPGAs, an Intellectual Property core (IP core) called Deep Learning Processor Unit (DPU) is released by Xilinx. Unlike previous FPGA-based hardware designs focusing on specific functions or CNNs, the DPU IP supports ample basic functions of deep learning, and the developers can take advantage of DPUs to accelerate CNN inference conveniently. In DPU-based CNN acceleration platform, an encapsulated scheduler plays a crucial role in task scheduling between heterogeneous ARM and multiple DPUs. However, the current scheduler is unsatisfactory because its low schedule efficiency. This paper thus presents a high performance task assignment framework built upon Xilinx hybrid CPU-FPGA MPSoC devices. We first evaluate the main causes of low schedule efficiency problem. Then, we explore the scheduler rules and improve shedule efficiency through purposeful observations and analysis. Finally, we integrate our optimizations, and propose an efficient task assignment framework to maximize performance on DPU-based CNN acceleration platform. Experimental results on Xilinx Zynq UltraScale+ MPSoC zcu104 show that our efficient task assignment framework significantly boosts schedule efficiency for small-scale CNNs (from 36% to 70%), medium-scale CNNs (from 65% to 95%), and large-scale CNNs (from 77% to 99%) compared with original schedule strategy.

An Energy-Efficient Implementation of Group Pruned CNNs on FPGA

In recent years, convolutional neural network (CNN)-based artificial intelligence algorithms have been widely applied to object recognition and image classification tasks. However, the high performance of convolutional neural networks comes at the cost of high-intensity computing and enormous numbers of parameters, which pose substantial challenges to terminal implementations. An end-to-end FPGA-based accelerator is proposed in this work that efficiently processes fine-grained pruned CNNs. A group pruning algorithm with group sparse regularization (GSR) is introduced to solve internal buffer misalignments and load imbalances of the accelerator after fine-grained pruning. A mathematical model of accelerator access and transmission is established to explore the optimal design scale and calculation mode. The accelerator is optimized by designing sparse processing elements and by scheduling the on- and off-chip buffers. The proposed approach reduces the computation of a state-of-the-art large-scale CNN, VGG16, by 86.9% with an accuracy loss on CIFAR-10 of only 0.48%. The accelerator achieves 188.41 GOPS at 100 MHz and consumes 8.15 W when implemented on a Xilinx VC707, making it more energy-efficient than previous approaches.

Energy-Efficient Architecture for CNNs Inference on Heterogeneous FPGA

Due to the huge requirements in terms of both computational and memory capabilities, implementing energy-efficient and high-performance Convolutional Neural Networks (CNNs) by exploiting embedded systems still represents a major challenge for hardware designers. This paper presents the complete design of a heterogeneous embedded system realized by using a Field-Programmable Gate Array Systems-on-Chip (SoC) and suitable to accelerate the inference of Convolutional Neural Networks in power-constrained environments, such as those related to IoT applications. The proposed architecture is validated through its exploitation in large-scale CNNs on low-cost devices. The prototype realized on a Zynq XC7Z045 device achieves a power efficiency up to 135 Gops/W. When the VGG-16 model is inferred, a frame rate up to 11.8 fps is reached.

Detection of Pulmonary Tuberculosis Manifestation in Chest X-Rays Using Different Convolutional Neural Network (CNN) Models

Tuberculosis (TB) is airborne infectious disease which has claimed many lives than any other infectious disease. Chest X-rays (CXRs) are often used in recognizing TB manifestation site in chest. Lately, CXRs are taken in digital formats, which has made a huge impact in rapid diagnosis using automated systems in medical field. In our current work, four simple Convolutional Neural Networks (CNN) models such as VGG-16, VGG-19, RestNet50, and GoogLenet are implemented in identification of TB manifested CXRs. Two public TB image datasets were utilized to conduct this research. This study was carried out to explore the limit of accuracies and AUCs acquired by simple and small-scale CNN with complex and large-scale CNN models. The results achieved from this work are compared with results of two previous studies. The results indicate that our proposed VGG-16 model has gained highest score overall compared to the models from other two previous studies.

High-Performance FPGA-Based CNN Accelerator With Block-Floating-Point Arithmetic

Convolutional neural networks (CNNs) are widely used and have achieved great success in computer vision and speech processing applications. However, deploying the large-scale CNN model in the embedded system is subject to the constraints of computation and memory. An optimized block-floating-point (BFP) arithmetic is adopted in our accelerator for efficient inference of deep neural networks in this paper. The feature maps and model parameters are represented in 16-bit and 8-bit formats, respectively, in the off-chip memory, which can reduce memory and off-chip bandwidth requirements by 50% and 75% compared to the 32-bit FP counterpart. The proposed 8-bit BFP arithmetic with optimized rounding and shifting-operation-based quantization schemes improves the energy and hardware efficiency by three times. One CNN model can be deployed in our accelerator without retraining at the cost of an accuracy loss of not more than 0.12%. The proposed reconfigurable accelerator with three parallelism dimensions, ping-pong off-chip DDR3 memory access, and an optimized on-chip buffer group is implemented on the Xilinx VC709 evaluation board. Our accelerator achieves a performance of 760.83 GOP/s and 82.88 GOP/s/W under a 200-MHz working frequency, significantly outperforming previous accelerators.

A Bi-layered Parallel Training Architecture for Large-Scale Convolutional Neural Networks

Benefitting from large-scale training datasets and the complex training network, Convolutional Neural Networks (CNNs) are widely applied in various fields with high accuracy. However, the training process of CNNs is very time-consuming, where large amounts of training samples and iterative operations are required to obtain high-quality weight parameters. In this paper, we focus on the time-consuming training process of large-scale CNNs and propose a Bi-layered Parallel Training (BPT-CNN) architecture in distributed computing environments. BPT-CNN consists of two main components: (a) an outer-layer parallel training for multiple CNN subnetworks on separate data subsets, and (b) an inner-layer parallel training for each subnetwork. In the outer-layer parallelism, we address critical issues of distributed and parallel computing, including data communication, synchronization, and workload balance. A heterogeneous-aware Incremental Data Partitioning and Allocation (IDPA) strategy is proposed, where large-scale training datasets are partitioned and allocated to the computing nodes in batches according to their computing power. To minimize the synchronization waiting during the global weight update process, an Asynchronous Global Weight Update (AGWU) strategy is proposed. In the inner-layer parallelism, we further accelerate the training process for each CNN subnetwork on each computer, where computation steps of convolutional layer and the local weight training are parallelized based on task-parallelism. We introduce task decomposition and scheduling strategies with the objectives of thread-level load balancing and minimum waiting time for critical paths. Extensive experimental results indicate that the proposed BPT-CNN effectively improves the training performance of CNNs while maintaining the accuracy.

An Image Classification Algorithm Based on Multidomain Convolution Neural Network

Deep Convolutional Neural Networks (CNNs) have outperformed humans in many computer vision tasks, such as object recognition and image classification, but it is almost impossible to run a large-scale CNN network structure in the platform and application scenarios with limited calculate ability. For the limited computing platform and application scenarios, we propose a novel CNN architecture: Multi-Domain CNN (MD-CNN). Multi-domain images are obtained by different pre-processing of the images input to the multi-domain convolutional neural network, and each image domain independently obtains the output features through a feedforward network. Then, the output features of multiple parallel multi-domain images are spliced together as the output characteristic of MD-CNN. This structure can effectively convert the network of CNN from depth to width, extract more efficient features quickly, which improves the training speed of the network. In the test of MNIST and CIFAR-10 datasets, the accuracy of our method reached 96.3% and 91.4% respectively, moreover, compared with the traditional CNN training speed increased about 1.5 times.

Deep Reinforcement Learning With Visual Attention for Vehicle Classification

Automatic vehicle classification is crucial to intelligent transportation system, especially for vehicle-tracking by police. Due to the complex lighting and image capture conditions, image-based vehicle classification in real-world environments is still a challenging task and the performance is far from being satisfactory. However, owing to the mechanism of visual attention, the human vision system shows remarkable capability compared with the computer vision system, especially in distinguishing nuances processing. Inspired by this mechanism, we propose a convolutional neural network (CNN) model of visual attention for image classification. A visual attention-based image processing module is used to highlight one part of an image and weaken the others, generating a focused image. Then the focused image is input into the CNN to be classified. According to the classification probability distribution, we compute the information entropy to guide a reinforcement learning agent to achieve a better policy for image classification to select the key parts of an image. Systematic experiments on a surveillance-nature dataset which contains images captured by surveillance cameras in the front view, demonstrate that the proposed model is more competitive than the large-scale CNN in vehicle classification tasks.