Convolutional Neural Networks Inference Research Articles

Towards practical and privacy-preserving CNN inference service for cloud-based medical imaging analysis: A homomorphic encryption-based approach.

Cloud-based Deep Learning as a Service (DLaaS) has transformed biomedicine by enabling healthcare systems to harness the power of deep learning for biomedical data analysis. However, privacy concerns emerge when sensitive user data must be transmitted to untrusted cloud servers. Existing privacy-preserving solutions are hindered by significant latency issues, stemming from the computational complexity of inner product operations in convolutional layers and the high communication costs of evaluating nonlinear activation functions. These limitations make current solutions impractical for real-world applications. In this paper, we address the challenges in mobile cloud-based medical imaging analysis, where users aim to classify private body-related radiological images using a Convolutional Neural Network (CNN) model hosted on a cloud server while ensuring data privacy for both parties. We propose PPCNN, a practical and privacy-preserving framework for CNN Inference. It introduces a novel mixed protocol that combines a low-expansion homomorphic encryption scheme with the noise-based masking method. Our framework is designed based on three key ideas: (1) optimizing computation costs by shifting unnecessary and expensive homomorphic multiplication operations to the offline phase, (2) introducing a coefficient-aware packing method to enable efficient homomorphic operations during the linear layer of the CNN, and (3) employing data masking techniques for nonlinear operations of the CNN to reduce communication costs. We implemented PPCNN and evaluated its performance on three real-world radiological image datasets. Experimental results show that PPCNN outperforms state-of-the-art methods in mobile cloud scenarios, achieving superior response times and lower usage costs. This study introduces an efficient and privacy-preserving framework for cloud-based medical imaging analysis, marking a significant step towards practical, secure, and trustworthy AI-driven healthcare solutions.

Dash: Accelerating Distributed Private Convolutional Neural Network Inference with Arithmetic Garbled Circuits

The adoption of machine learning solutions is rapidly increasing across all parts of society. As the models grow larger, both training and inference of machine learning models is increasingly outsourced, e.g. to cloud service providers. This means that potentially sensitive data is processed on untrusted platforms, which bears inherent data security and privacy risks. In this work, we investigate how to protect distributed machine learning systems, focusing on deep convolutional neural networks. The most common and best-performing mixed MPC approaches are based on HE, secret sharing, and garbled circuits. They commonly suffer from large performance overheads, big accuracy losses, and communication overheads that grow linearly in the depth of the neural network. To improve on these problems, we present Dash, a fast and distributed private convolutional neural network inference scheme secure against malicious attackers. Building on arithmetic garbling gadgets [BMR16] and fancy-garbling [BCM+19], Dash is based purely on arithmetic garbled circuits. We introduce LabelTensors that allow us to leverage the massive parallelity of modern GPUs. Combined with state-of-the-art garbling optimizations, Dash outperforms previous garbling approaches up to a factor of about 100. Furthermore, we introduce an efficient scaling operation over the residues of the Chinese remainder theorem representation to arithmetic garbled circuits, which allows us to garble larger networks and achieve much higher accuracy than previous approaches. Finally, Dash requires only a single communication round per inference step, regardless of the depth of the neural network, and a very small constant online communication volume.

Reducing inference energy consumption using dual complementary CNNs

Energy efficiency of Convolutional Neural Networks (CNNs) has become an important area of research, with various strategies being developed to minimize the power consumption of these models. Previous efforts, including techniques like model pruning, quantization, and hardware optimization, have made significant strides in this direction. However, there remains a need for more effective on device AI solutions that balance energy efficiency with model performance. In this paper, we propose a novel approach to reduce the energy requirements of inference of CNNs. Our methodology employs two small Complementary CNNs that collaborate with each other by covering each other’s “weaknesses” in predictions. If the confidence for a prediction of the first CNN is considered low, the second CNN is invoked with the aim of producing a higher confidence prediction. This dual-CNN setup significantly reduces energy consumption compared to using a single large deep CNN. Additionally, we propose a memory component that retains previous classifications for identical inputs, bypassing the need to re-invoke the CNNs for the same input, further saving energy. Our experiments on a Jetson Nano computer demonstrate an energy reduction of up to 85.8% achieved on modified datasets where each sample was duplicated once. These findings indicate that leveraging a complementary CNN pair along with a memory component effectively reduces inference energy while maintaining high accuracy.

End-to-End Latency Optimization for Resilient Distributed Convolutional Neural Network Inference in Resource-Constrained Unmanned Aerial Vehicle Swarms

An unmanned aerial vehicle (UAV) swarm has emerged as a powerful tool for mission execution in a variety of applications supported by deep neural networks (DNNs). In the context of UAV swarms, conventional methods for efficient data processing involve transmitting data to cloud and edge servers. However, these methods often face limitations in adapting to real-time applications due to the low latency of cloud-based approaches and weak mobility of edge-based approaches. In this paper, a new system called deep reinforcement learning-based resilient layer distribution (DRL-RLD) for distributed inference is designed to minimize end-to-end latency in UAV swarm, considering the resource constraints of UAVs. The proposed system dynamically allocates CNN layers based on UAV-to-UAV and UAV-to-ground communication links to minimize end-to-end latency. It can also enhance resilience to maintain mission continuity by reallocating layers when inoperable UAVs occur. The performance of the proposed system was verified through simulations in terms of latency compared to the comparison baselines, and its robustness was demonstrated in the presence of inoperable UAVs.

Practical Privacy-Preserving Convolutional Neural Network Inference Framework With Edge Computing for Health Monitoring

Practical Privacy-Preserving Convolutional Neural Network Inference Framework With Edge Computing for Health Monitoring

EcoFlow: Efficient Convolutional Dataflows on Low-Power Neural Network Accelerators

Dilated and transposed convolutions are widely used in modern convolutional neural networks (CNNs). These kernels are used extensively during CNN training and inference of applications such as image segmentation and high-resolution image generation. We find that commonly-used low-power CNN inference accelerators are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">not</i> optimized for both these convolutional kernels. Dilated and transposed convolutions introduce significant zero padding when mapped to the underlying spatial architecture, significantly degrading performance and energy efficiency. Existing approaches that address this issue require significant design changes to the otherwise simple, efficient, and well-adopted architectures used to compute direct convolutions. To address this challenge, we propose EcoFlow, a new set of dataflows and mapping algorithms for dilated and transposed convolutions. These algorithms are tailored to execute efficiently on existing low-cost, small-scale spatial architectures and requires minimal changes to existing accelerators. At its core, EcoFlow eliminates zero padding through careful dataflow orchestration and data mapping tailored to the spatial architecture. We evaluate EcoFlow on CNN training workloads and Generative Adversarial Network (GAN) workloads. Experiments in our new cycle-accurate simulator show that, using a common CNN inference accelerator, EcoFlow 1) reduces end-to-end CNN training time between 7-85%, and 2) improves end-to-end GAN training performance between 29-42%, compared to state-of-the-art CNN dataflows.

A binary particle swarm optimization-based pruning approach for environmentally sustainable and robust CNNs

Deep Convolutional Neural Networks (CNNs), continue to demonstrate remarkable performance across various tasks. However, their computational demands and energy consumption present significant drawbacks, restricting their practical deployment and contributing to a substantial carbon footprint. This paper addresses this challenge by proposing a novel method named Binary Particle Swarm Optimization Layer Pruner (BPSO-LPruner), aimed at achieving substantial computational reduction and mitigating environmental impact during CNN inference. BPSO-LPruner utilizes a constrained Binary Particle Swarm Optimization for CNN layer pruning, integrating a masked-bit strategy and a new population initialization strategy to enhance search performance. We illustrate the effectiveness of our method in reducing model computational costs and carbon footprint emissions while improving performance across multiple models (VGG16, VGG19, DenseNet-40, ResNet18, ResNet20, ResNet34, ResNet44, ResNet56, ResNet110, ResNet50, and MobileNetv2) and diverse datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet, COVID-19 X-ray dataset). Promising results underscore the performance of the proposed method. Additionally, we demonstrate that layer pruning yields benefits beyond enhanced computational performance. Our experimentation reveals that BPSO-LPruner enhances the model’s reliability and robustness by effectively addressing variations in input data, inherent ambiguity in model parameters, and adversarial images.

Revealing CNN Architectures via Side-Channel Analysis in Dataflow-based Inference Accelerators

Convolutional Neural Networks (CNNs) are widely used in various domains, including image recognition, medical diagnosis and autonomous driving. Recent advances in dataflow-based CNN accelerators have enabled CNN inference in resource-constrained edge devices. These dataflow accelerators utilize inherent data reuse of convolution layers to process CNN models efficiently. Concealing the architecture of CNN models is critical for privacy and security. This paper evaluates memory-based side-channel information to recover CNN architectures from dataflow-based CNN inference accelerators. The proposed attack exploits spatial and temporal data reuse of the dataflow mapping on CNN accelerators and architectural hints to recover the structure of CNN models. Experimental results demonstrate that our proposed side-channel attack can recover the structures of popular CNN models, namely Lenet, Alexnet, VGGnet16, and YOLOv2.

A CNN pruning approach using constrained binary particle swarm optimization with a reduced search space for image classification

Deep convolutional neural networks (CNNs) have exhibited exceptional performance in a range of computer vision tasks. However, these deep CNNs typically demand significant computational resources, which not only hinders their practical deployment but also contributes to a considerable carbon footprint. To tackle this issue, several filter pruning methods based on evolutionary algorithms have been proposed to provide significant memory and energy savings during CNN inference. However, due to the curse of high dimensionality in the structure of deep CNNs, the search space expands dramatically, presenting significant challenges for these methods. This paper proposes a novel algorithm called BPSO-FPruner for CNN filter pruning. BPSO-FPruner utilizes a constrained binary particle swarm optimization algorithm for filter pruning, incorporating a new initialization strategy based on filter weighting information and a reduced search space strategy. Extensive validation using VGG, ResNet, DenseNet, and MobileNetv2 architectures on the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets demonstrates the effectiveness of BPSO-FPruner in reducing model computational costs and carbon footprint emissions while maintaining or improving performance.

Researching the CNN Collaborative Inference Mechanism for Heterogeneous Edge Devices.

Convolutional Neural Networks (CNNs) have been widely applied in various edge computing devices based on intelligent sensors. However, due to the high computational demands of CNN tasks, the limited computing resources of edge intelligent terminal devices, and significant architectural differences among these devices, it is challenging for edge devices to independently execute inference tasks locally. Collaborative inference among edge terminal devices can effectively utilize idle computing and storage resources and optimize latency characteristics, thus significantly addressing the challenges posed by the computational intensity of CNNs. This paper targets efficient collaborative execution of CNN inference tasks among heterogeneous and resource-constrained edge terminal devices. We propose a pre-partitioning deployment method for CNNs based on critical operator layers, and optimize the system bottleneck latency during pipeline parallelism using data compression, queuing, and "micro-shifting" techniques. Experimental results demonstrate that our method achieves significant acceleration in CNN inference within heterogeneous environments, improving performance by 71.6% compared to existing popular frameworks.

Design of Fully Spectral CNNs for Efficient FPGA-Based Acceleration.

Computing convolutional layers in the frequency domain using fast Fourier transformation (FFT) has been demonstrated to be effective in reducing the computational complexity of convolutional neural networks (CNNs). Nevertheless, the main challenge of this approach lies in the frequent and repeated transformations between the spatial and frequency domains due to the absence of nonlinear functions in the spectral domain, as such it makes the benefit less attractive for low-latency inference, especially on embedded platforms. To overcome the drawbacks in the existing FFT-based convolution, we propose a fully spectral CNN using a novel spectral-domain adaptive rectified linear unit (ReLU) layer, which completely removes the compute-intensive transformations between the spatial and frequency domains within the network. The proposed fully spectral CNNs maintain the nonlinearity of the spatial CNNs while taking into account the hardware efficiency. We then propose a deeply customized and compute-efficient hardware architecture to accelerate the fully spectral CNN inference on field programmable gate array (FPGA). Different hardware optimizations, such as spectral-domain intralayer and interlayer pipeline techniques, are introduced to further improve the performance of throughput. To achieve a load-balanced pipeline, a design space exploration (DSE) framework is proposed to optimize the resource allocation between hardware modules according to the resource constraints. On an Intel's Arria 10 SX160 FPGA, our optimized accelerator achieves a throughput of 204 Gop/s with 80% of compute efficiency. Compared with the state-of-the-art spatial and FFT-based implementations on the same device, our accelerator is 4×∼ 6.6× and 3.0×∼ 4.4× faster while maintaining a similar level of accuracy across different benchmark datasets.

Optimizing CNN inference speed over big social data through efficient model parallelism for sustainable web of things

The rapid development of artificial intelligence and networking technologies has catalyzed the popularity of intelligent services based on deep learning in recent years, which in turn fosters the advancement of Web of Things (WoT). Big social data (BSD) plays an important role during the processing of intelligent services in WoT. However, intelligent BSD services are computationally intensive and require ultra-low latency. End or edge devices with limited computing power cannot realize the extremely low response latency of those services. Distributed inference of deep neural networks (DNNs) on various devices is considered a feasible solution by allocating the computing load of a DNN to several devices. In this work, an efficient model parallelism method that couples convolution layer (Conv) split with resource allocation is proposed. First, given a random computing resource allocation strategy, the Conv split decision is made through a mathematical analysis method to realize the parallel inference of convolutional neural networks (CNNs). Next, Deep Reinforcement Learning is used to get the optimal computing resource allocation strategy to maximize the resource utilization rate and minimize the CNN inference latency. Finally, simulation results show that our approach performs better than the baselines and is applicable for BSD services in WoT with a high workload.

ADS-CNN: Adaptive Dataflow Scheduling for lightweight CNN accelerator on FPGAs

Lightweight convolutional neural networks (CNNs) enable lower inference latency and data traffic, facilitating deployment on resource-constrained edge devices such as field-programmable gate arrays (FPGAs). However, CNNs inference requires access to off-chip synchronous dynamic random-access memory (SDRAM), which significantly degrades inference speed and system power efficiency. In this paper, we propose an adaptive dataflow scheduling method for lightweight CNN accelerator on FPGAs named ADS-CNN. The key idea of ADS-CNN is to efficiently utilize on-chip resources and reduce the amount of SDRAM access. To achieve the reuse of logical resources, we design a time division multiplexing calculation engine to be integrated in ADS-CNN. We implement a configurable module for the convolution controller to adapt to the data reuse of different convolution layers, thus reducing the off-chip access. Furthermore, we exploit on-chip memory blocks as buffers based on the configuration of different layers in lightweight CNNs. On the resource-constrained Intel CycloneV SoC 5CSEBA6 FPGA platform, we evaluated six common lightweight CNN models to demonstrate the performance advantages of ADS-CNN. The evaluation results indicate that, compared with accelerators that use traditional tiling strategy dataflow, our ADS-CNN can achieve up to 1.29× speedup with the overall dataflow scale compression of 23.7%.

Zero-Ran Sniff: A zero-day ransomware early detection method based on zero-shot learning

Ransomware attacks, which blackmail victims into paying a ransom by locking their devices or encrypting their files, have become one of the major threats to network security. Conventional anti-ransomware tools often fail to detect zero-day ransomware attacks due to the inability to obtain zero-day ransomware signatures in advance to train detection models. In addition, zero-day ransomware attacks often use sophisticated encryption techniques to launch attacks on new vulnerabilities, and these encryption attacks cause irreversible damage to victims' digital files even if they choose to pay a ransom. Hence, it is imperative and urgent to detect unknown ransomware attacks at the earliest possible stage, ideally before the encryption phase. To this end, this paper proposes Zero-Ran Sniff (ZRS), an early zero-day ransomware detection method based on zero-shot learning, which can detect zero-day ransomware attacks in the early stage. ZRS leverages the portable executable header (PE header) feature from executable files to identify ransomware. It comprises two stages: an auto-encoding network-based core attribute learning (AE-CAL) stage and a self-attentive mechanism-based convolutional neural network inference Stage (SA-CNN-IS). During the AE-CAL stage, the core features of known and unknown classes of ransomware are extracted using self-encoding networks, and the SA-CNN-IS phase identifies ransomware. To the best of our knowledge, we are the first to explore the use of zero-shot learning for zero-day ransomware early detection. Experimental results demonstrate that the proposed ZRS outperforms traditional machine learning methods. Compared to previous zero-day detection work, ZRS achieves a recall of 98.47% and an accuracy of 96.31%

CNN based plant disease identification using PYNQ FPGA

This research presents a novel approach for plant disease identification utilizing Convolutional Neural Networks (CNNs) and the PYNQ FPGA platform. The study leverages the parallel processing capabilities of FPGAs to accelerate CNN inference, aiming to enhance the efficiency of plant disease detection in agricultural settings. The implementation involves optimizing the CNN architecture for deployment on the PYNQ FPGA, considering factors such as image size and learning rates. Through experimentation, the research refines hyper parameters, achieving improved accuracy and F1 scores. Visualizations using heat maps highlight the CNN's reliance on color, shape, and texture for feature extraction in disease identification. The integration of FPGA technology demonstrates promising advancements in real-time, high-performance plant disease classification, offering potential benefits for precision agriculture and crop management. This research contributes to the growing field of FPGA-accelerated deep learning applications in agro technology, addressing challenges in plant health monitoring and fostering sustainable agricultural practices.

PICO: Pipeline Inference Framework for Versatile CNNs on Diverse Mobile Devices

Distributing the inference of convolutional neural network (CNN) to multiple mobile devices has been studied in recent years to achieve real-time inference without losing accuracy. However, how to map CNN to devices remains a challenge. On the one hand, scheduling the workload of state-of-the-art CNNs with multiple devices is NP-Hard because the structures of CNNs are directed acyclic graphs (DAG) rather than simple chains. On the other hand, distributing the inference workload suffers from expensive communication and unbalanced computation due to the wireless environment and heterogeneous devices. This paper presents PICO, a pipeline cooperation framework to accelerate the inference of versatile CNNs on diverse mobile devices. At its core, PICO features: (1) a generic graph partition algorithm that considers the characteristics of any given CNN and orchestrates it into a list of model pieces with suitable granularity, and (2) a many-to-many mapping algorithm that produces the best pipeline configuration for heterogeneous devices. In our experiment with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2 \sim 8$</tex-math></inline-formula> Raspberry-Pi devices, the throughput can be improved by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.8 \sim 6.8 \times$</tex-math></inline-formula> under different CPU frequencies.

Coarse-to-Fine: A hierarchical DNN inference framework for edge computing

Deep neural networks (DNNs) have been increasingly used in recent years to achieve higher inference accuracy; however, implementing deeper networks in edge-computing environments can be challenging. Current methods for accelerating CNN inference focus on finding a trade-off between accuracy and latency under an assumed uniform distribution, ignoring the impact of real-world data distributions. To address this, we propose the Coarse-to-Fine (C2F) framework, which includes a C2F model and a corresponding C2F inference architecture to better exploit distributional differences in the edge environment. The C2F model is derived from various adaptations of Convolutional Neural Networks (CNNs). By deconstructing the original CNNs into multiple smaller models, the C2F model increases memory consumption within an acceptable range to improve inference speed without sacrificing accuracy. The C2F architecture deploys C2F models more logically in complex edge environments, reducing inference costs and memory consumption. We conduct experiments on the CIFAR dataset with different backbone networks and show that our C2F framework can simultaneously reduce latency and improve accuracy in complex edge environments.

Pflow: An end-to-end heterogeneous acceleration framework for CNN inference on FPGAs

Field-Programmable Gate Arrays (FPGAs), renowned for their high performance per watt, are extensively utilized to accelerate Convolutional Neural Networks (CNNs) in edge computing environments, primarily employing dataflow-based and instruction set-based approaches. Compared to the instruction set-based approach that features fast and versatile circuit design, the dataflow-based approach can significantly enhance performance at the expense of design versatility. Nevertheless, edge computing environments require both high energy efficiency and adaptability to various scenarios. This paper proposes a novel end-to-end heterogeneous acceleration framework for CNN inference on FPGAs, named Pflow. The basic idea is to decouple network deployment and hardware details with a hardware–software co-design approach. First, a dataflow accelerator with an adaptive scheduling strategy is proposed. The adaptive scheduling strategy, along with a scalable design, maximizes hardware utilization in terms of computing resources and bandwidth. Secondly, we design a novel operator-perception method to automate the processes of network reconstruction and operator fusion. Thirdly, we integrate Pflow into the industrial-grade deep learning framework Paddle-Lite. We evaluate Pflow by implementing several networks on two representative FPGA platforms. Experimental results demonstrate that Pflow achieves energy efficiencies of 46.5 GOPS/W on Xilinx Zynq Ultrascale+ MPSoC 3EG and 59.4 GOPS/W on Virtex UltraScale+ XCVU13P. It also reaches a throughput of up to 255.7 GOPS on the former and 3.686 TOPS on the latter.

TinyAD: Memory-Efficient Anomaly Detection for Time-Series Data in Industrial IoT

Monitoring and detecting abnormal events in cyber-physical systems is crucial to industrial production. With the prevalent deployment of the Industrial Internet of Things (IIoT), an enormous amount of time series data is collected to facilitate machine learning models for anomaly detection, and it is of the utmost importance to directly deploy the trained models on the IIoT devices. However, it is most challenging to deploy complex deep learning models such as Convolutional Neural Networks (CNNs) on these memory-constrained IIoT devices embedded with microcontrollers (MCUs). To alleviate the memory constraints of MCUs, we propose a novel framework named Tiny Anomaly Detection (TinyAD) to efficiently facilitate onboard inference of CNNs for real-time anomaly detection. First, we conduct a comprehensive analysis of depthwise separable CNNs and regular CNNs for anomaly detection and find that the depthwise separable convolution operation can reduce the model size by 50-90% compared with the traditional CNNs. Then, to reduce the peak memory consumption of CNNs, we explore two complementary strategies, in-place, and patch-by-patch memory rescheduling, and integrate them into a unified framework. The in-place method decreases the peak memory of the depthwise convolution by sparing a temporary buffer to transfer the activation results, while the patch-by-patch method further reduces the peak memory of layer-wise execution by slicing the input data into corresponding receptive fields and executing in order. Furthermore, by adjusting the dimension of convolution filters, these strategies apply to both univariate time series and multidomain time series features. Extensive experiments on real-world industrial datasets show that our framework can reduce peak memory consumption by 2-5x with negligible computation overhead.

Research on Convolutional Neural Network Inference Acceleration and Performance Optimization for Edge Intelligence.

In recent years, edge intelligence (EI) has emerged, combining edge computing with AI, and specifically deep learning, to run AI algorithms directly on edge devices. In practical applications, EI faces challenges related to computational power, power consumption, size, and cost, with the primary challenge being the trade-off between computational power and power consumption. This has rendered traditional computing platforms unsustainable, making heterogeneous parallel computing platforms a crucial pathway for implementing EI. In our research, we leveraged the Xilinx Zynq 7000 heterogeneous computing platform, employed high-level synthesis (HLS) for design, and implemented two different accelerators for LeNet-5 using loop unrolling and pipelining optimization techniques. The experimental results show that when running at a clock speed of 100 MHz, the PIPELINE accelerator, compared to the UNROLL accelerator, experiences an 8.09% increase in power consumption but speeds up by 14.972 times, making the PIPELINE accelerator superior in performance. Compared to the CPU, the PIPELINE accelerator reduces power consumption by 91.37% and speeds up by 70.387 times, while compared to the GPU, it reduces power consumption by 93.35%. This study provides two different optimization schemes for edge intelligence applications through design and experimentation and demonstrates the impact of different quantization methods on FPGA resource consumption. These experimental results can provide a reference for practical applications, thereby providing a reference hardware acceleration scheme for edge intelligence applications.