Deep Learning Inference Research Articles

Integrating contact tracing data to enhance outbreak phylodynamic inference: a deep learning approach.

Phylodynamics is central to understanding infectious disease dynamics through the integration of genomic and epidemiological data. Despite advancements, including the application of deep learning to overcome computational limitations, significant challenges persist due to data inadequacies and statistical unidentifiability of key parameters. These issues are particularly pronounced in poorly resolved phylogenies, commonly observed in outbreaks such as SARS-CoV-2. In this study, we conducted a thorough evaluation of PhyloDeep, a deep learning inference tool for phylodynamics, assessing its performance on poorly resolved phylogenies. Our findings reveal the limited predictive accuracy of PhyloDeep (and other state-of-the-art approaches) in these scenarios. However, models trained on poorly resolved, realistically simulated trees demonstrate improved predictive power, despite not being infallible, especially in scenarios with superspreading dynamics, whose parameters are challenging to capture accurately. Notably, we observe markedly improved performance through the integration of minimal contact tracing data, which refines poorly resolved trees. Applying this approach to a sample of SARS-CoV-2 sequences partially matched to contact tracing from Hong Kong yields informative estimates of superspreading potential, extending beyond the scope of contact tracing data alone. Our findings demonstrate the potential for enhancing phylodynamic analysis through complementary data integration, ultimately increasing the precision of epidemiological predictions crucial for public health decision making and outbreak control.

A resource-aware workload scheduling method for unbalanced GEMMs on GPUs

Abstract GEMM (General Matrix Multiplication) serves as a fundamental operator for deep learning computations. Especially in attention-based deep learning models, such as Bert, GPT, and SAM, the sizes of matrices involved in GEMMs exhibit an unbalanced distribution due to the variable input, resulting in the low utilization of hardware resources. To address the issue, this paper proposes inserting a novel GEMM processing layer into the deep learning inference stack and using an adaptive load balancing method to partition and schedule GEMM computation tasks. The method is implemented with hardware runtime resource information, such as the occupancy of computing units, etc. Experiment results show the remarkable performance of our method in unbalanced input GEMM scenarios, achieving an average performance improvement of 2.3x. The method also performs well in attention-based models (GPT-2 and SAM), achieving an average inference speed improvement of 1.1x. These findings highlight the effectiveness of resource-aware algorithm optimization, especially for computation task scheduling.

Unmanned Ground Vehicles for Continuous Crop Monitoring in Agriculture: Assessing the Readiness of Current ICT Technology

Continuous crop monitoring enables the early detection of field emergencies such as pests, diseases, and nutritional deficits, allowing for less invasive interventions and yielding economic, environmental, and health benefits. The work organization of modern agriculture, however, is not compatible with continuous human monitoring. ICT can facilitate this process using autonomous Unmanned Ground Vehicles (UGVs) to navigate crops, detect issues, georeference them, and report to human experts in real time. This review evaluates the current state of ICT technology to determine if it supports autonomous, continuous crop monitoring. The focus is on shifting from traditional cloud-based approaches, where data are sent to remote computers for deferred processing, to a hybrid design emphasizing edge computing for real-time analysis in the field. Key aspects considered include algorithms for in-field navigation, AIoT models for detecting agricultural emergencies, and advanced edge devices that are capable of managing sensors, collecting data, performing real-time deep learning inference, ensuring precise mapping and navigation, and sending alert reports with minimal human intervention. State-of-the-art research and development in this field suggest that general, not necessarily crop-specific, prototypes of fully autonomous UGVs for continuous monitoring are now at hand. Additionally, the demand for low-power consumption and affordable solutions can be practically addressed.

SCL: A sustainable deep learning solution for edge computing ecosystem in smart manufacturing

Edge computing empowered Deep Learning (DL) solutions have risen as the foremost facilitators of automation in a multitude of smart manufacturing applications. These models are implemented on edge devices with frozen learning capabilities to execute DL inference task(s). Nevertheless, the data they process is susceptible to intermittent alterations amidst the ever-changing landscape of dynamic smart manufacturing ecosystem. It sparks the demand for model maintenance solution(s) to address adaptability and dynamism issues to enhance the efficiency of smart manufacturing solutions. Moreover, additional issue(s), such as the non-availability of comprehensive data (or the availability of solely contemporary data), near-real-time execution of DL model maintenance task, etc., imposes daunting obstructions in devising an efficient DL model maintenance strategy. This work proposes a novel approach that encompasses the merits of Continual Learning (CL) and Split Learning (SL) driven by edge intelligence, amalgamating them into a hybrid solution aptly named Split-based Continual Learning (SCL). CL ensures the sustained performance of the DL model amidst constraints related to data availability. At the same time, SL empowers near-real-time execution at the edge to achieve improved efficiency. An extension of the SCL scheme, termed as Extended SCL (ESCL), is implemented to addresses the interaction soundness aspects among the mobile edge devices in a collaborative execution environment. Evaluation of a vision-based product-quality inspection use case in an emulated hardware test-bed setup signifies that the performance of SCL and ESCL schemes have the potential to meet the needs of smart manufacturing. SCL attains an appreciable reduction in the model maintenance cost in the range of 21 to 48 and 12 to 29 percent compared to the ECN-only and basic-SL schemes. The ESCL scheme further improved the performance by 18 to 34 and 20 to 36 percent respectively over the basic-SL and SCL.

An initial game-theoretic assessment of enhanced tissue preparation and imaging protocols for improved deep learning inference of spatial transcriptomics from tissue morphology.

The application of deep learning to spatial transcriptomics (ST) can reveal relationships between gene expression and tissue architecture. Prior work has demonstrated that inferring gene expression from tissue histomorphology can discern these spatial molecular markers to enable population scale studies, reducing the fiscal barriers associated with large-scale spatial profiling. However, while most improvements in algorithmic performance have focused on improving model architectures, little is known about how the quality of tissue preparation and imaging can affect deep learning model training for spatial inference from morphology and its potential for widespread clinical adoption. Prior studies for ST inference from histology typically utilize manually stained frozen sections with imaging on non-clinical grade scanners. Training such models on ST cohorts is also costly. We hypothesize that adopting tissue processing and imaging practices that mirror standards for clinical implementation (permanent sections, automated tissue staining, and clinical grade scanning) can significantly improve model performance. An enhanced specimen processing and imaging protocol was developed for deep learning-based ST inference from morphology. This protocol featured the Visium CytAssist assay to permit automated hematoxylin and eosin staining (e.g. Leica Bond), 40×-resolution imaging, and joining of multiple patients' tissue sections per capture area prior to ST profiling. Using a cohort of 13 pathologic T Stage-III stage colorectal cancer patients, we compared the performance of models trained on slide prepared using enhanced versus traditional (i.e. manual staining and low-resolution imaging) protocols. Leveraging Inceptionv3 neural networks, we predicted gene expression across serial, histologically-matched tissue sections using whole slide images (WSI) from both protocols. The data Shapley was used to quantify and compare marginal performance gains on a patient-by-patient basis attributed to using the enhanced protocol versus the actual costs of spatial profiling. Findings indicate that training and validating on WSI acquired through the enhanced protocol as opposed to the traditional method resulted in improved performance at lower fiscal cost. In the realm of ST, the enhancement of deep learning architectures frequently captures the spotlight; however, the significance of specimen processing and imaging is often understated. This research, informed through a game-theoretic lens, underscores the substantial impact that specimen preparation/imaging can have on spatial transcriptomic inference from morphology. It is essential to integrate such optimized processing protocols to facilitate the identification of prognostic markers at a larger scale.

Anatomizing Deep Learning Inference in Web Browsers

Web applications have increasingly adopted Deep Learning (DL) through in-browser inference , wherein DL inference performs directly within Web browsers. The actual performance of in-browser inference and its impacts on the quality of experience ( QoE ) remain unexplored, and urgently require new QoE measurements beyond traditional ones, e.g., mainly focusing on page load time. To bridge this gap, we make the first comprehensive performance measurement of in-browser inference to date. Our approach proposes new metrics to measure in-browser inference: responsiveness, smoothness, and inference accuracy. Our extensive analysis involves 9 representative DL models across Web browsers of 50 popular PC devices and 20 mobile devices. The results reveal that in-browser inference exhibits a substantial latency gap, averaging 16.9 times slower on CPU and 4.9 times slower on GPU compared to native inference on PC devices. The gap on mobile CPU and mobile GPU is 15.8 times and 7.8 times, respectively. Furthermore, we identify contributing factors to such latency gap, including underutilized hardware instruction sets, inherent overhead in the runtime environment, resource contention within the browser, and inefficiencies in software libraries and GPU abstractions. Additionally, in-browser inference imposes significant memory demands, at times exceeding 334.6 times the size of the DL models themselves, partly attributable to suboptimal memory management. We also observe that in-browser inference leads to a significant 67.2% increase in the time it takes for GUI components to render within Web browsers, significantly affecting the overall user QoE of Web applications reliant on this technology.

End-to-end reproducible AI pipelines in radiology using the cloud

Artificial intelligence (AI) algorithms hold the potential to revolutionize radiology. However, a significant portion of the published literature lacks transparency and reproducibility, which hampers sustained progress toward clinical translation. Although several reporting guidelines have been proposed, identifying practical means to address these issues remains challenging. Here, we show the potential of cloud-based infrastructure for implementing and sharing transparent and reproducible AI-based radiology pipelines. We demonstrate end-to-end reproducibility from retrieving cloud-hosted data, through data pre-processing, deep learning inference, and post-processing, to the analysis and reporting of the final results. We successfully implement two distinct use cases, starting from recent literature on AI-based biomarkers for cancer imaging. Using cloud-hosted data and computing, we confirm the findings of these studies and extend the validation to previously unseen data for one of the use cases. Furthermore, we provide the community with transparent and easy-to-extend examples of pipelines impactful for the broader oncology field. Our approach demonstrates the potential of cloud resources for implementing, sharing, and using reproducible and transparent AI pipelines, which can accelerate the translation into clinical solutions.

A Hybrid Neuromorphic Object Tracking and Classification Framework for Real-Time Systems.

Deep learning inference that needs to largely take place on the "edge" is a highly computational and memory intensive workload, making it intractable for low-power, embedded platforms such as mobile nodes and remote security applications. To address this challenge, this article proposes a real-time, hybrid neuromorphic framework for object tracking and classification using event-based cameras that possess desirable properties such as low-power consumption (5-14 mW) and high dynamic range (120 dB). Nonetheless, unlike traditional approaches of using event-by-event processing, this work uses a mixed frame and event approach to get energy savings with high performance. Using a frame-based region proposal method based on the density of foreground events, a hardware-friendly object tracking scheme is implemented using the apparent object velocity while tackling occlusion scenarios. The frame-based object track input is converted back to spikes for TrueNorth (TN) classification via the energy-efficient deep network (EEDN) pipeline. Using originally collected datasets, we train the TN model on the hardware track outputs, instead of using ground truth object locations as commonly done, and demonstrate the ability of our system to handle practical surveillance scenarios. As an alternative tracker paradigm, we also propose a continuous-time tracker with C++ implementation where each event is processed individually, which better exploits the low latency and asynchronous nature of neuromorphic vision sensors. Subsequently, we extensively compare the proposed methodologies to state-of-the-art event-based and frame-based methods for object tracking and classification, and demonstrate the use case of our neuromorphic approach for real-time and embedded applications without sacrificing performance. Finally, we also showcase the efficacy of the proposed neuromorphic system to a standard RGB camera setup when simultaneously evaluated over several hours of traffic recordings.

Automatic Determination of Endothelial Cell Density From Donor Cornea Endothelial Cell Images.

To determine endothelial cell density (ECD) from real-world donor cornea endothelial cell (EC) images using a self-supervised deep learning segmentation model. Two eye banks (Eversight, VisionGift) provided 15,138 single, unique EC images from 8169 donors along with their demographics, tissue characteristics, and ECD. This dataset was utilized for self-supervised training and deep learning inference. The Cornea Image Analysis Reading Center (CIARC) provided a second dataset of 174 donor EC images based on image and tissue quality. These images were used to train a supervised deep learning cell border segmentation model. Evaluation between manual and automated determination of ECD was restricted to the 1939 test EC images with at least 100 cells counted by both methods. The ECD measurements from both methods were in excellent agreement with rc of 0.77 (95% confidence interval [CI], 0.75-0.79; P < 0.001) and bias of 123 cells/mm2 (95% CI, 114-131; P < 0.001); 81% of the automated ECD values were within 10% of the manual ECD values. When the analysis was further restricted to the cropped image, the rc was 0.88 (95% CI, 0.87-0.89; P < 0.001), bias was 46 cells/mm2 (95% CI, 39-53; P < 0.001), and 93% of the automated ECD values were within 10% of the manual ECD values. Deep learning analysis provides accurate ECDs of donor images, potentially reducing analysis time and training requirements. The approach of this study, a robust methodology for automatically evaluating donor cornea EC images, could expand the quantitative determination of endothelial health beyond ECD.

Symmetric silicon microring resonator optical crossbar array for accelerated inference and training in deep learning

Photonic integrated circuits are emerging as a promising platform for accelerating matrix multiplications in deep learning, leveraging the inherent parallel nature of light. Although various schemes have been proposed and demonstrated to realize such photonic matrix accelerators, the in situ training of artificial neural networks using photonic accelerators remains challenging due to the difficulty of direct on-chip backpropagation on a photonic chip. In this work, we propose a silicon microring resonator (MRR) optical crossbar array with a symmetric structure that allows for simple on-chip backpropagation, potentially enabling the acceleration of both the inference and training phases of deep learning. We demonstrate a 4×4 circuit on a Si-on-insulator platform and use it to perform inference tasks of a simple neural network for classifying iris flowers, achieving a classification accuracy of 93.3%. Subsequently, we train the neural network using simulated on-chip backpropagation and achieve an accuracy of 91.1% in the same inference task after training. Furthermore, we simulate a convolutional neural network for handwritten digit recognition, using a 9×9 MRR crossbar array to perform the convolution operations. This work contributes to the realization of compact and energy-efficient photonic accelerators for deep learning.

Fault-tolerant deep learning inference on CPU-GPU integrated edge devices with TEEs

CPU-GPU integrated edge devices and deep learning algorithms have received significant progress in recent years, leading to increasingly widespread application of edge intelligence. However, deep learning inference on these edge devices is vulnerable to Fault Injection Attacks (FIAs) that can modify device memory or execute instructions with errors. We propose DarkneTF, a Fault-Tolerant (FT) deep learning inference framework for CPU-GPU integrated edge devices, to ensure the correctness of model inference results by detecting the threat of FIAs. DarkneTF introduces algorithm-based verification to implement the FT deep learning inference. The verification process involves verifying the integrity of model weights and validating the correctness of time-intensive calculations, such as convolutions. We improve the Freivalds algorithm to enhance the ability to detect tiny perturbations by strengthening randomization. As the verification process is also susceptible to FIAs, DarkneTF offloads the verification process into Trusted Execution Environments (TEEs). This scheme ensures the verification process’s security and allows for accelerated model inference using the integrated GPUs. Experimental results show that GPU-accelerated FT inference on HiKey 960 achieves notable speedups ranging from 3.46x to 5.57x compared to FT inference on a standalone CPU. The extra memory overhead incurred FT inference remains at an exceedingly low level, with a range of 0.46% to 10.22%. The round-off error of the improved Freivalds algorithm is below 2.50×10−4, and the accuracy of detecting FIAs is above 92.73%.

IoT Embedded Smart Monitoring System with Edge Machine Learning for Beehive Management

The need of an automated support system that helps beekeepers maintain and improve beehive population was always a very stressing aspect of their work considering the importance of a healthy bee population. This paper presents a proof of concept, further referred as a PoC solution, based on the Internet of Things technology which proposes a smart monitoring system using machine learning processes, diligently combining the power of edge computing by means of communication and control. Beehive maintenance is improved, having an optimal state of health due to the Deep Learning inference triggered at the edge level of devices which processes hive’s noises. All this is achieved by using IoT sensors to collect data, extract important features and a Tiny ML network for decision support. Having Machine Learning inference to be performed on low-power microcontroller devices leads to significant improvements in the autonomy of beekeeping solutions.

Deep Smart Cities: A Review of Smart Cities Applications-Based an Inference of Deep Learning Upon IoT Devices

Recent years have witnessed an increase in the use of IoT devices in Smart Cities (SC). With the remarkable success of integrating deep learning (DL) in various fields of SC such as smart environment, smart agriculture, and more. There is an emerging field to deploy DL into IoT devices to meet the requirement of SC applications as real-time applications and lightweight computation. However, the limited resource of IoT devices poses a challenge to fulfilling the DL models that demand large storage and massive computation. The aim of this paper is focused on the integration of DL with IoT in SC and conducts up-to-date state-of-the-art studies on the SC applications. First, a present overview of SC, IoT and DL. Second, we review studies that integrate DL and IoT into four SC applications (smart environment, smart transportation, smart agriculture, and smart home). It covers a series of crucial applications in SC such as Air pollution, water pollution, autonomous driving, activity recognition and detection, indoor localization, and more. Third, present the main challenges once integrate DL on IoT devices in SC. Lastly, we introduced open issues for the future direction obtained when integrate DL models with IoT devices in SC applications to inspire further research in this area.

Extraction of block walls from point clouds measured by Mobile Mapping System

Abstract. To solve the problem of collapsing block walls widely used in Japan, this study proposes a method for extracting block walls using 3D point cloud data measured by the Mobile Mapping System (MMS). Unlike conventional methods, this method identifies block walls based on geometric features without relying on MMS trajectory data or deep learning inference results. In addition, the computational load is low and manual correction can be minimized. In our experiments, we used point cloud data collected in urban areas in Japan and achieved a precision of 0.750, recall of 0.810, and F-measure of 0.779. The results demonstrate the effectiveness of this method for automatic extraction of block walls and rapid assessment of collapse risk and are expected to contribute to safety measures in areas with high seismic risk.

DIFFUSION MODEL-BASED POSTERIOR DISTRIBUTION PREDICTION FOR KINETIC PARAMETER ESTIMATION IN DYNAMIC PET.

Positron Emission Tomography (PET) is a valuable imaging method for studying molecular-level processes in the body, such as hyperphosphorylated tau (p-tau) protein aggregates, a hallmark of several neurodegenerative diseases including Alzheimer's disease. P-tau density and cerebral perfusion can be quantified from PET data using tracer kinetic modeling techniques. However, noise in PET images leads to uncertainty in the estimated kinetic parameters. This can be quantified in a Bayesian framework by the posterior distribution of kinetic parameters given PET measurements. Markov Chain Monte Carlo (MCMC) techniques can be employed to estimate the posterior distribution, although with significant computational needs. In this paper, we propose to leverage deep learning inference efficiency to infer the posterior distribution. A novel approach using denoising diffusion probabilistic model (DDPM) is introduced. The performance of the proposed method was evaluated on a [18F]MK6240 study and compared to an MCMC method. Our approach offered significant reduction in computation time (over 30 times faster than MCMC) and consistently predicted accurate (< 0.8 % mean error) and precise (< 5.77 % standard deviation error) posterior distributions.

A high-performance dataflow-centric optimization framework for deep learning inference on the edge

Edge computing has been emerging as a popular scenario for model inference. However, the inference performance on edge devices (e.g., Multi-Core DSP, FGPA, etc.) suffers from inefficiency due to the lack of highly optimized inference frameworks. Previous model inference frameworks are mainly developed in an operator-centric way, which provides insufficient acceleration to edge-based inference. Besides, the operator-centric framework incurs significant costs for continuous development and maintenance.Targeting the existing drawbacks of operator-centric frameworks, we design Xenos, which can automatically conduct dataflow-centric optimization of the computation graph and accelerate inference in two dimensions. Vertically, Xenos develops operator linking technique to improve data locality by restructuring the inter-operator dataflow. Horizontally, Xenos develops DSP-aware operator split technique to enable higher parallelism across multiple DSP units. Our evaluation demonstrates the effectiveness of vertical and horizontal dataflow optimization, which reduce the inference time by 15.0%–84.9% and 17.9%–89.9% , respectively. Besides, Xenos also outperforms the widely-used TVM by 1.1×–1.9×. Moreover, we extend Xenos to a distributed solution, which we call d-Xenos. d-Xenos employs multiple edge devices to jointly conduct the inference task and achieves a speedup of 3.68×–3.78× compared with the single device.

An Approach to the Systematic Characterization of Multitask Accelerated CNN Inference in Edge MPSoCs

Deep Learning is ubiquitous today and is increasingly moving from the cloud down to the edge of networked infrastructures, where it enables embedded applications to perform complex inference tasks close to the data sources, reducing long-distance data movement and alleviating the need for a powerful cloud infrastructure. Edge-class multi-processor system on chip (MPSoC) devices featuring an on-chip FPGA fabric offer key advantages for Deep Learning inference tasks, especially for complex applications where multiple models may be run concurrently in the same platform. In this work, we propose an approach and a practical framework for the systematic characterization of multithreaded Deep Learning inference on edge FPGA MPSoCs. We instantiate the framework into a real-world MPSoC platform, targeting Xilinx Vitis-AI as a representative example of a commercial Deep Learning acceleration toolkit for edge environments. We design a comprehensive experimental campaign and apply it to the platform for several convolutional neural networks, each trained on three different datasets. We show that our approach can be used for both hardware- and software-level analysis of a target system. Among other findings, the analysis revealed a suboptimal behavior of the underlying toolkit runtime, involving the utilization of the accelerator cores and the uneven software latency of the support library, influenced by the shapes of the input tensors.

HyBNN: Quantifying and Optimizing Hardware Efficiency of Binary Neural Networks

Binary neural network (BNN), where both the weight and the activation values are represented with one bit, provides an attractive alternative to deploy highly efficient deep learning inference on resource-constrained edge devices. However, our investigation reveals that, to achieve satisfactory accuracy gains, state-of-the-art (SOTA) BNNs, such as FracBNN and ReActNet, usually have to incorporate various auxiliary floating-point components and increase the model size, which in turn degrades the hardware performance efficiency. In this article, we aim to quantify such hardware inefficiency in SOTA BNNs and further mitigate it with negligible accuracy loss. First, we observe that the auxiliary floating-point (AFP) components consume an average of 93% DSPs, 46% LUTs, and 62% FFs, among the entire BNN accelerator resource utilization. To mitigate such overhead, we propose a novel algorithm-hardware co-design, called FuseBNN , to fuse those AFP operators without hurting the accuracy. On average, FuseBNN reduces AFP resource utilization to 59% DSPs, 13% LUTs, and 16% FFs. Second, SOTA BNNs often use the compact MobileNetV1 as the backbone network but have to replace the lightweight 3 × 3 depth-wise convolution (DWC) with the 3 × 3 standard convolution (SC, e.g., in ReActNet and our ReActNet-adapted BaseBNN) or even more complex fractional 3 × 3 SC (e.g., in FracBNN) to bridge the accuracy gap. As a result, the model parameter size is significantly increased and becomes 2.25× larger than that of the 4-bit direct quantization with the original DWC (4-Bit-Net); the number of multiply-accumulate operations is also significantly increased so that the overall LUT resource usage of BaseBNN is almost the same as that of 4-Bit-Net. To address this issue, we propose HyBNN , where we binarize depth-wise separation convolution (DSC) blocks for the first time to decrease the model size and incorporate 4-bit DSC blocks to compensate for the accuracy loss. For the ship detection task in synthetic aperture radar imagery on the AMD-Xilinx ZCU102 FPGA, HyBNN achieves a detection accuracy of 94.8% and a detection speed of 615 frames per second (FPS), which is 6.8× faster than FuseBNN+ (94.9% accuracy) and 2.7× faster than 4-Bit-Net (95.9% accuracy). For image classification on the CIFAR-10 dataset on the AMD-Xilinx Ultra96-V2 FPGA, HyBNN achieves 1.5× speedup and 0.7% better accuracy over SOTA FracBNN.

High-Speed Power Allocation in NOMA System Using FPGA-Based DNN

Artificial Intelligence (AI) is rapidly transforming the healthcare, finance and transportation industries. This paper presents a field-programmable gate array (FPGA)-based neural network accelerator (NNA) design for power allocation in downlink nonorthogonal multiple access (NOMA) networks. The proposed hardware accelerator effectively cuts computational costs while delivering performance on par with the highest sum capacity. Numerical results show that this NNA offers a remarkable computational speed increase of up to 99% compared to the conventional exhaustive search method. Furthermore, the deep learning (DL) model achieved high accuracy (0.92 training, 0.93 testing), and the hardware accelerator design for this DL inference model was implemented on the PYNQ-Z2 board-constrained edge device to predict power allocation coefficients in NOMA systems.

EPIDL: Towards efficient and privacy‐preserving inference in deep learning

SummaryDeep learning has shown its great potential in real‐world applications. However, users(clients) who want to use deep learning applications need to send their data to the deep learning service provider (server), which can make the client's data leak to the server, resulting in serious privacy concerns. To address this issue, we propose a protocol named EPIDL to perform efficient and secure inference tasks on neural networks. This protocol enables the client and server to complete inference tasks by performing secure multi‐party computation (MPC) and the client's private data is kept secret from the server. The work in EPIDL can be summarized as follows: First, we optimized the convolution operation and matrix multiplication, such that the total communication can be reduced; Second, we proposed a new method for truncation following secure multiplication based on oblivious transfer and garbled circuits, which will not fail and can be executed together with the ReLU activation function; Finally, we replace complex activation function with MPC‐friendly approximation function. We implement our work in C++ and accelerate the local matrix computation with CUDA support. We evaluate the efficiency of EPIDL in privacy‐preserving deep learning inference tasks, such as the time to execute a secure inference on the MNIST dataset in the LeNet model is about 0.14 s. Compared with the state‐ofthe‐art work, our work is 1.8–98 faster over LAN and WAN, respectively. The experimental results show that our EPIDL is efficient and privacy‐preserving.