Inference Services Research Articles

FastPTM: Fast weights loading of pre-trained models for parallel inference service provisioning

Pre-trained models (PTMs) have demonstrated great success in a variety of NLP and CV tasks and have become a significant development in the field of deep learning. However, the large memory and high computational requirements associated with PTMs can increase the cost and time of inference, limiting their service provisioning in practical applications. To improve the Quality of Service (QoS) of PTM applications by reducing waiting and response times, we propose the FastPTM framework. This general framework aims to accelerate PTM inference services in a multi-tenant environment by reducing model loading time and switching overhead on GPUs. The framework utilizes a fast weights loading method based on weights and model separation of PTMs to efficiently accelerate parallel inference services in resource-constrained environments. Furthermore, an online scheduling algorithm is designed to reduce the inference service time. The results of the experiments indicate that FastPTM can improve the throughput of inference services by an average of 4x and up to 8.2x, while reducing the number of switches by 4.7x and the number of overtimes by 15.3x.

CollectNET: a web server for integrated inference of cell-cell communication network.

Cell-cell communication (CCC) through ligand-receptor (L-R) pairs forms the cornerstone for complex functionalities in multicellular organisms. Deciphering such intercellular signaling can contribute to unraveling disease mechanisms and enable targeted therapy. Nonetheless, notable biases and inconsistencies are evident among the inferential outcomes generated by current methods for inferring CCC network. To fill this gap, we developed collectNET (http://health.tsinghua.edu.cn/collectnet) as a comprehensive web platform for analyzing CCC network, with efficient calculation, hierarchical browsing, comprehensive statistics, advanced searching, and intuitive visualization. collectNET provides a reliable online inference service with prior knowledge of three public L-R databases and systematic integration of three mainstream inference methods. Additionally, collectNET has assembled a human CCC atlas, including 126 785 significant communication pairs based on 343 023 cells. We anticipate that collectNET will benefit researchers in gaining a more holistic understanding of cell development and differentiation mechanisms. Database URL: http://health.tsinghua.edu.cn/collectnet.

Internet of Conscious Things: Ontology-Based Social Capabilities for Smart Objects

Emerging distributed intelligence paradigms for the Internet of Things (IoT) call for flexible and dynamic reconfiguration of elementary services, resources and devices. In order to achieve such capability, this paper faces complex interoperability and autonomous decision problems by proposing a thorough framework based on the integration of the Semantic Web of Things (SWoT) and Social Internet of Things (SIoT) paradigms. SWoT enables low-power knowledge representation and autonomous reasoning at the edge of the network through carefully optimized inference services and engines. This layer provides service/resource management and discovery primitives for a decentralized collaborative social protocol in the IoT, based on the Linked Data Notifications(LDN) over Linked Data Platform on Constrained Application Protocol (LDP-CoAP). The creation and evolution of friend and follower relationships between pairs of devices is regulated by means of novel dynamic models assessing trust as a usefulness reputation score. The close SWoT-SIoT integration overcomes the functional limitations of existing proposals, which focus on either social device or semantic resource management only. A smart mobility case study on Plug-in Electric Vehicles (PEVs) illustrates the benefits of the proposal in pervasive collaborative scenarios, while experiments show the computational sustainability of the dynamic relationship management approach.

BCEdge: SLO-Aware DNN Inference Services With Adaptive Batch-Concurrent Scheduling on Edge Devices

BCEdge: SLO-Aware DNN Inference Services With Adaptive Batch-Concurrent Scheduling on Edge Devices

ESEN: Efficient GPU sharing of Ensemble Neural Networks

Ensemble neural networks are widely applied in cloud-based inference services due to their remarkable performance, while the growing demand for low-latency services leads researchers to pay more attention to the execution efficiency of these models, especially the device utilization. It is highly desirable to fully utilize GPUs by multiplexing different inference tasks on the same GPU with advanced sharing technique, such as Multi-Process-Service (MPS). However, we find it struggling when applying MPS to Ensemble Neural Networks, which consist of multiple related sub-models. The critical challenge in this predicament revolves around the efficient allocation of resources within an ensemble, aiming to minimize job completion time.To tackle this challenge, we initially examine the interplay among individual neural networks within an ensemble, outlining a guideline for achieving the shortest job completion time. Subsequently, we establish a mathematical model to formalize the resource requirements of each neural network. We introduce a search-based allocation algorithm designed to swiftly identify optimal solutions. Finally, we introduce ESEN, comprising the search-based resource allocation algorithm and efficient model execution mechanisms within PyTorch. ESEN is augmented with customized execution mechanisms for user-friendly implementation. Experimental results demonstrate that proposed ESEN can attain an efficiency improvement up to 17.84% and a GPU utilization increase of 28.09% compared to the default strategy. With the optimization of GPU resource allocation, ESEN significantly improves the efficiency of ensemble models. It provides a low-latency and high-accuracy solution for online interactive services.

Flexible Deployment of Machine Learning Inference Pipelines in the Cloud–Edge–IoT Continuum

Currently, deploying machine learning workloads in the Cloud–Edge–IoT continuum is challenging due to the wide variety of available hardware platforms, stringent performance requirements, and the heterogeneity of the workloads themselves. To alleviate this, a novel, flexible approach for machine learning inference is introduced, which is suitable for deployment in diverse environments—including edge devices. The proposed solution has a modular design and is compatible with a wide range of user-defined machine learning pipelines. To improve energy efficiency and scalability, a high-performance communication protocol for inference is propounded, along with a scale-out mechanism based on a load balancer. The inference service plugs into the ASSIST-IoT reference architecture, thus taking advantage of its other components. The solution was evaluated in two scenarios closely emulating real-life use cases, with demanding workloads and requirements constituting several different deployment scenarios. The results from the evaluation show that the proposed software meets the high throughput and low latency of inference requirements of the use cases while effectively adapting to the available hardware. The code and documentation, in addition to the data used in the evaluation, were open-sourced to foster adoption of the solution.

SecureTLM: Private inference for transformer-based large model with MPC

Transformer-based Large Models (TLM), such as generative pre-trained models (GPT), have become increasingly popular for practical applications through Deep Learning as a Service (DLaaS). They have been extensively used in natural language processing and computer vision. However, concerns regarding potential private data leakage arise with this type of inference service. While some private inference techniques can protect privacy, they often introduce high latency and approximate replacements in the design protocols, resulting in changes to the model structure and decreased accuracy. In this research, we present SecureTLM, a private inference method based on secure multi-party computation (MPC) that does not require modifications to the underlying model structure. SecureTLM offers protocols for crucial computations in TLM, such as Multiplication, Softmax, GeLU, and LayerNorm, without altering the model structure. Experimental results demonstrate that SecureTLM ensures data privacy, maintains correctness, and achieves efficiency in private inference tasks.

Proposed Fuzzy-Stranded-Neural Network Model That Utilizes IoT Plant-Level Sensory Monitoring and Distributed Services for the Early Detection of Downy Mildew in Viticulture

Novel monitoring architecture approaches are required to detect viticulture diseases early. Existing micro-climate decision support systems can only cope with late detection from empirical and semi-empirical models that provide less accurate results. Such models cannot alleviate precision viticulture planning and pesticide control actions, providing early reconnaissances that may trigger interventions. This paper presents a new plant-level monitoring architecture called thingsAI. The proposed system utilizes low-cost, autonomous, easy-to-install IoT sensors for vine-level monitoring, utilizing the low-power LoRaWAN protocol for sensory measurement acquisition. Facilitated by a distributed cloud architecture and open-source user interfaces, it provides state-of-the-art deep learning inference services and decision support interfaces. This paper also presents a new deep learning detection algorithm based on supervised fuzzy annotation processes, targeting downy mildew disease detection and, therefore, planning early interventions. The authors tested their proposed system and deep learning model on the grape variety of protected designation of origin called debina, cultivated in Zitsa, Greece. From their experimental results, the authors show that their proposed model can detect vine locations and timely breakpoints of mildew occurrences, which farmers can use as input for targeted intervention efforts.

6G Goal-Oriented Communications: How to Coexist with Legacy Systems?

6G will connect heterogeneous intelligent agents to make them natively operate complex cooperative tasks. When connecting intelligence, two main research questions arise to identify how artificial intelligence and machine learning models behave depending on (i) their input data quality, affected by errors induced by interference and additive noise during wireless communication; (ii) their contextual effectiveness and resilience to interpret and exploit the meaning behind the data. Both questions are within the realm of semantic and goal-oriented communications. With this paper, we investigate how to effectively share communication spectrum resources between a legacy communication system (i.e., data-oriented) and a new goal-oriented edge intelligence one. Specifically, we address the scenario of an enhanced Mobile Broadband (eMBB) service, i.e., a user uploading a video stream to a radio access point, interfering with an edge inference system, in which a user uploads images to a Mobile Edge Host that runs a classification task. Our objective is to achieve, through cooperation, the highest eMBB service data rate, subject to a targeted goal effectiveness of the edge inference service, namely the probability of confident inference on time. We first formalize a general definition of a goal in the context of wireless communications. This includes the goal effectiveness, (i.e., the goal achievability rate, or the probability of achieving the goal), as well as goal cost (i.e., the network resource consumption needed to achieve the goal with target effectiveness). We argue and show, through numerical evaluations, that communication reliability and goal effectiveness are not straightforwardly linked. Then, after a performance evaluation aiming to clarify the difference between communication performance and goal effectiveness, a long-term optimization problem is formulated and solved via Lyapunov stochastic network optimization tools to guarantee the desired target performance. Finally, our numerical results assess the advantages of the proposed optimization and the superiority of the goal-oriented strategy against baseline 5G-compliant legacy approaches, under both stationary and non-stationary communication (and computation) environments.

A novel open-source CADs platform for 3D CT pulmonary analysis

Computer-aided diagnosis (CAD) systems play vital roles in the early detection of pulmonary nodules for reducing lung cancer mortality rates. To provide better services for professional doctors, this paper proposes an efficient open-source CAD platform with flexible equipments, user-friendly interfaces, and completed functions for 3D CT pulmonary nodule analysis. For the platform’s design and implementation, we fully consider application scenarios and system requirements. The platform supplies core functions for (1) Basic Image Processing, (2) Intelligent Image Analysis, (3) Multi-View Image Visualization, (4) Report Editing and Generation, (5) User Information Management, and (6) Inference Service Monitoring. Specifically, other state-of-the-art or user-defined algorithms can be integrated as plugin modules with no interference for system architecture. System evaluation with use-case testing demonstrates the effectiveness and universality of the proposed platform.

QAVAN: Query-answering approach for actionable numerical relationships over Knowledge Graphs

Semantic Web (SW) technologies are suitable to represent taxonomic knowledge in Knowledge Graphs (KG). However, cases that do not fall in this category such as numerical relationships (e.g., algebraic operations or unit conversion) that can enrich the KGs data are poorly represented. Current approaches mostly trigger these relationships in a materialised fashion (i.e., perform all the relationships in advance), which leads to storage and updating problems. Besides, these relationships are often not represented using linked data formats ignoring the FAIR and LOD guidelines, which leads to issues of interoperability, reproducibility and adoption. In this work, we propose QAVAN, a query-answering approach that computes efficiently only the necessary numerical relationships relevant to answer to a given query. QAVAN exploits SW technologies such as SHACL for representing and computing these relationships while plays in favour of FAIR and LOD principles. We evaluated and compared QAVAN against a materialised approach using two real datasets extracted from two very well-known data sources, DBpedia and MeteoFrance. Results expose the numerous benefits of QAVAN in terms of execution time, memory usage, and disk space in different scenarios for example against big datasets. We also evaluated two QAVAN extensions, the nested numerical relationships and the inference services. Our results show that these extensions allow to apply the numerical relationships for a greater number of instances to provide more query results.

Collaborative Inference Acceleration Integrating DNN Partitioning and Task Offloading in Mobile Edge Computing

In mobile edge computing environment, intelligent inference services driven by DNN are highly sensitive to latency. Recently, collaborative inference between User Devices and Edge Servers (ESs) based on Deep Neural Networks (DNN) partition has achieved success in service acceleration. However, most of the existing collaborative acceleration schemes are partitioned for a single DNN inference task, which cannot quickly make partition decisions for a set of concurrent inference tasks, and often sacrifice inference accuracy. In addition, due to the limited resources of ESs, there is resource competition among concurrent requests, which makes the partitioned tasks cannot be offloaded to ESs in time for processing. Therefore, designing an efficient offloading scheme becomes essential. The task offloading schemes based on deep reinforcement learning can solve complex decision-making problems in high-dimensional state space, but they have problems such as insufficient sample diversity and easily falling into local optimum. In this paper, a Collaborative Inference Acceleration Scheme integrating DNN Partitioning and Task Offloading (CIAS-PnO) is proposed. First, while ensuring inference accuracy, the Collaborative DNN Layer Partitioning (CDLP) algorithm is designed with the goal of optimal latency. CDLP can reduce the problem scale of concurrent inference tasks partition by pruning operation and determine the partition decisions in time. Then, the Distributed Soft Actor-Critic (SAC)-based Partition Task Offloading algorithm (DSACO) is designed. DSACO supports SAC Agents to explore samples in parallel and share learning experiences, and uses the automatic entropy adjustment mechanism to improve the exploration efficiency of Agents, so as to avoid falling into local optimum and achieve efficient offloading of partition tasks. Experimental results on DNN benchmarks show that compared with the baseline acceleration schemes, CIAS-PnO achieves more than 19.8% acceleration performance improvement, and has higher convergence performance and task success rate.

FPCNN: A fast privacy-preserving outsourced convolutional neural network with low-bandwidth

Convolutional neural networks (CNNs) have been widely employed in image and video recognition tasks due to their superior performance. With the increasing popularity of machine learning as a service, CNN model owners often outsource their models to the cloud to provide inference services. However, the user data and outsourced models processed by cloud-based CNN inference applications suffer from privacy leakage issues. Therefore, several studies have proposed different privacy-preserving CNN architectures. Most of these approaches are based on heavyweight cryptographic methods, which require significant computational resources. In this study, we propose a novel scheme, a Fast Privacy-Preserving Outsourced Convolutional Neural Network with Low Bandwidth (FPCNN), to provide secure inference services on the cloud using outsourced data and models. Specifically, for linear CNN layers, FPCNN significantly reduces the bandwidth requirements between servers by applying scaled noise models and scaled secret sharing. For nonlinear CNN layers, we first optimize the multiplication operation under secret sharing to save bandwidth. Subsequently, based on the nonlinear input characteristics of CNNs, we introduce a secure parallel rectified linear unit (ReLU) computation protocol that significantly reduces the number of communication rounds. Finally, the experimental results demonstrate that FPCNN achieves a speedup of 2.66× and a 32.8× reduction in bandwidth compared with state-of-the-art methods.

InferFair: Towards QoS-aware scheduling for performance isolation guarantee in heterogeneous model serving systems

With the popularity of Deep Neural Network (DNN) models in diverse fields, DNN inference services have been widely deployed on cloud for resource-limited devices to support intelligent applications. Serving DNN inference often requires GPU acceleration to meet latency-sensitive interactive targets. A common approach to improving GPU utilization is to let multiple models share a GPU. However, this approach probably degrades both the responsiveness and throughput of model serving systems, due to that concurrent DNN inference tasks contend for GPU resources. In addition, interferences among heterogeneous DNN inference tasks probably incur performance isolation problem that heterogeneous models suffer from different levels of serving performance degradation. Existing works fail to ensure performance isolation among users of heterogeneous DNN models. To solve the aforementioned problem, we propose InferFair, a QoS-aware scheduling framework for ensuring performance isolation in heterogeneous model serving systems. InferFair focuses on two key designs: (1) periodically estimating effective throughput requirements of all active models online and (2) applying fine-grained adjustments to minimize the impact differences of GPU sharing on heterogeneous model services. We conduct intensive experiments on a variety of DNN models to demonstrate the effectiveness of InferFair. Compared to a prior competitor named Clockwork, InferFair alleviates the performance isolation problem by up to 1.7×, as well as improving the overall goodput by up to 25.6%.

Manto: A Practical and Secure Inference Service of Convolutional Neural Networks for IoT

As convolutional neural networks (CNNs) exhibit remarkable performance in various inference tasks, it is increasingly important to enable Internet of Things (IoT) devices to perform CNN-based applications. Many companies provide their carefully-trained neural networks as inference services for resource-constrained clients (e.g., IoT devices). However, the use of CNN inference in many IoT applications raises privacy concerns. Cryptographic inference services provide a way to perform neural inference efficiently and, at the same time, preserve both the privacy of the client’s input data and the server’s proprietary model. Unfortunately, the existing solutions incur severe latency costs, stemming mostly from nonlinear activations such as ReLUs, which make them still unsuitable for deployment in real IoT devices. In this paper, we propose Manto, a secure inference system of convolutional neural networks for IoT. Manto makes the following two specific efforts by combining the insights of machine learning and cryptography. First, we customize different quadratic activation functions to replace specific ReLU layers and further propose a sliding-window-based fine-tuning method to produce CNN models involving no or few ReLUs. These techniques allow us to speedup cryptographic inference and guarantee inference accuracy. Second, we develop a series of cryptographic protocols that support ReLU activations and its approximation variants (i.e., polynomial activations), which purely rely on the lightweight secret sharing techniques in the online execution and can well cope with the above-entioned optimized CNN models in the ciphertext domain. Our experimental results show Manto obtains state-of-the-art performance, reducing online inference latency by 66.2%~87.7% over prior works on CIFAR-100 and TinyImageNet datasets.

FEBench: A Benchmark for Real-Time Relational Data Feature Extraction

As the use of online AI inference services rapidly expands in various applications (e.g., fraud detection in banking, product recommendation in e-commerce), real-time feature extraction (RTFE) systems have been developed to compute the requested features from incoming data tuples in ultra-low latency. Similar to relational databases, these RTFE procedures can be expressed using SQL-like languages. However, there is a lack of research on the workload characteristics and specialized benchmarks for RTFE, especially in comparison with existing database workloads and benchmarks (e.g., concurrent transactions in TPC-C). In this paper, we study the RTFE workload characteristics using over one hundred real datasets from open repositories (e.g. Kaggle, Tianchi, UCI ML, KiltHub) and those from 4Paradigm. The study highlights the significant differences between RTFE workloads and existing database benchmarks in terms of application scenarios, operator distributions, and query structures. Based on these findings, we propose to develop a realtime feature extraction benchmark named FEBench based on the four important criteria for a domain-specific benchmark proposed by Jim Gray. FEBench consists of selected representative datasets, query templates, and an online request simulator. We use FEBench to evaluate the effectiveness of feature extraction systems including OpenMLDB and Flink and find that each system exhibits distinct advantages and limitations in terms of overall latency, tail latency, and concurrency performance.

SecGNN: Privacy-Preserving Graph Neural Network Training and Inference as a Cloud Service

Graphs are widely used to model the complex relationships among entities. As a powerful tool for graph analytics, graph neural networks (GNNs) have recently gained wide attention due to its end-to-end processing capabilities. With the proliferation of cloud computing, it is increasingly popular to deploy the services of complex and resource-intensive model training and inference in the cloud due to its prominent benefits. However, GNN training and inference services, if deployed in the cloud, will raise critical privacy concerns about the information-rich and proprietary graph data (and the resulting model). While there has been some work on secure neural network training and inference, they all focus on convolutional neural networks handling images and text rather than complex graph data with rich structural information. In this paper, we design, implement, and evaluate SecGNN, the first system supporting privacy-preserving GNN training and inference services in the cloud. SecGNN is built from a synergy of insights on lightweight cryptography and machine learning techniques. We deeply examine the procedure of GNN training and inference, and devise a series of corresponding secure customized protocols to support the holistic computation. Extensive experiments demonstrate that SecGNN achieves comparable plaintext training and inference accuracy, with promising performance.

Optimizing Secure Decision Tree Inference Outsourcing

Outsourcing decision tree inference services to the cloud is highly beneficial, yet raises critical privacy concerns on the proprietary decision tree of the model provider and the private input data of the client. In this paper, we design, implement, and evaluate a new system that allows highly efficient outsourcing of decision tree inference. Our system significantly improves upon prior art in the overall online end-to-end secure inference service latency at the cloud as well as the local-side performance of the model provider. We first present a new scheme which securely shifts most of the processing of the model provider to the cloud, resulting in a substantial reduction on the model provider&#x0027;s performance complexities. We further devise a scheme which substantially optimizes the performance for secure decision tree inference at the cloud, particularly the communication round complexities. The synergy of these techniques allows our new system to achieve up to <inline-formula><tex-math notation="LaTeX">$8 \times$</tex-math></inline-formula> better overall online end-to-end secure inference latency at the cloud side over realistic WAN environment, as well as bring the model provider up to <inline-formula><tex-math notation="LaTeX">$19 \times$</tex-math></inline-formula> savings in communication and <inline-formula><tex-math notation="LaTeX">$18 \times$</tex-math></inline-formula> savings in computation.

Energy-Aware, Device-to-Device Assisted Federated Learning in Edge Computing

The surging of deep learning brings new vigor and vitality to shape the prospect of intelligent Internet of Things (IoT), and the rise of edge intelligence enables provisioning real-time deep neural network (DNN) inference services for mobile users. To perform efficient and effective DNN model training in edge computing environments while preserving training data security and privacy of IoT devices, federated learning has been envisioned as an ideal learning paradigm for this purpose. In this paper, we study energy-aware DNN model training in edge computing. We first formulate a novel energy-aware, Device-to-Device (D2D) assisted federated learning problem with the aim to minimize the global loss of a training DNN model, subject to bandwidth capacity on an edge server and energy capacity on each IoT device. We then devise a near-optimal learning algorithm for the problem when the training data follows the i.i.d. data distribution. The crux of the proposed algorithm is to explore using the energy of neighboring devices of each device for its local model uploading, by reducing the problem to a series of weighted maximum matching problems in corresponding auxiliary graphs. We also consider the problem without the assumption of the i.i.d. data distribution, for which we propose an efficient heuristic algorithm. We finally evaluate the performance of the proposed algorithms through experimental simulations. Experimental results show that the proposed algorithms are promising.

MPDM: A Multi-Paradigm Deployment Model for Large-Scale Edge-Cloud Intelligence

The development of cloud and edge computing has enabled the easy access of artificial intelligence (AI) services for massive heterogeneous and resource-constrained devices. Particularly, computation-intensive AI services can be orchestrated and deployed in the cloud or edge according to varying performance and cost requirements. Nonetheless, the improved accessibility of deep learning (DL) model variants and the evolving of computational intelligence paradigms pose great challenges for orchestrating large-scale DL inference services in the cloud-edge continuum. Focusing on cloud or edge-based deployment, existing work on multi-variant service orchestration often has a limited solution space of deployment plans. To address this limitation, we first propose a novel Multi-Paradigm Deployment Model (MPDM) for service orchestration, which not only considers the model variants but also allows the co-existence of multiple paradigms for large-scale inference service deployment. The service deployment in the MPDM model is then formulated as a multi-objective optimization problem of seeking a better trade-off among the system accuracy, service scale and deployment cost. To solve the multi-objective optimization, we further propose a weighted metric based constructive heuristic algorithm (WCH), which can efficiently obtain an approximately optimal Pareto frontier. Extensive experimental results have validated the effectiveness and efficiency of WCH, and revealed the impacts of both multi-paradigm deployment and edge-cloud collaborative intelligence (ECCI) paradigm on large-scale DL serving systems.