Request Latency Research Articles

Edge Caching Placement Strategy based on Evolutionary Game for Conversational Information Seeking in Edge Cloud Computing

In Internet applications, network conversation is the primary communication between the user and server. The server needs to efficiently and quickly return the corresponding service according to the conversation sent by the user to improve the users’ Quality of Service. Thus, Conversation Information Seeking (CIS) research has become a hot topic today. In Cloud Computing (CC), a central service mode, the conversation is transmitted between the user and the remote cloud over a long distance. With the explosive growth of Internet applications, network congestion, long-distance communication, and single point of failure have brought new challenges to the centralized service mode. People put forward Edge Cloud Computing (ECC) to meet the new challenges of the centralized service mode of CC. As a distributed service mode, ECC is an extension of CC. By migrating services from the remote cloud to the network edge closer to users, ECC can solve the above challenges in CC well. In ECC, people solve the problem of CIS through edge caching. The current research focuses on designing the edge cache strategy to achieve more predictable caching. In this article, we propose an edge cache placement method Evolutionary Game based Caching Placement Strategy (EG-CPS). This method consists of three modules: the user preference prediction module, the content popularity calculation module, and the cache placement decision module. To maximize the predictability of the cache strategy, we are committed to optimizing the cache hit rate and service latency. The simulation experiment compares the proposed strategy with several other cache strategies. The experimental results illustrate that EG-CPS can reduce up to 2.4% of the original average content request latency, increase the average direct cache hit rate by 1.7%, and increase the average edge cache hit rate by 3.3%.

Organic 6G networks: ultra-flexibility through extensive stateless functional split

AbstractWith the increase in hardware performance, the 5G mobile network architecture shifted from physical components to software-only micro-services. The very modular network functions can be deployed flexibly on commodity hardware. However, the extensive modularity of these network functions is increasing the number of managed entities, and the core network request latency. Also, it requires extensive procedures to be able to re-select the components for specific devices, a fundamental condition for a potential system scale-down. In this paper, we propose a new organic 6G network architecture that handles these challenges through a new functionality split based on the experience of IT software services. Furthermore, we provide an analysis based on main 5G procedures, showing that the newly proposed architecture is handling the re-selection of functionality significantly better, which is a cornerstone of high-speed scaling (especially scaling-out), as well as migration of functionality and users.

PcGC: A Parity-Check Garbage Collection for Boosting 3-D NAND Flash Performance

Garbage collection or GC running in the controller of 3D NAND flash-based solid-state disks -SSDs -plays a critical role in the performance of storage systems. SSD manufacturers have developed various garbage collection solutions based on internal data movement or IDM to mitigate the impacts of garbage collection on request latency. Due to the circuit characteristics of flash memory, existing IDM-based garbage collection strategies are restricted by the page parity during data movement: odd pages must be migrated to odd pages, and even pages to even pages. When migrating two consecutive pages with the same parity, the free page between the two migrated pages will be wasted after the migration is complete. This ever-increasing page waste problem inevitably deteriorates the storage space utilization of flash memory, thereby degrading the overall performance of 3D NAND flash-based SSDs. To address this issue, we propose a parity-check garbage collection scheme called PcGC to revamp SSD performance by alleviating page waste during garbage collection. We build a parity-check unit in PcGC to facilitate checking the parity of migrated valid pages and destination pages. According to the parity results offered by the parity-check unit, PcGC dynamically adjusts the migration order of valid pages during the course of garbage collection. In doing so, PcGC fundamentally averts page waste caused by the page parity restriction, thereby enhancing 3D NAND flash performance. We quantitatively evaluate the performance of PcGC in terms of wasted pages, storage utilization, GC counts, write amplification, and average response time. We compare PcGC against the two state-of-the-art schemes -Amphibian and TTflash. The experimental results derived from the nine real-world workload traces unfold that compared with Amphibian and TTflash, (1) PcGC curtails the number of wasted pages by up to 91.4% with an average of 53.75%, (2) cuts back the number of GC counts by up to 52.2% with an average of 11.9%, and (3) slashes average write response time by up to 77.8% with an average of 13.0%.

Network shortcut in data plane of service mesh with eBPF

In recent years, the adoption of the service mesh as a dedicated infrastructure layer to support cloud-native systems has gained significant popularity. Service meshes involve the incorporation of proxies to handle communication between microservices, thereby speeding up the development and deployment of microservice applications. However, the use of service meshes also increases the request latency because they elongate the packet transmission between services. After investigating the transmission path of packets in a representative service mesh Istio, we observed that the service mesh dedicates approximately 25% of its time to packet transmission in the Linux kernel network stack. To shorten this process, we propose a non-intrusive solution that enables packets to bypass the kernel network stack through the implementation of socket redirection and tc (traffic control) redirection with eBPF (extended Berkeley Packet Filter). We also conduct comprehensive experiments on the widely-used Istio. The evaluation results show that our approach can significantly reduce the request latency by up to 21%. Furthermore, our approach decreases CPU usage by 1.73% and reduces memory consumption by approximately 0.98% when compared to the original service mesh implementation.

UAV-Assisted Caching Strategy Based on Content Cache Pricing in Vehicular Networks

A UAV-assisted caching strategy considering content cache pricing in vehicular networks is proposed to address the problem of high communication load and high backhaul link overhead in vehicular networks. Consider a traffic scenario consisting of a content provider (CP), a network operator (NO), and multiple mobile users, where the NO has a set of cache-enabled roadside units (RSUs) and an unmanned aerial vehicle (UAV). The CP leases some popular contents to the NO for its benefit and the NO places this leased content in its RSU’s local cache to save expensive backhaul transmission overhead and latency. However, both NO and CP are selfish and their interests conflict with each other because they have opposing expectations for content pricing. In order to take into account the interests of both, this paper defines the utilities of CP and MNO and uses the Stackelberg game framework to model the competition between the two entities, where CP acts as a leader and sets the rental price of the content and NO acts as a follower responding to CP’s actions. An iteration-based dynamic programming algorithm is also designed to find the Stackelberg equilibrium. Meanwhile, a caching-capable UAV is introduced into the vehicular network and, based on this, a Dijkstra-based path planning algorithm is designed to further increase the total utility of NO by optimizing the trajectory of the UAV. The simulation results show that the strategy in this paper can reasonably allocate the benefits of CP and NO, reduce the average request delay, and increase the utility of NO; for example, we reduced the request latency for vehicle users by 27% and increased the total utility of NO by 13%.

Caching Placement Optimization Strategy Based on Comprehensive Utility in Edge Computing

With the convergence of the Internet of Things, 5G, and artificial intelligence, limited network bandwidth and bursts of incoming service requests seem to be the most important factors affecting user experience. Therefore, caching technology was introduced. In this paper, a caching placement optimization strategy based on comprehensive utility (CPOSCU) in edge computing is proposed. Firstly, the strategy involves quantifying the placement factors of data blocks, which include the popularity of data blocks, the remaining validity ratio of data blocks, and the substitution rate of servers. By analyzing the characteristics of cache objects and servers, these placement factors are modeled to determine the cache value of data blocks. Then, the optimization problem for cache placement is quantified comprehensively based on the cache value of data blocks, data block retrieval costs, data block placement costs, and replacement costs. Finally, to break out of the partial optimal solution for cache placement, a penalty strategy is introduced, and an improved tabu search algorithm is used to find the best edge server placement for cached objects. Experimental results demonstrate that the proposed caching strategy enhances the cache service rate, reduces user request latency and system overhead, and enhances the user experience.

Buffer-Based High-Coverage and Low-Overhead Request Event Monitoring in the Cloud

Request latency directly affects the performance of modern cloud applications. Due to various causes in hosts and networks, can suffer from latency anomalies (), which may violate the Service-Level Agreement. However, existing performance monitoring tools have incomplete coverage and inconsistent semantics for monitoring and cannot accurately diagnose . This paper presents, a high-coverage and low-overhead event monitoring system, which monitors <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">buffers</i> to capture most -related abnormal events with consistent -level semantics in the end-to-end datapath of . First, models the datapath of as a buffer chain and defines events based on three properties of buffers, so as to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">end-to-end monitor</i> the root causes of . Then, to achieve <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">consistent semantics</i> for captured events, designs a -level semantics injection mechanism to make events captured in networks have the victim requests’ ID. Finally, offloads the semantics operations and event collection in software to SmartNICs for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">low CPU overhead</i> . We have implemented on commodity SmartNICs and programmable switches. Evaluation results show that can diagnose 98% with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&lt;$</tex-math> </inline-formula> 0.08% network bandwidth overhead and 0.6% application throughput decline.

CEC: A Containerized Edge Computing Framework for Dynamic Resource Provisioning

Container has been widely used in application development and management systems. However, there are two major challenges faced in the real deployment at edge servers. The varying workload of service requests and the startup delay of containers force a flexible resource provisioning scheme in containerized edge computing. To this end, we propose CEC, a containerized edge computing framework for dynamic resource provisioning, and especially for the smart connected community where exists multiple intelligent applications. CEC integrates workload prediction and resource pre-provisioning to enable low latency of user service requests and high utilization of edge resources. First, we present an online periodic request prediction algorithm. Then, we designed a control-based resource pre-provisioning algorithm based on the predicted request distribution, which is a self-adaptive controller to tune the resource for containers. We evaluate the performance of CEC by simulation and system experiments. The simulation experiment shows that the prediction accuracy of the proposed algorithm is higher than other two prediction algorithms. The testbed experiments demonstrate that the control-based resource pre-provisioning algorithm has low service latency and high resource utilization compared with baselines.

Using Microbenchmark Suites to Detect Application Performance Changes

Software performance changes are costly and often hard to detect pre-release. Similar to software testing frameworks, either application benchmarks or microbenchmarks can be integrated into quality assurance pipelines to detect performance changes before releasing a new application version. Unfortunately, extensive benchmarking studies usually take several hours which is problematic when examining dozens of daily code changes in detail; hence, trade-offs have to be made. Optimized microbenchmark suites, which only include a small subset of the full suite, are a potential solution for this problem, given that they still reliably detect the majority of the application performance changes such as an increased request latency. It is, however, unclear whether microbenchmarks and application benchmarks detect the same performance problems and one can be a proxy for the other. In this paper, we explore whether microbenchmark suites can detect the same application performance changes as an application benchmark. For this, we run extensive benchmark experiments with both the complete and the optimized microbenchmark suites of two time-series database systems, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">InfluxDB</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VictoriaMetrics</i> , and compare their results to the results of corresponding application benchmarks. We do this for 70 and 110 commits, respectively. Our results show that it is not trivial to detect application performance changes using an optimized microbenchmark suite. The detection (i) is only possible if the optimized microbenchmark suite covers all application-relevant code sections, (ii) is prone to false alarms, and (iii) cannot precisely quantify the impact on application performance. For certain software projects, an optimized microbenchmark suite can, thus, provide fast performance feedback to developers (e.g., as part of a local build process), help estimating the impact of code changes on application performance, and support a detailed analysis while a daily application benchmark detects major performance problems. Thus, although a regular application benchmark cannot be substituted for both studied systems, our results motivate further studies to validate and optimize microbenchmark suites.

Constraint‐aware and multi‐objective optimization for micro‐service composition in mobile edge computing

AbstractAs a new paradigm of distributed computing, mobile edge computing (MEC) has gained increasing attention due to its ability to expand the capabilities of centralized cloud computing. In MEC environments, a software application typically consists of multiple micro‐services, which can be composed together in a flexible manner to achieve various user requests. However, the composition of micro‐services in MEC is still a challenging research issue arising from three aspects. Firstly, composite micro‐services constructed by ignoring the processing capabilities of different micro‐services may cause waste of edge resources. Secondly, edge servers' limitations in terms of computational power can easily cause service occupancy between composite micro‐services, severely affecting the user experience. Thirdly, in dynamic and unstable mobile environments, different edge users have different sensitivities to request latency, which increases the complexity of micro‐service composition. In order to improve edge resource utilization and user experience on micro‐service invocations, in this paper, we comprehensively consider the above three factors, and we first model the micro‐services composition problem in MEC as a constrained multi‐objective optimization problem. Then, a micro‐service composition optimization method M3C combining graph search and branch‐and‐bound strategy is proposed to find a composition solution set with low energy consumption and high success rate for multiple edge users. Finally, we perform a series of experiments on two widely used datasets. Experimental results show that our proposed approach significantly outperforms the four competing baseline approaches, and that it is sufficiently efficient for practical deployment.

Queries allocation in WSNs with fuzzy control system

AbstractProcessing queries in wireless sensor networks causes quality of service challenges such as data latency and validity. In fact, this paper is going to shed light on identifying the right trade‐off between latency and the freshness of the data. For this purpose, a fuzzy logic‐based decision‐making system that assigns incoming queries either to the sensor network, where the queries are waiting to be processed into a queue, or a central database which provides stored and potentially stale data or to a cache server which in turn provides acceptable latency and recent data, is provided. Thus, a new rule‐based system decision is introduced, that minimizes the request latency and returns the most recent data. In addition, a comparison between the approach and other work that is restricted to the database server and sensor network with no focus on the caching server is provided. This paper is going to carry on with the theory of Markov Chain which is more complicated compared to the fuzzy logic system. This approach is confirmed in the basis of an experience is developed with python.

Overbooking-Enabled Task Scheduling and Resource Allocation in Mobile Edge Computing Environments

Mobile Edge Computing (MEC) is proposed to solve the needs of Internet of Things (IoT) users for high resource utilization, high reliability and low latency of service requests. However, the backup virtual machine is idle when its primary virtual machine is running normally, which will waste resources. Overbooking the backup virtual machine under the above circumstances can effectively improve resource utilization. First, these virtual machines are deployed into slots randomly, and then some tasks with cooperative relationship are offloaded to virtual machines for processing. Different deployment locations have different resource utilization and average service response time. We want to find a balanced solution that minimizes the average service response time of the IoT application while maximizing resource utilization. In this paper, we propose a task scheduler and exploit a Task Deployment Algorithm (TDA) to obtain an optimal virtual machine deployment scheme. Finally, the simulation results show that the TDA can significantly increase the resource utilization of the system, while reducing the average service response time of the application by comparing TDA with the other two classical methods. The experimental results confirm that the performance of TDA is better than that of other two methods.

Challenges in service discovery for microservices deployed in a Kubernetes cluster – a case stud

With Kubernetes emerging as one of the most popular infrastructures in the cloud-native era, the utilization of containerization and tools alongside Kubernetes is steadily gaining traction. The main goal of this paper is to evaluate the service discovery mechanisms and DNS management (CoreDNS) of Kubernetes, and to present a general study of an experiment on service discovery challenges. In large scale Kubernetes clusters, running pods, services, requests, and workloads can be substantial. The increased number of HTTP-requests often result in resource utilization concerns, e.g., spikes of errors [24], [25]. This paper investigates potential optimization strategies for enhancing the performance and scalability of CoreDNS in Kubernetes. We propose a solution to address the concerns related to CoreDNS and provide a detailed explanation of how our implementation enhances service discovery functionality. Experimental results in a real-world case show that our solution for the CoreDNS ensures consistency of the workload. Compared with the default CoreDNS configuration, our customized approach achieves better performance in terms of number of errors for requests, average latency of DNS requests, and resource usage rate.

Latency Minimization Using an Adaptive Load Balancing Technique in Microservices Applications

Advancements in cloud computing and virtualization technologies have revolutionized Enterprise Application Development with innovative ways to design and develop complex systems. Microservices Architecture is one of the recent techniques in which Enterprise Systems can be developed as fine-grained smaller components and deployed independently. This methodology brings numerous benefits like scalability, resilience, flexibility in development, faster time to market, etc. and the advantages; Microservices bring some challenges too. Multiple microservices need to be invoked one by one as a chain. In most applications, more than one chain of microservices runs in parallel to complete a particular requirement To complete a user’s request. It results in competition for resources and the need for more inter-service communication among the services, which increases the overall latency of the application. A new approach has been proposed in this paper to handle a complex chain of microservices and reduce the latency of user requests. A machine learning technique is followed to predict the weighting time of different types of requests. The communication time among services distributed among different physical machines are estimated based on that and obtained insights are applied to an algorithm to calculate their priorities dynamically and select suitable service instances to minimize the latency based on the shortest queue waiting time. Experiments were done for both interactive as well as non interactive workloads to test the effectiveness of the solution. The approach has been proved to be very effective in reducing latency in the case of long service chains.

Application Performance Monitoring System Design Using Opentelemetry and Grafana Stack

The increasingly massive use of digital technology requires that the application architecture be designed to have high availability and reliability. This is because when an application cannot be accessed, it will cause no small loss to the organization. Therefore, the development and operation teams must be able to detect when their system is not working well. For that, we need a system that can monitor application performance. In this research, a system is developed to collect telemetry data, namely metrics and traces from an online donation backend application based on the REST API. OpenTelemetry produces telemetry as an open-source telemetry instrumentation tool. Then the telemetry data is collected by the OpenTelemetry Collector which is then stored on the backend of each telemetry. Metrics are sent to Prometheus and traces are sent to Jaeger. The data metrics collected are throughput, request latency, and error rate which are visualized using the Grafana dashboard. The test results show that the monitoring system can collect real-time metrics data with an average delay of 13,8 seconds. The system can also detect when an anomaly occurs in the app and sends notifications via Slack. In addition, the trace data collected can be used to simplify the debugging process when an error occurs in the application. However, the implementation of OpenTelemetry in a REST API-based backend application to monitor metrics and traces has a negative impact on the performance of the application itself, which can reduce the number of request throughput with an average decrease of 23.32% and increase request latency with an average increase of 22.80%.

IMPROVING THE EFFICIENCY OF AUTOMATION THE SCALING OF MICROSERVICES IN THE KUBERNETES CONTAINERIZED APPLICATION MANAGEMENT SYSTEM

Automatic container scaling in Kubernetes plays an important role in handling incoming requests from web application users. This article analyzes the stages of container scaling, their initialization and subsequent software launch. In order to achieve low latency of user requests in the case of dynamic load, the analysis of the process of automatic scaling of containers and the factors that will affect the scaling process was carried out. Having a full understanding of the processes and mechanisms by which scaling takes place, it became possible to develop a method and strategy for cleaning obstacles that slow down the autoscaling process itself and at the same time preserve the necessary properties of the existing scaling. Acceleration of scaling of containers, which will directly affect the speed of web services, becomes possible precisely because of the elimination of the delay in automatic scaling of containers. The work considered scaling optimization using not only container pre-creation networks, but also the use of container images, which enable the sharing of linked libraries in memory and the extension of Kubernetes’ declarative configuration management approach for parallel creation of multiple container instances. Based on the obtained research results, a method for optimizing the automatic scaling of containerized applications by eliminating the delay during a cold start has been developed. This latency manifests itself in the case of microservice autoscaling, where Kubernetes is expected to scale containers horizontally by creating additional replicas to the required number to handle the required traffic from the outside. The delay caused by the autoscaler affects the processing time of the user’s web service requests.

Integrating deep reinforcement learning with pointer networks for service request scheduling in edge computing

With the increasing popularity of edge computing, service providers are more likely to deploy services at the edge of the network to reduce the latency of service requests. However, the resources offered by edge servers are extremely limited compared to those in the cloud. Therefore, a challenging issue in edge computing is how to sufficiently utilize service resources at the edge to satisfy as many service requests as possible. Existing service request scheduling methods mainly use a single optimization objective, e.g., resource utilization or running time. In this paper, the issue of service request scheduling with multiple requests is modeled as a sequential problem, where multiple optimization objectives, including resource utilization, running time, and waiting time, are involved. A reinforcement learning model with pointer networks is proposed to construct scheduling policies. Experiments conducted on three representative real-world datasets show that our proposed approach outperforms several state-of-the-art methods on the three metrics.

A priority queue-based telemonitoring system for automatic diagnosis of heart diseases in integrated fog computing environments.

Various studies have shown the benefits of using distributed fog computing for healthcare systems. The new pattern of fog and edge computing reduces latency for data processing compared to cloud computing. Nevertheless, the proposed fog models still have many limitations in improving system performance and patients' response time.This paper, proposes a new performance model by integrating fog computing, priority queues and certainty theory into the Edge computing devices and validating it by analyzing heart disease patients' conditions in clinical decision support systems (CDSS). In this model, a Certainty Factor (CF) value is assigned to each symptom of heart disease. When one or more symptoms show an abnormal value, the patient's condition will be evaluated using CF values in the fog layer. In the fog layer, requests are categorized in different priority queues before arriving into the system. The results demonstrate that network usage, latency, and response time of patients' requests are respectively improved by 25.55%, 42.92%, and 34.28% compared to the cloud model. Prioritizing patient requests with respect to CF values in the CDSS provides higher system Quality of Service (QoS) and patients' response time.

IoT Edge Caching: Taxonomy, Use Cases and Perspectives

Low latency is an essential constraint for many emerging Internet of Things (IoT) applications. Cloud-based solutions can impose excessive network and processing latency that might hinder the system's operation. On the other hand, offloading tasks to the edge layer - close to where data is consumed - might be unfeasible in many situations due to the low computational resources of edge devices or because the system utilizes third-party applications. One potential solution is caching data on the edge of the network, thus decreasing the latency of subsequent requests, saving bandwidth, and avoiding redundant computation. At the same time, caching IoT data on the edge has its challenges and features that deviate from the standard methods of network caching, such as the content dynamicity and the power constraints. This paper performs a comprehensive review of the state-of-art in IoT Edge Caching and proposes a novel taxonomy focused on five orthogonal features of edge caching: placement, distance, strategy, metrics, and design. Moreover, we illustrate the characteristics identified in five prevalent use cases for the IoT. Finally, we conduct an in-depth analysis and performance evaluation of edge caching deployed in a real Structural Health Monitoring (SHM) scenario. Results show that different caching strategies decrease the system latency by more than 95 percent.

Deterministic Latency/Jitter-Aware Service Function Chaining Over Beyond 5G Edge Fabric

Deterministic Networking (DetNet) has recently attracted much attention. It aims at studying the deterministic bounded latency and low latency variation for time-sensitive applications (e.g., industrial automation). To improve the quality of service (QoS) guarantee and make the network management efficient, it is desirable for Internet Service Provider (ISP) to obtain an optimal service function chain (SFC) provision strategy while providing deterministic service performance for the time-sensitive applications. In this paper, we will study the deterministic SFC lifetime management problem in beyond 5G edge fabric with the objective of maximizing the overall profits and ensuring the deterministic latency and jitter of SFC requests. We first formulate this problem as a mathematical model with the maximal profits for ISP. Then, the novel Deterministic SFC Deployment algorithm (Det-SFCD) and SFC Adjustment algorithm (Det-SFCA) due to traffic load variation are proposed to efficiently solve the SFC lifetime management problem. Extensive simulation results show that our proposed algorithms can achieve better performance in terms of SFC request acceptance rates, overall profits and latency variation compared with the benchmark algorithm.