Network Traffic Cost Research Articles

NBFTP: a dedicated data transfer system for remote astronomical observation at Dome A* *The source code has been uploaded on gitee ( https://gitee.com/AstroTJU/NBFTP ) and china-vo ( https://code.china-vo.org/ast3/nbftp ).

Dome A, Antarctica, has been thought to be one of the best astronomical sites on the Earth for decades. Since it was first visited by astronomers in 2008, dozens of facilities for astronomical observation and site testing were deployed. Due to its special geographical location, the data and message exchange between Dome A and the domestic control center could only depend on Iridium. Because the link bandwidth of Iridium is extremely limited, the network traffic cost is quite expensive and the network is rather unstable, the commonly used data transfer tools, such as rsync and scp, are not suitable in this case. In this paper, we design and implement a data transfer tool called NBFTP (narrow bandwidth file transfer protocol) for the astronomical observation of Dome A. NBFTP uses a uniform interface to arrange all types of data and matches specific transmission schemes for different data types according to rules. Break-point resuming and extensibility functions are also implemented. Our experimental results show that NBFTP consumes 60% less network traffic than rsync when detecting the data pending to be transferred. When transferring small files of 1KB, the network traffic consumption of NBFTP is 40% less than rsync. However, as the file size increases, the network traffic consumption of NBFTP tends to approach rsync, but it is still smaller than rsync.

Boosting Edge Intelligence With Collaborative Cross-Edge Analytics

Edge intelligence is emerging as a new interdiscipline to accelerate the convergence of artificial intelligence (AI) and Internet of Things (IoT). By applying edge computing, edge intelligence enables resource-limited IoT devices to offload computation-intensive AI applications to the network edge for execution. While existing efforts have focused on optimizing the model training and inference phases of edge intelligence, the initial phase of edge intelligence-collaborative cross-edge analytics that preprocess the unlabeled raw data dispersed at multiple edge sites to obtain labeled and trainable data samples-has greatly overlooked. To bridge this gap, in this article, we study how to jointly optimize the input data and task placement, with the goal of speeding up collaborative cross-edge analytics at low network traffic cost. This problem is by no means trivial since it is nonconvex and involving future uncertainty of the query characteristics. To accommodate these dual challenges, we blend the advantages of the convex relaxation method and a two-stage optimization. Specifically, based on a prediction of the query characteristics, the input data placement is first determined when it is generated, and then when the query job arrives, the actual value of the query characteristics is used to optimize the task placement. The problem becomes more complicated when we have multiple queries simultaneously. In response, we develop an efficient flow scheduling for the intermediate data transfers of competing queries by extending the classic shortest remaining processing time (SRPT) policy. Extensive trace-driven simulations verify the efficacy of our proposed solution.

An Outer Bound for Linear Multi-Node Exact Repair Regenerating Codes

In distributed storage, erasure codes provide fault-tolerance while reducing the storage overhead compared to replication. The network traffic cost during the repair of node failures, called repair bandwidth, is an important metric in code design. Regenerating codes are a class of erasure codes developed with the aim of reducing the repair bandwidth while maintaining a high level of fault-tolerance. They exhibit a tradeoff between the storage overhead per node and the repair bandwidth. However, a fundamental understanding of the storage-repair bandwidth tradeoff under exact repair is open in general. In this work, we consider the exact repair problem of multiple failures in a centralized way. Building upon techniques from the literature, we first provide an alternative proof of the functional repair bound. Then, we derive a new outer bound for linear centralized multi-node exact repair codes and illustrate its performance under various parameter settings. The derived outer bound shows that, in general, the centralized multi-node functional repair tradeoff is not achievable under linear exact repair.

Learning automata-based algorithms for MapReduce data skewness handling

One of the most successful techniques for large-scale data processing is MapReduce. However, the performance of this technique is significantly reduced when there is skewness in data. The hash function is the default partitioner in Big Data frameworks such as Hadoop and Spark. Hash works perfectly when there is no data skewness, which is not the case in natural events. In this paper, we proposed two new algorithms, namely learning automata partitioner (LAP) and traffic cost-aware partitioner (TCAP) based on learning automata for handling reducer-side data skewness in MapReduce applications. LAP is based on clusters combination and performs well when data skewness degree is low. TCAP, on the other hand, has the advantage of considering network topology and balancing network traffic cost in the shuffling phase. TCAP supports cluster splitting and performs well in any data skewness degree. LAP and TCAP can also be used in heterogeneous environments. The performance of our algorithms was evaluated by several experiments and simulations by well-known benchmarks. The results confirmed that our algorithms performed better than other similar algorithms in most cases.

Multi-Task oriented data diffusion and transmission paradigm in crowdsensing based on city public traffic

As mobile smart devices become increasingly popular and are equipped with increasingly powerful sensors, they have been pervasively applied in crowdsensing as effective tools to solve large-scale sensing tasks in urban areas. Task requesters can allocate sensing tasks to mobile nodes through a crowdsensing platform, eliminating the cost of deploying and maintaining large numbers of fixed sensors. However, several kinds of crowdsensing tasks (e.g., audio and video transmission) that generate large-scale sensed data may bring high network traffic costs to participants using a 3G/4G network, which may affect their satisfaction. In this paper, we build a data diffusion and transmission paradigm in crowdsensing based on City Public Traffic System (PTS), and thoroughly discuss a paradigm for Multi-Task diffusion and transmission within budget constraints. This paradigm makes full use of the advantages of a bus in PTS to realize the rapid transmission of large-scale sensed data: predictable trajectory, wide coverage area, fast moving speed and long contact duration among passengers. Further, we also propose a new data transmission algorithm called BUI-BA that chooses mobile nodes to transfer data by maximizing the transmission utility increment. The experimental results demonstrate that BUI-BA has better overall performance than widely used Greedy and effSense, achieving a tradeoff between overall transmission utility and transmission redundancy.

Joint Content Placement and Storage Allocation in C-RANs for IoT Sensing Service

The Internet of Things (IoT) sensing service allows systems and users to monitor environment states by transmitting the content sensed by a variety of sensors. Owing to billions of sensors and devices deployed in the IoT system, a huge amount of data (big data) are generated, thus injecting tremendous traffic into the network. Cloud radio access network (C-RAN) is a promising wireless network architecture to accommodate the fast growing IoT traffic and improve the performance of IoT services. Caching in C-RAN, which brings content to the edges, not only alleviates the network traffic, thus improving the end-to-end user quality of service (QoS), but also avoids activating the sensors too frequently, thus reducing their energy consumption. The content placement problem determines what and where to cache in C-RAN. However, the caching performance is highly related to the caching storages. The storage allocation problem determines the storage capacities of network entities. In this paper, we jointly optimize the storage allocation problem and content placement problem in a hierarchical cache-enabled C-RAN architecture for IoT sensing service. We formulate the joint problem as an integer linear programming (ILP) model with the objective to minimize the total network traffic cost. The storage allocation problem and content placement problem are constrained by caching storage budgets and cache capacities, respectively. Two heuristic algorithms are proposed in order to reduce the computational complexity of ILP. Extensive simulations have been conducted to demonstrate that the performances of our proposed algorithms approximate the optimal solutions.

A network traffic-aware mobile application recommendation system based on network traffic cost consideration

A network traffic-aware mobile application recommendation system based on network traffic cost consideration

A network traffic-aware mobile application recommendation system based on network traffic cost consideration

A large amount and different types of mobile applications (or apps) are being offered to end users via app markets. Existing mobile app recommender systems generally recommend the most popular mobile apps to mobile users to facilitate the proper selection of mobile apps. However, these apps normally generate network traffic, which will consume the user's mobile data plan and may even cause potential security issues. Therefore, more and more mobile users are hesitant or even reluctant to use the mobile apps that are recommended by the mobile app markets. To fill this crucial gap, this paper proposes a mobile app recommendation approach which can provide app recommendations by considering both the apps' popularity and their traffic cost. To achieve this goal, this paper first estimates app network traffic score based on bipartite graph. Then, this paper proposes a flexible approach based on benefit-cost analysis, which can recommend apps by maintaining a balance between the apps' popularity and the traffic cost concern. Finally, this paper evaluates our approach with extensive experiments on a large-scale data set collected from Google Play. The experimental results clearly validate the effectiveness and efficiency of our approach.

Anchor Free IP Mobility

Efficient mobility management techniques are critical in providing seamless connectivity and session continuity between a mobile node and the network during its movement. However, current mobility management solutions generally require a central entity in the network core, tracking IP address movement, and anchoring traffic from source to destination through point-to-point tunnels. Intuitively, this approach suffers from scalability limitations as it creates bottlenecks in the network, due to sub-optimal routing via the anchor point. This is often termed “dog-leg” routing. Meanwhile, alternative anchorless, solutions are not feasible due to the current limitations of the IP semantics, which strongly tie addressing information to location. In contrast, this paper introduces a novel anchorless mobility solution that overcomes these limitations by exploiting a new path-based forwarding fabric together with emerging mechanisms from information-centric networking. These mechanisms decouple the end-system IP address from the path based data forwarding to eliminate the need for anchoring traffic through the network core; thereby, allowing flexible path calculation and service provisioning. Furthermore, by eliminating the limitation of routing via the anchor point, our approach reduces the network cost compared to anchored solutions through bandwidth saving while maintaining comparable handover delay. The proposed solution is applicable to both cellular and large-scale wireless LAN networks that aim to support seamless handover in a single operator domain scenario. The solution is modeled as a Markov-chain which applies a topological basis to describe mobility. The validity of the proposed Markovian model was verified through simulation of both random walk mobility on random geometric networks and trace information from a large-scale, city wide data set. Evaluation results illustrate a significant reduction in the total network traffic cost by 45 percent or more when using the proposed solution, compared to Proxy Mobile IPv6.

Map Reduce for big data processing based on traffic aware partition and aggregation

Big data refers to data sets whose volume is 500+ terabytes of data per day. The velocity makes it difficult to capture, manage, process and analyze 2 million records per day. Another characteristics of big data is variability which makes it difficult to identify the reason for losses in i.e., images, audio, video, sensor data and log files etc., Hadoop can be used to analyze this huge amount of data using Hadoop an approximate early result for executing the job partially becomes available for the user even before completion of job which reduce the response time. In Layers 3 Traffic aware clustering programming model is used for processing big data which includes the data processing function map by sort and reducing techniques. The implementation of the layers three traffic aware clustering method will be on the top of Hadoop which is partitioned into HDFS fixed sized blocks and generates intermediate output as a collection of pairs. The conventional hash function method is used for partitioning intermediate data among reduced task but it is not traffic efficient. In this paper to reduce network traffic cost, a Map Reduce task is done by designing data partition and aggregator that can reduce task merged traffic from multiple map tasks. The proposed algorithm is more efficient to reduce response time and the simulation results have showed proposal can reduce network traffic.

Utility-Centric Networking: Balancing Transit Costs With Quality of Experience

This paper is focused on techniques for maximizing utility across all users within a total network transit cost budget. We present a new method for selecting between replicated servers distributed over the Internet. First, we introduce a novel utility framework that factors in quality of service metrics. Then we design an optimization algorithm, solvable in polynomial time, to allocate user requests to servers based on utility while satisfying network transit cost constraints, mapping service names to service instance locators. We then describe an efficient, low overhead distributed model which only requires knowledge of a fraction of the data required by the global optimization formulation. Next, a load-balancing variant of the algorithm is explored that substantially reduces blocking caused by congested servers. Extensive simulations show that our method is scalable and leads to higher user utility compared with mapping user requests to the closest service replica, while meeting network traffic cost constraints. We discuss several options for real-world deployment that require no changes to end-systems based on either the use of SDN controllers or extensions to the current DNS system.

SCM: A method to improve network service layout efficiency with network evolution

Network services are an important component of the Internet, which are used to expand network functions for third-party developers. Network function virtualization (NFV) can improve the speed and flexibility of network service deployment. However, with the evolution of the network, network service layout may become inefficient. Regarding this problem, this paper proposes a service chain migration (SCM) method with the framework of “software defined network + network function virtualization” (SDN+NFV), which migrates service chains to adapt to network evolution and improves the efficiency of the network service layout. SCM is modeled as an integer linear programming problem and resolved via particle swarm optimization. An SCM prototype system is designed based on an SDN controller. Experiments demonstrate that SCM could reduce the network traffic cost and energy consumption efficiently.

ENHANCEMENT ON TRAFFIC AWARE PARTITION IN MAPREDUCE USING CLUSTERING TECHNIQUES

The Map reduce is a programming model for handling and processing the huge data sets using map and reduce tasks in parallel distributing. To increase the execution of mapreduce many number of activities have been made, but they ignore to deal with network traffic produced in shuffle stage. The existing mapreduce traffic-aware partitions suffer from partition skew issue, where the output of map tasks is unevenly distributed among reduces tasks. Existing arrangements take after a comparative rule that re partitions workload among diminish undertakings. In any case, those methodologies frequently cause elite overhead because of the segment estimate expectation and re-partitioning. The proposed work chooses dynamic data aware parallel with k-Means algorithm (DDAP-kM), a framework that provides dynamic partitioning skew reduction and clustering map reduce jobs. These works cope with partitioning skew by adjusting run time resource allocation to reduce tasks. By the experimental results network traffic cost is compared in terms of traffic aware partition algorithm and DDAP-kM algorithm.

A decentralized redundancy generation scheme for codes with locality in distributed storage systems

SummaryThe increasing data volume in a large number of applications presents a dire need for supporting the reliable data management in distributed storage systems. Existing classical erasure codes, such as the Reed‐Solomon codes and locally reconstruction codes, are widely adopted by many distributed storage systems. However, existing researches mainly focus on proposing new optimized codes, ignoring the optimization of the encoding process with the classical codes, where inefficient encoding process greatly degrades the encoding performance of the distributed storage systems. Thus, how to complete the encoding process in an efficient way has become the challenge for adopting the classical codes. In this paper, we propose a decentralized redundancy generation scheme on the basis of the codes with locality, called D2CP, where a 2‐step framework is proposed to support both the data patterns (replication to encoding and direct encoding) and codes with locality with any parameter set. For improving the insertion throughput, D2CP adopts a data placement technique with consistent hashing to guide the selection of nodes. For reducing the network traffic cost, D2CP adopts a data sending scheduling technique to schedule the transmission of the source nodes and a cooperative parity generation technique to generate the parity data cooperatively. To evaluate the performance of D2CP, we conduct experiments on our RAID distributed storage system under various parameter settings with both 30 physical and 200 virtual servers. Extensive experiments confirm that D2CP can improve the encoding throughput by 20% and 32% and reduce the network traffic cost by 16% and 33% compared with the typical approaches on average for the 2 data patterns respectively.

Network-aware virtual machine migration in an overcommitted cloud

Virtualization, which acts as the underlying technology for cloud computing, enables large amounts of third-party applications to be packed into virtual machines (VMs). VM migration enables servers to be reconsolidated or reshuffled to reduce the operational costs of data centers. The network traffic costs for VM migration currently attract limited attention.However, traffic and bandwidth demands among VMs in a data center account for considerable total traffic. VM migration also causes additional data transfer overhead, which would also increase the network cost of the data center.This study considers a network-aware VM migration (NetVMM) problem in an overcommitted cloud and formulates it into a non-deterministic polynomial time-complete problem. This study aims to minimize network traffic costs by considering the inherent dependencies among VMs that comprise a multi-tier application and the underlying topology of physical machines and to ensure a good trade-off between network communication and VM migration costs.The mechanism that the swarm intelligence algorithm aims to find is an approximate optimal solution through repeated iterations to make it a good solution for the VM migration problem. In this study, genetic algorithm (GA) and artificial bee colony (ABC) are adopted and changed to suit the VM migration problem to minimize the network cost. Experimental results show that GA has low network costs when VM instances are small. However, when the problem size increases, ABC is advantageous to GA. The running time of ABC is also nearly half than that of GA. To the best of our knowledge, we are the first to use ABC to solve the NetVMM problem.

An Efficient Traffic-Aware Partition and Aggregation for Big Data Applications using Map-Reduce

Objective: The objective of this system to reduce network traffic cost for a Map-Reduce job by designing a novel intermediate data partition scheme. Methods/Analysis: The Map-Reduce model streamlines the large scale information handling on commodities group by abusing parallel map and reduces assignments. Even though numerous endeavors have been made to increase the execution of Map-Reduce works, they disregard the network activity produced in the mix stage, which assumes a basic part in execution upgrade. Findings: Generally, a hash capacity is utilized to segment middle of the road information among decrease assignments, which, nonetheless, is not movement effective in light of the fact that network topology and its data size connected with every key are not thought seriously about. Reexamine to lessen system movement cost for a Map-Reduce work by planning a novel moderate information segment plan. Applications/Improvement: Decomposition based dynamic algorithm and hc algorithm is proposed to manage the huge scale optimization issue for enormous information application is likewise intended to change information parcel and conglomeration in a dynamic way. Finally, broad reproduction results show that the proposed recommendations can altogether decrease network movement cost under offline cases.

On Traffic-Aware Partition and Aggregation in MapReduce for Big Data Applications

The MapReduce programming model simplifies large-scale data processing on commodity cluster by exploiting parallel map tasks and reduce tasks. Although many efforts have been made to improve the performance of MapReduce jobs, they ignore the network traffic generated in the shuffle phase, which plays a critical role in performance enhancement. Traditionally, a hash function is used to partition intermediate data among reduce tasks, which, however, is not traffic-efficient because network topology and data size associated with each key are not taken into consideration. In this paper, we study to reduce network traffic cost for a MapReduce job by designing a novel intermediate data partition scheme. Furthermore, we jointly consider the aggregator placement problem, where each aggregator can reduce merged traffic from multiple map tasks. A decomposition-based distributed algorithm is proposed to deal with the large-scale optimization problem for big data application and an online algorithm is also designed to adjust data partition and aggregation in a dynamic manner. Finally, extensive simulation results demonstrate that our proposals can significantly reduce network traffic cost under both offline and online cases.

Repairing multiple failures adaptively with erasure codes in distributed storage systems

SummaryRepairs of multiple failures in distributed storage systems have posed the challenges for erasure coding: how to minimize the repair time with the least extra repair network traffic cost. However, existing repair schemes designed for single failure suffer from the high network traffic cost due to the serial repairs for multiple failures. Repair schemes designed for multiple failures suffer from long repair time due to the centralized repair structure. In this paper, we propose a decentralized adaptive repair scheme, called DARS, to minimize the repair time with the least extra network traffic cost. Specially, we propose a three‐layer repair model to support the repairs for both the single and multiple failures. For low repair time, a bandwidth‐aware node selection technique is proposed to guide the selection of nodes, and a line‐structured data transmission technique is proposed to organize the data transmission between the providers and the newcomer. For the least extra network traffic cost, a core‐based data distribution technique is proposed to organize the data transmission between the coordinator and other newcomers, and an intersection provider adjustment technique is proposed to adaptively adjust the number of intersection providers. Moreover, we adopt the ‘lazy repair’ within a stripe to further reduce the repair network traffic cost. We implement and evaluate DARS on our raid distributed storage system under various parameter settings with 30 physical machines and 200 virtual machines. Experimental results confirm that DARS reduces the repair time by 29% and 55% on average compared with tree‐structured repair and CORE, respectively. Copyright © 2015 John Wiley & Sons, Ltd.

EnDAS: Efficient Encrypted Data Search as a Mobile Cloud Service

Document storage in the cloud infrastructure is rapidly gaining popularity throughout the world. However, it poses risks to consumers unless the data are encrypted for security. Encrypted data should be effectively searchable and retrievable without any privacy leaks, particularly for the mobile client. Although recent research has solved many security issues, the architecture cannot be applied on mobile devices directly under the mobile cloud environment. This is due to the challenges imposed by wireless networks, such as latency sensitivity, poor connectivity, and low transmission rates. This leads to a long search time and extra network traffic costs when using the traditional search schemes. This paper addresses these issues by proposing an efficient encrypted data search (EnDAS) scheme as a mobile cloud service. This innovative scheme uses a lightweight trapdoor (encrypted keyword) compression method, which optimizes the data communication process by reducing the trapdoor’s size for traffic efficiency. In this paper, we also propose two optimization methods for document search, called the trapdoor mapping table module and ranked serial binary search algorithm, to speed the search time. Results show that EnDAS reduces search time by 34% to 47% as well as network traffic by 17% to 41%.

Distributed Cache Management in Information-Centric Networks

The main promise of current research efforts in the area of Information-Centric Networking (ICN) architectures is to optimize the dissemination of information within transient communication relationships of endpoints. Efficient caching of information is key to delivering on this promise. In this paper, we look into achieving this promise from the angle of managed replication of information. Management decisions are made in order to efficiently place replicas of information in dedicated storage devices attached to nodes of the network. In contrast to traditional off-line external management systems we adopt a distributed autonomic management architecture where management intelligence is placed inside the network. Particularly, we present an autonomic cache management approach for ICNs, where distributed managers residing in cache-enabled nodes decide on which information items to cache. We propose four on-line intra-domain cache management algorithms with different level of autonomicity and compare them with respect to performance, complexity, execution time and message exchange overhead. Additionally, we derive a lower bound of the overall network traffic cost for a certain category of network topologies. Our extensive simulations, using realistic network topologies and synthetic workload generators, signify the importance of network wide knowledge and cooperation.