Large-scale Internet Applications Research Articles

IoT Analytics and Agile Optimization for Solving Dynamic Team Orienteering Problems with Mandatory Visits

Transport activities and citizen mobility have a deep impact on enlarged smart cities. By analyzing Big Data streams generated through Internet of Things (IoT) devices, this paper aims to show the efficiency of using IoT analytics, as an agile optimization input for solving real-time problems in smart cities. IoT analytics has become the main core of large-scale Internet applications, however, its utilization in optimization approaches for real-time configuration and dynamic conditions of a smart city has been less discussed. The challenging research topic is how to reach real-time IoT analytics for use in optimization approaches. In this paper, we consider integrating IoT analytics into agile optimization problems. A realistic waste collection problem is modeled as a dynamic team orienteering problem with mandatory visits. Open data repositories from smart cities are used for extracting the IoT analytics to achieve maximum advantage under the city environment condition. Our developed methodology allows us to process real-time information gathered from IoT systems in order to optimize the vehicle routing decision under dynamic changes of the traffic environments. A series of computational experiments is provided in order to illustrate our approach and discuss its effectiveness. In these experiments, a traditional static approach is compared against a dynamic one. In the former, the solution is calculated only once at the beginning, while in the latter, the solution is re-calculated periodically as new data are obtained. The results of the experiments clearly show that our proposed dynamic approach outperforms the static one in terms of rewards.

Efficient Compression Method Based on Object Model for Massive Real-time Data in Power IOT

With rapid development of the Internet of things (IOT) in power field, equipment of various majors began to access the smart IOT system on a large scale. In this paper, a new data compression algorithm based on object model (CABM) is proposed to solve the problems of large-scale applications of power Internet of Things, such as high-frequency transmission of large quantities, real-time short data can not be effectively compressed, resulting in low efficiency of cloud edge information transmission and serious waste of bandwidth resources. The CABM adds a compression and decompression module between the IOT management platform and the edge agent; the IOT management platform automatically generates a compression dictionary based on the object model file and saves it synchronously in the edge agent. During data transmission, both the sender and receiver complete automatic compression and decompression according to the compression dictionary. The experimental results show that for real-time message data below 1KB, the compression rate using CABM algorithm is 3-5 times higher than that using GZIP compression algorithm; and the compression time is 10-20 times lower than that using GZIP; the efficiency using CABM algorithm is 4.5 times higher than that using GZIP algorithm; that is, CABM can reduce redundant data transmission between cloud edges and improve the efficiency of cloud edge transmission.

A Novel Cost-Effective Controller Placement Scheme for Software-Defined Vehicular Networks

Energy costs have dramatically increased in data center networks as an increasing number of large-scale Internet applications are used. In software-defined vehicular networks (SDVN), the communication delay between two vehicles and between vehicles and the controller will dramatically climb up as the number of vehicles increases. This requires more controllers to provide communication service to minimize the latency. More controllers lead to high energy costs. Therefore, the number of controllers and their placement, the so-called controller placement problem (CPP), should be addressed. The appropriate placement of controllers can decrease the energy cost, enabling green communication in SDVN. Although CPP has been studied for static networks, it has not been effectively solved in highly dynamic and complex networks. In this article, a novel cost-effective CPP scheme for SDVN is proposed. First, our proposed minimum controller selection mechanism (MOSA) can reduce the number of controllers and guarantee the coverage of the area. Besides, an improved multiobjective artificial bee colony algorithm (IMABC) is proposed based on the original artificial bee colony algorithm. The IMABC can judge which controller should be switched on for data transmission based on real-time traffic flow. A route computation mechanism is proposed to evaluate the performance of our CPP scheme. The experimental results confirm that compared to other existing CPP schemes, our scheme can achieve a higher packet delivery ratio while greatly reducing energy consumption and latency.

A Collocation Method for Numerical Solution of Nonlinear Delay Integro-Differential Equations for Wireless Sensor Network and Internet of Things.

Wireless sensor network and industrial internet of things have been a growing area of research which is exploited in various fields such as smart home, smart industries, smart transportation, and so on. There is a need of a mechanism which can easily tackle the problems of nonlinear delay integro-differential equations for large-scale applications of Internet of Things. In this paper, Haar wavelet collocation technique is developed for the solution of nonlinear delay integro-differential equations for wireless sensor network and industrial Internet of Things. The method is applied to nonlinear delay Volterra, delay Fredholm and delay Volterra–Fredholm integro-differential equations which are based on the use of Haar wavelets. Some examples are given to show the computational efficiency of the proposed technique. The approximate solutions are compared with the exact solution. The maximum absolute and mean square roots errors for distant number of collocation points are also calculated. The results show that Haar method is efficient for solving these equations for industrial Internet of Things. The results are compared with existing methods from the literature. The results exhibit that the method is simple, precise and efficient.

An IoT-based task scheduling optimization scheme considering the deadline and cost-aware scientific workflow for cloud computing

Large-scale applications of Internet of things (IoT), which require considerable computing tasks and storage resources, are increasingly deployed in cloud environments. Compared with the traditional computing model, characteristics of the cloud such as pay-as-you-go, unlimited expansion, and dynamic acquisition represent different conveniences for these applications using the IoT architecture. One of the major challenges is to satisfy the quality of service requirements while assigning resources to tasks. In this paper, we propose a deadline and cost-aware scheduling algorithm that minimizes the execution cost of a workflow under deadline constraints in the infrastructure as a service (IaaS) model. Considering the virtual machine (VM) performance variation and acquisition delay, we first divide tasks into different levels according to the topological structure so that no dependency exists between tasks at the same level. Three strings are used to code the genes in the proposed algorithm to better reflect the heterogeneous and resilient characteristics of cloud environments. Then, HEFT is used to generate individuals with the minimum completion time and cost. Novel schemes are developed for crossover and mutation to increase the diversity of the solutions. Based on this process, a task scheduling method that considers cost and deadlines is proposed. Experiments on workflows that simulate the structured tasks of the IoT demonstrate that our algorithm achieves a high success rate and performs well compared to state-of-the-art algorithms.

Effect of Service Broker Policies and Load Balancing Algorithms on the Performance of Large Scale Internet Applications in Cloud Datacenters

Cloud computing is advancing rapidly. With such advancement, it has become possible to develop and host large scale distributed applications on the Internet more economically and more flexibly. However, the geographical distribution of user bases, the available Internet infrastructure within those geographical areas, and the dynamic nature of usage patterns of the user bases are critical factors that affect the performance of these applications. Therefore, it is necessary to compromise between datacenters, service broker policies, and load balancing algorithms to optimize the performance of the application and the cost to the owners. This paper aims at studying the effect of service broker policies and load balancing algorithms on the performance of large-scale Internet applications under different configurations of datacenters. To achieve this goal, we modeled the behavior of the popular Facebook application with the most recent worldwide users’ statistics. Then, we evaluated the performance of this application under different configurations of datacenters using: 1) two different service broker policies, namely, closest datacenter and optimum response time; and 2) three load-balancing algorithms, namely, round robin, equally spread current execution, and throttled load balancer. The overall average response time of the application and the overall average time spent for processing a user request by a datacenter are measured and the results are discussed. This study would help service providers generate valuable insights on coordination between datacenters, service policies, and load balancing algorithms when designing Cloud infrastructure services in geographically distributed areas. In addition, application designers would benefit greatly from this study in identifying the optimal arrangement for their applications.

Joint Optimization of Bandwidth for Provider and Delay for User in Software Defined Data Centers

In large-scale Internet applications running on geographically distributed datacenters, such as video streaming, it is important to efficiently allocate requests among datacenters. To the best of our knowledge, existing approaches, however, either solely focus on minimizing total cost for provider, or guaranteeing QoS for end-users. In this paper, we apply the software defined network (SDN) controller to enable the central control of the entire network, and propose a joint optimization model to consider high bandwidth utilization for provider and low delay for users. We present the Nash bargaining solution (NBS) based method to model both requirements of provider’s high bandwidth utilization and end-users’ low delay. Specifically, we formulate the design of request allocation under those requirements as an optimization problem, which is NP-hard. To solve such hard optimization problem, we develop an efficient algorithm blending the advantages of Logarithmic Smoothing technique and the auxiliary variable method. According to the theoretical analysis, we verify the existence and uniqueness of our solution and the convergence of our algorithm. We conduct a large amount of experiments based on real-world workload traces and demonstrate the efficiency of our algorithm compared to both greedy and locality algorithms.

Enabling Secure XMPP Communications in Federated IoT Clouds Through XEP 0027 and SAML/SASL SSO.

Nowadays, in the panorama of Internet of Things (IoT), finding a right compromise between interactivity and security is not trivial at all. Currently, most of pervasive communication technologies are designed to work locally. As a consequence, the development of large-scale Internet services and applications is not so easy for IoT Cloud providers. The main issue is that both IoT architectures and services have started as simple but they are becoming more and more complex. Consequently, the web service technology is often inappropriate. Recently, many operators in both academia and industry fields are considering the possibility to adopt the eXtensible Messaging and Presence Protocol (XMPP) for the implementation of IoT Cloud communication systems. In fact, XMPP offers many advantages in term of real-time capabilities, efficient data distribution, service discovery and inter-domain communication compared to other technologies. Nevertheless, the protocol lacks of native security, data confidentiality and trustworthy federation features. In this paper, considering an XMPP-based IoT Cloud architectural model, we discuss how can be possible to enforce message signing/encryption and Single-Sign On (SSO) authentication respectively for secure inter-module and inter-domain communications in a federated environment. Experiments prove that security mechanisms introduce an acceptable overhead, considering the obvious advantages achieved in terms of data trustiness and privacy.

Exploiting user feedback for online filtering in event-based systems

Modern large-scale internet applications represent today a fundamental source of information for millions of users. The larger is the user base, the more difficult it is to control the quality of data that is spread from producers to consumers. This can easily hamper the usability of such systems as the amount of low quality data received by consumers grows uncontrolled. In this paper we propose a novel solution to automatically filter new data injected in event-based systems with the aim of delivering only content consumers are actually interested in. Filtering is executed by profiling producers and consumers, and matching their profiles as new data is produced. Profiles are built by aggregating feedback submitted by consumers on previously received data.

Green latency-aware data placement in data centers

Large-scale Internet applications provide service to end users by routing service requests to geographically distributed data centers. Two concerns exist in service provisioning by the data centers. One is that users require to experience low latency while accessing data from data centers. The other is to reduce the energy consumed by network transport and the servers in the data centers. In this paper, we tackle the problem of green data placement in data centers to strike a tradeoff among access latency, energy consumption of data centers and network transport. We propose two request-routing algorithms, GLDP-NS (Green Latency-aware Data Placement - No consideration of the current data placement Status of the server) and GLDP-WS (Green Latency-aware Data Placement - With consideration of the current data placement Status of the server). We show that the green latency-aware data placement problem is NP-complete and algorithm GLDP-NS is a 3-approximation algorithm for the data placement problem without considering the data placement status of the server. We evaluate the performance of the proposed algorithms through simulations, and the simulation results demonstrate that the proposed algorithms can achieve good integrated cost performance of the latency, the energy consumption of data centers and network transport.

An Improved Lightweight Privacy Preserving Authentication Scheme for SIP-Based-VoIP Using Smart Card (MPhil – Dissertation)

In the past few years, secure information sharing became very popular in the area of immigration, military applications, healthcare, education, foreign affairs, etc. As secure communication utilizes both wireless and wired communication mechanizations for exchanging sensitive information, so security and privacy of the information exchange cannot be easily compromised. To moderate the security, integrity, authenticity, and privacy issues related to information exchange, numerous authentication mechanisms have been recommended by different researcher in the literature in recent times, but are vulnerable to prospective security flaws such as masquerade, insider, replay, impersonation, password guessing, server spoofing, denial-of-service attacks and in addition failed to deliver mutual authentication. In the past few years, we have also seen a balanced growth in the acceptance of VoIP (Voice over IP) facilities, because the numerous Web and VoIP applications depend on huge and extremely distributed infrastructures to process requests from millions of users in an appropriate manner. Due to their extraordinary desires, these large-scale Internet applications have frequently surrendered security for other objectives such as performance, scalability, and availability. As a result, these applications have characteristically favored weaker, but well-organized security mechanisms in their foundations. Session Initiation Protocol (SIP) is an application and presentation layers signaling protocol that initiates, modifies, and terminates IP-based multimedia sessions. Implementing SIP for secure communication has been a topic of study for the past decade, and several proposals are available in the research domain. However, security aspects are not addressed in most of these proposals, because SIP is exposed to several threats and faces security issue at these layers. Probes for SIP (Session Initiation Protocol) servers have been conveyed for many years, and to gather more details about these activities we simply design a for SIP servers in a network and composed data about some popular attacks. What will follow is an explanation of our interpretations and guidance on how to prevent these attacks from being successful. Biometrics a new field of research has also been materialized in this research, entitled “a three-factor authentication scheme in which one factor is biometrics. In biometric cryptosystems, the benefits of biometric confirmation are presented to basic cryptographic key supervisory systems to enhance security. Anyhow, this research delivers a general outline of the basics, permitting to biometrics as well as cryptography. This work also gives biometric cryptosystems based on iris biometrics and using smart card as well as a password for authentication.

Predicting Inter-Data-Center Network Traffic Using Elephant Flow and Sublink Information

With the ever increasing number of large scale Internet applications, inter-data-center (inter-DC) data transfers are becoming more and more common. Traditional inter-DC transfers suffer from both low utilization and congestion, and traffic prediction is an important method to optimize these transfers. Inter-DC traffic is harder to predict than many other types of network traffic because it is dominated by a few large applications. We propose a model that significantly reduces the prediction errors. In our model, we combine wavelet transform with artificial neural network to improve prediction accuracy. Specifically, we explicitly add information of sublink traffic and elephant flows, the least predictable yet dominating traffic in inter-DC network, into our prediction model. To reduce the amount of monitoring overhead for the elephant flow information, we add interpolation to fill in the unknown values in the elephant flows. We demonstrate that we can reduce prediction errors over existing methods by 5% $ \sim 30$ %. Our prediction is in production as part of the traffic scheduling system at Baidu, one of the largest Internet companies in China, helping to reduce the peak network bandwidth.

Survey-based Comparison of Chord Overlay Networks

Chord is a structured peer-to-peer (P2P) overlay network in which participating peers share resources as equals. To find a specific data item within the network, Chord system provide a lookup mechanism that matches a given key to a network node responsible for the value associated with that key. Chord is recently proposed to become one of the new approaches for building large-scale Internet applications. This paper aims to survey the Chord network, study its main characteristics, and compare its original performance with the performance of the enhancements being deployed over the original Chord. General Terms Distributed systems.

It's not easy being green

Large-scale Internet applications, such as content distribution networks, are deployed across multiple datacenters and consume massive amounts of electricity. To provide uniformly low access latencies, these datacenters are geographically distributed and the deployment size at each location reflects the regional demand for the application. Consequently, an application's environmental impact can vary significantly depending on the geographical distribution of end-users, as electricity cost and carbon footprint per watt is location specific. In this paper, we describe FORTE: Flow Optimization based framework for request-Routing and Traffic Engineering. FORTE dynamically controls the fraction of user traffic directed to each datacenter in response to changes in both request workload and carbon footprint. It allows an operator to navigate the three-way tradeoff between access latency, carbon footprint, and electricity costs and to determine an optimal datacenter upgrade plan in response to increases in traffic load. We use FORTE to show that carbon taxes or credits are impractical in incentivizing carbon output reduction by providers of large-scale Internet applications. However, they can reduce carbon emissions by 10% without increasing the mean latency nor the electricity bill.

Lightweight and Informative Traffic Metrics for Data Center Monitoring

In recent years, thousands of commodity servers have been deployed in Internet data centers to run large scale Internet applications or cloud computing services. Given the sheer volume of data communications between servers and millions of end users, it becomes a daunting task to continuously monitor the availability, performance and security of data centers in real-time operational environments. In this paper, we propose and evaluate a lightweight and informative traffic metric, streaming frequency, for network monitoring in Internet data centers. The power-series based metric that is extracted from the aggregated IP traffic streams, not only carries temporal characteristics of data center servers, but also helps uncover traffic patterns of these servers. We show the convergence and reconstructability properties of this metric through theoretical proof and algorithm analysis. Using real data-sets collected from multiple data centers of a large Internet content provider, we demonstrate its applications in detecting unwanted traffic towards data center servers. To the best of our knowledge, this paper is the first to introduce a streaming metric with a unique reconstruction capability that could aid data center operators in network management and security monitoring.

Perfect Disruption: The Paradigm Shift from Mental Agents to ORGs

This paper examines how a perfect disruption, brought on by changes to hardware, software, and applications, is causing a paradigm shift from mental agents to organizations of restricted generality (ORGs) as the foundation for implementing large-scale Internet applications.

Implementation of an Intelligent Urban Traffic Management System Based on a City Grid Infrastructure

Grid computing is becoming the preferred basis for large-scale distributed computing and Internet applications. ShanghaiGrid launched in Shanghai, China is a grid infrastructure that aggregates several heterogeneous supercomputers, data centers and other applications scattered in different organizations in Shanghai. One of the major focuses in the ShanghaiGrid project is research and development of an intelligent urban traffic management system based on the ShanghaiGrid system software. The system software provides a platform in a service-oriented way for resource encapsulation and management, service scheduling and accounting, data aggregation and adaptive transmission, as well as service composition and reliability support. This paper presents the main components of the system software, and investigates key techniques for the implementation of ShanghaiGrid and the intelligent urban traffic management system, concentrating on grid transaction and workflow management. Finally, we propose a new wireless sensor network model and routing protocol characteristic to mobile vehicles for dynamical traffic data collection.

Content subscribing mechanism in P2P streaming based on gamma distribution prediction

P2P systems are categorized into tree-based and mesh-based systems according to their topologies. Mesh-based systems are considered more suitable for large-scale Internet applications, but require optimization on latency issue. This paper proposes a content subscribing mechanism (CSM) to eliminate unnecessary time delays during data relaying. A node can send content data to its neighbors as soon as it receives the data segment. No additional time is taken during the interactive stages prior to data segment transmission of streaming content. CSM consists of three steps. First, every node records its historical segments latency, and adopts gamma distribution, which possesses powerful expression ability, to express latency statistics. Second, a node predicts subscribing success ratio of every neighbor by comparing the gamma distribution parameters of the node and its neighbors before selecting a neighbor node to subscribe a data segment. The above steps would not increase latency as they are executed before the data segments are ready at the neighbor nodes. Finally, the node, which was subscribed to, sends the subscribed data segment to the subscriber immediately when it has the data segment. Experiments show that CSM significantly reduces the content data transmission latency.

Estimation of network distances using off-line measurements

Several emerging large-scale Internet applications such as Content Distribution Networks, and Peer-to-Peer networks could potentially benefit from knowing the underlying Internet topology and the distances (i.e., round-trip-time) between different hosts. Most existing techniques for distance estimation either use a dedicated infrastructure or use on-line measurements in which probe packets are injected into the network during estimation. Our goal in this paper is to study off-line techniques for distance estimation that do not require a dedicated infrastructure. To this end, we propose a metric termed “depth” and we observe that together with a quadratic function on the geographic distance, it can predict the network distance with high accuracy using multi-variable regression. When used for closest server selection, our approach performs much better than random server selection, and similar to the on-line metrics. Our approach incurs low overhead and can be deployed easily with some DNS extensions.

Vivaldi

Large-scale Internet applications can benefit from an ability to predict round-trip times to other hosts without having to contact them first. Explicit measurements are often unattractive because the cost of measurement can outweigh the benefits of exploiting proximity information. Vivaldi is a simple, light-weight algorithm that assigns synthetic coordinates to hosts such that the distance between the coordinates of two hosts accurately predicts the communication latency between the hosts. Vivaldi is fully distributed, requiring no fixed network infrastructure and no distinguished hosts. It is also efficient: a new host can compute good coordinates for itself after collecting latency information from only a few other hosts. Because it requires little com-munication, Vivaldi can piggy-back on the communication patterns of the application using it and scale to a large number of hosts. An evaluation of Vivaldi using a simulated network whose latencies are based on measurements among 1740 Internet hosts shows that a 2-dimensional Euclidean model with height vectors embeds these hosts with low error (the median relative error in round-trip time prediction is 11 percent).