Total Data Transfer Research Articles

Optimization of Parallel Data Transfers in Multi-Tiered Persistent Storage

A logical model of multi-tiered persistent storage provides a view of data where all available storage resources are distributed over a number of levels depending on the data transfer parameters and capacities. The efficient parallelization of data transfers in multi-tiered persistent storage is a significant challenge for a pipelined data processing model. This work examines a category of database applications implemented as sequences of operations that transfer data between the levels of multi-tiered persistent storage. The concept of EPN: Extended Petri Nets represents how database applications can be processed in parallel. A proposed transformation involves converting EPN into sequences of parallel data transfers. Additionally, a method is demonstrated for partitioning these sequences of data transfers, with the goal of reducing the total number of conflicts when data transfers occur between the levels of multi-tiered persistent storage. The paper proposes new rule-based algorithms for scheduling parallel data transfers that minimize total data transfer time. The objectives of the new algorithms are to evenly distribute the workload among the data transfer processes and reduce their idle time. Several experiments have confirmed the effectiveness of the new algorithms in generating parallel data transfer plans.

PROSE: Multi-round fair coflow scheduling without prior knowledge

Coflow scheduling plays a crucial role in enhancing network-level performance in datacenter applications. Existing works on coflow scheduling predominantly concentrate on clairvoyant schedulers, which assume prior knowledge of coflow sizes before transmission. However, this assumption is often unrealistic in practical scenarios like pipelined computation. Consequently, research on non-clairvoyant schedulers has garnered increasing attention. Most studies on non-clairvoyant schedulers focus on either improving efficiency or promoting fairness, with limited emphasis on striking a balance between the two objectives. Additionally, for non-cooperative environments like multi-tenant datacenter networks, it is essential to ensure strategy-proofness in coflow scheduling. Without adequate measures, tenants may manipulate their reported requirements to gain an unfair advantage in resource allocation. Ideally, in non-cooperative environments, a coflow scheduler should achieve efficiency, fairness, and strategy-proofness, all without prior knowledge of coflow sizes. However, few existing efforts have managed to simultaneously achieve these objectives.To fill this gap, we develop PROSE, a non-clairvoyant scheduler for non-cooperative environments. PROSE divides the scheduling time into multiple rounds and employs a two-stage scheduling mechanism based on these rounds. In the first stage, PROSE estimates the sizes of the coflows with unknown sizes by transmitting the probe flows, using a policy that combines coflow competition with batch processing. Once the information about coflow sizes is obtained, PROSE prioritizes coflows among tenants for bandwidth allocation in the second stage, aiming to accelerate coflow completion within each round. Furthermore, PROSE ensures optimal isolation guarantee by fairly limiting the total data transfer for each tenant and dynamically adjusts the scheduling order of coflows in a round-robin manner across multiple rounds. PROSE prevents tenants from attempting to consume excessive network resources through manipulative actions that deceive the scheduler. Trace-driven simulations demonstrate that PROSE outperforms the fair scheduler, achieving up to a 2.37× reduction in average coflow completion time, indicating substantial efficiency improvements.

Novel Modification of the Collective Dynamic Routing Method for Sensors' Communication in Wi-Fi Public Networks.

The widespread use of the Internet of Things makes it relevant to use public IP networks for simultaneous access by both users and wireless sensors. To achieve this, a significant reduction in the subscriber devices' energy consumption is required. This paper analyzes the application features of the collective dynamic routing method both with and without the use of a robust method for estimating the channel data rate for sensors' communication in wireless public networks. Based on the analysis, a novel modification of the collective dynamic routing method has been developed that reduces the sensors' energy consumption while keeping a high data rate and short delivery time for users. An analysis of the network load, the total data transfer rate over the network, and the parameters affecting the sensors' energy consumption was carried out for a segment of a seamless IEEE 802.11ax network. The simulation demonstrated a high efficiency of using a novel modification of the collective dynamic routing method for access to users and wireless sensors.

Minimizing Image Quality Loss After Channel Count Reduction for Plane Wave Ultrasound via Deep Learning Inference.

High-frame-rate ultrasound imaging uses unfocused transmissions to insonify an entire imaging view for each transmit event, thereby enabling frame rates over 1000 frames per second (fps). At these high frame rates, it is naturally challenging to realize real-time transfer of channel-domain raw data from the transducer to the system back end. Our work seeks to halve the total data transfer rate by uniformly decimating the receive channel count by 50% and, in turn, doubling the array pitch. We show that despite the reduced channel count and the inevitable use of a sparse array aperture, the resulting beamformed image quality can be maintained by designing a custom convolutional encoder-decoder neural network to infer the radio frequency (RF) data of the nullified channels. This deep learning framework was trained with in vivo human carotid data (5-MHz plane wave imaging, 128 channels, 31 steering angles over a 30° span, and 62 799 frames in total). After training, the network was tested on an in vitro point target scenario that was dissimilar to the training data, in addition to in vivo carotid validation datasets. In the point target phantom image beamformed from inferred channel data, spatial aliasing artifacts attributed to array pitch doubling were found to be reduced by up to 10 dB. For carotid imaging, our proposed approach yielded a lumen-to-tissue contrast that was on average within 3 dB compared to the full-aperture image, whereas without channel data inferencing, the carotid lumen was obscured. When implemented on an RTX-2080 GPU, the inference time to apply the trained network was 4 ms, which favors real-time imaging. Overall, our technique shows that with the help of deep learning, channel data transfer rates can be effectively halved with limited impact on the resulting image quality.

Hub Location-allocation in Computer-based Networks under Disruption Using Whale Optimization Algorithm

In this research, the location of hubs in computer networks is investigated using the whale optimization algorithm. The problem of locating hubs in computer networks is an optimization problem and requires the definition of a suitable fit function. Therefore, the total data transfer time and the cost of creating hubs is used as a fit function. Capacitive hubs increase network availability because hubs have more response capacity than the number of node requests connected to the hubs. In hub location issues, the study seeks to connect the nodes to the nearest hub and create a computer network with the least cost of connecting the nodes to the hubs. The present study attempts to reduce disruptions in computer networks as a research innovation. Therefore, by using the whale optimization algorithm and solving the model with its help, the effective factors that affect computer network disruptions and examining the effect of each are identified. Given the results, the model’s reaction in terms of time and cost led to an increase in temporal and cost parameters.

A Parallel Fuzzy Load Balancing Algorithm for Distributed Nodes Over a Cloud System

Cloud Computing is an IT organization concept that places the Internet at the heart of business activity, allowing it to use hardware resources. In recent years, Load Balancing has been an active area of research, and has played a very important role in the case of the Cloud environment. There is a wide range of improvements in this arena. In this paper, we propose a new Load Balancing algorithm, based on the weights of the nearest servers in the cloud platform. We use the fuzzy logic to represent the weight of the different nodes. Moreover, we implement separate requests in parallel, and use a token to dispatch tasks efficiently. The several scenarios in this paper are considered for experimentation and compare the result of existing Round Robin, Throttled Load Balancing, Equal Spread Load and proposed algorithm .The experiment results show that this approach improves the Load Balancing process effectively in terms of overall response time, data center processing time, total virtual machine cost, and total data transfer cost.

Real-Time Fault Detection and Condition Monitoring for Industrial Autonomous Transfer Vehicles Utilizing Edge Artificial Intelligence.

Early fault detection and real-time condition monitoring systems have become quite significant for today’s modern industrial systems. In a high volume of manufacturing facilities, fleets of equipment are expected to operate uninterrupted for days or weeks. Any unplanned interruptions to equipment uptime could jeopardize manufacturers’ cycle time, capacity, and, most significantly, credibility for their customers. With the help of smart manufacturing technologies, companies have started to develop and integrate fault detection and classification systems where end-to-end constant monitoring of equipment is facilitated, and smart algorithms are adapted for the early generation of fault alarms and classification. This paper proposes a generic real-time fault diagnosis and condition monitoring system utilizing edge artificial intelligence (edge AI) and a data distributor open source middleware platform called FIWARE. The implemented system architecture is flexible and includes interfaces that can be easily expanded for various devices. This work demonstrates it for condition monitoring of autonomous transfer vehicle (ATV) equipment targeting a smart factory use case. The system is verified in a designated industrial model environment in a lab with a single ATV operation. The anomaly conditions of the ATV are diagnosed by a deep learning-based fault diagnosis method performed in the Edge AI unit, and the results are transferred to the data storage via a data pipeline setup. The proposed system’s Edge AI solution for the ATV use case provides significant real-time performance. The network bandwidth requirement and total elapsed data transfer time have been reduced by 43 and 37 times, respectively. The proposed system successfully enables real-time monitoring of ATV fault conditions and expands to a fleet of equipment in a real manufacturing facility.

Information Transmission Bounds Between Moving Terminals

In networks of mobile autonomous agents, e.g. for data acquisition, we may wish to maximize data transfer or to reliably transfer a minimum amount of data, subject to quality of service or energy constraints. These requirements can be guaranteed through both offline node design/specifications and online trajectory/communications design. Regardless of the distance between them, for a stationary point-to-point transmitter-receiver pair communicating across a single link under average power constraints, the total data transfer is unbounded as time tends to infinity. In contrast, we show that if the transmitter/receiver is moving at any constant speed away from each other, then the maximum transmittable data is bounded. Although general closed-form expressions as a function of communication and mobility profile parameters do not yet exist, we provide closed-form expressions for particular cases, such as ideal free space path loss. Under more general scenarios we instead give lower bounds on the total transmittable information across a single link between mobile nodes.

Multi-objective communication-aware optimization for virtual machine placement in cloud datacenters

This paper formulates a new multi-objective virtual machine placement (VMP) problem, which is a challenging task in cloud datacenters (DCs). In cloud environment, there are two stakeholders, namely, users and providers. Both sides try to take more benefit whereas a trade-off between conflicting benefits is crucial. From providers’ perspective, power consumption and resource wastage are two objectives to be optimized whereas gaining high quality of service (QoS) is a critical point for users. The unpleasant issue that a user endures in the cloud environment is network delay; this is affected by common bandwidth linkage which is shared between different users’ applications; the reason for considering bandwidth usage optimization as the third objective function in users’ viewpoint. However, inefficient network bandwidth usage has drastic impact on overall performance even makes network links to get saturated and throttles communication-intensive applications. Therefore, VMs with high affinity and traffic dependency must be physically placed as close as possible so less traffic is sent on network layers. To figure out this combinatorial multi-objective problem, we extend a hybrid multi-objective genetic-based optimization solution. To evaluate this work, we conducted extensive scenarios with variable correlation coefficients between resources in requested VMs. The simulation results prove that our proposed hybrid meta-heuristic algorithm outperforms against state-of-the-art ACO-based, well-known heuristic-based FFD algorithms, and random-based approach in terms of total power consumption, resource wastage, the total data transfer rate in network, and number of active servers in use. Also, the simulations in larger search space demonstrated proposed approach has high potential of scalability.

Quantitative Mapping of Nanoscale Chemical Dynamics in Sub‐Sampled Operando (S)TEM Images using Spatio‐Temporal Analytics

AbstractOne of the main limitations in the use of operando scanning transmission electron microscopes to study dynamic chemical processes is the effect of the electron beam on the kinetics of the reaction being observed. Here we demonstrate that a flexible Gaussian mixture model can be used to extract quantitative information directly from sub‐sampled images, that is, images in which not all the pixels in the image are illuminated with the beam. The use of this method is demonstrated on the growth of silver (Ag) nanoparticles in an aqueous solution and the charge/discharge cycle of a lithium battery. In both videos, the imaged chemical reactions can be accurately quantified for sub‐sampling levels down to 1 %. In addition, our analysis of the Ag nucleation video sheds new light on the interplay between heterogeneous and homogeneous nucleation dynamics. Performing operando imaging using a small fraction of the pixels means that the observations can significantly reduce the effect of the beam, automatically increase the imaging speed and decrease the total data transfer rate required. Such new software capabilities offer the potential to significantly widen the application of operando hardware approaches to study nanoscale dynamics in materials.

FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling

Layout-Aware Data Scheduler (LADS) data transfer tool, identifies and addresses the issues that lead to congestion on the path of an end-to-end data transfer in the terabit network environments. It exploits the underlying storage layout at each endpoint to maximize throughput without negatively impacting the performance of shared storage resources for other users. LADS can avoid congested storage elements within the shared storage resource, improving input/output bandwidth, and hence the data transfer rates across the high speed networks. However, absence of FT (fault tolerance) support in LADS results in data re-transmission overhead along with the possible integrity issues upon errors. In this paper, we propose object based logging methods to avoid transmitting the objects which are successfully written to Parallel File System (PFS) at the sink end. Depending on the number of logger files created, for the whole dataset, we classified our fault tolerance mechanisms into three different categories: File logger, Transaction logger and Universal logger. Also, to address space overhead of these object based logging mechanisms, we have proposed different methods of populating logger files with the information of the completed objects. We have evaluated the data transfer performance and recovery time overhead of the proposed object based logging fault tolerant mechanisms on LADS data transfer tool. Our experimental results show that, LADS in conjunction with proposed object based fault tolerance mechanisms exhibit an overhead of less than 1% with respect to data transfer time and total recovery time overhead is around 10% of total data transfer time at any fault point.

RESTful economic-ADS model for cost-effective chain-wide traceability system-based cloud computing

Chain-wide traceability systems conforming to the Electronic Product Code Information Services (EPCIS) standard which is a well-recognized global standard allow supply chain operators to digitally capture and directly access the traceability data within and across enterprises. Although recent studies proposed the cloud-based deployment of EPCIS-based traceability systems by which supply chain operators are likely to save their investment cost in IT infrastructure, resource utilization remains a major concern in cloud-based deployment and those systems are required to deliver optimal performance. In this research, we propose the RESTful Economic-Aggregating Discovery Service (RECO-ADS) model to improve the cost effectiveness of resource use of the EPCIS system, which runs in the cloud. In the proposed model, we select the ADS model as the query traceability data approach. We design the internal and centralized traceability systems to comply with the EPCIS specification and Electronic Product Code Discovery Service (EPCDS) deployed to the Infrastructure as a Service (IaaS) cloud business model. Moreover, we integrate the RESTful web service with the Interfaces of EPCIS and EPCDS traceability systems. The evaluation criteria to measure resource efficiency, including computing resources, data transfers, and response time, are provided. The RECO-ADS is compared with the conventional traceability models, Directory Service (DS), Query Relay (QR) and Aggregating Discovery Service (ADS), using the evaluation criteria. The comparison shows that the computing resources, which consist of the CPU and Memory percentages of EPCIS and EPCDS systems, are rather similar, whereas the total data transfer (kB/s) of RECO-ADS (661.97) was less than that of the DS (807.73), QR (794.80) and ADS (796.66) models by approximately 18.05%, 16.71%, and 16.91%, respectively. The response time of the RECO-ADS model was the shortest. Moreover, the design of the cache in the EPCDS repository in the RECO-ADS model yielded a reduced operational cost based on the performance evaluation. These findings improve on the cost-effective resource consumption in the IaaS cloud of not only the EPCIS system but also the EPCDS system.

Query Based Resource Allocation and Performance Prediction in Distributed Database

Resource allocation is one of the main issues for providing services in an efficient way on distributed database. Improving the performance is one of the key research issues by proper design of efficient distributed database and proper usage of resources in information technology. The cost of each computational service depends on the amount of computations. The system resources have to be allocated in order to handle the workload and minimize the cost of computing by using proper allocation strategy. Performance is strongly related to the allocation of resources and data fragments in distributed environment. The query requires data to be accessed from one or multiple sites. The required data are fragmented and placed in various data center, where resources exist. In this paper we propose an algorithm to minimize the total data-transfer cost required for processing the queries by proper allocation of resources. We are finding the optimum allocation strategy, which predicts the performance by estimating the cost.

A dynamic block device reconfiguration algorithm in virtual MapReduce cluster

With the advances of cloud computing and virtualization technologies, running MapReduce applications over clouds has been attracting more and more attention in recent years. However, as a fundamental problem, the performance of MapReduce applications can sometimes be severely degraded due to the overheads from I/O virtualization and resource competitions among virtual machines. In this paper, we propose a dynamic block device reconfiguration algorithm in virtual MapReduce clusters, which reduces the data transfer time between virtual machines and thereby improving the performance of MapReduce applications on top of the clouds. The proposed algorithm utilizes a block device reconfiguration scheme, where a block device attached to a virtual machine can be dynamically detached and reattached to other virtual machines at runtime. This scheme allows us to move files easily across different virtual machines without any network transfers between virtual machines. This algorithm is also dynamic in a sense that it estimates the total data transfer times between virtual machines using multiple regression analysis based on CPU utilization and data size, and adaptively determines a least-cost data transfer path between a mapper virtual machine and a reducer virtual machine. We have implemented our algorithm in Hadoop MapReduce. The benchmarking results showed that the overheads incurred by transferring data from mapper virtual machines to reducer virtual machines are minimized and the execution times of MapReduce applications are shortened up to 14 %.

Nonreplicated Static Data Allocation in Distributed Databases Using Biogeography-Based Optimization

Allocation of data is one of the key design issues of distributed database. A major cost of query execution in a distributed database system is the data transfer cost from one site to another site. The allocation of fragments among the different sites over the network plays an important role in performance of the distributed database system. The main objective of a data allocation in distributed database is to place the data fragments at different sites in such a way, so that the total data transfer cost can be minimized while executing a set of queries. In this paper, a new biogeography-based optimization (BBO) algorithm has been used to allocate the fragments during the design of distributed database system. The goal of this paper is to design a fragments allocation algorithm, so that the total data transmission cost can be minimized. To show the performance of proposed algorithm, results of biogeography-based optimization algorithm for data allocation are compared with genetic algorithm.

A middleware based network hot swapping solution for SCA compliant radio

Software communication architecture (SCA) provides a framework for developing software defined radios (SDR). Theoretically, SCA compliant SDRs are supposed to have the capability of self-configuring and switching from one wireless protocol to any other. However, conventional SDRs running standard upper layer protocol stack fail to switch wireless protocols on the fly retaining all the connections and incur large delay. In this paper, we propose a mechanism for wireless protocol switching that is free from these problems. We base our approach on dual or multiple antennas operable by SDR software. The main contribution of this paper is the hot swapping middleware (HSM) that ensures seamless switching between wireless protocols without affecting the application performance. HSM manages IP addressing issues transparently and ensures in order delivery of data packets without loss. Our proposal is free from any requirement of infrastructure support, light weight and easily SCA component upgradable. Simulation results show that our proposal improves the total data transfer of applications in an SDR by more than 25%. We also have developed a proof-of-concept prototype.

Placement problems for transparent data replication proxy services

Transparent data replication has been considered a promising technique for improving system performance for a large distributed network. In this paper, a hybrid transparent replication model is presented. We address the problems of replication proxy placement in the network and data replica placement on the installed proxies given that a maximum of M proxies are allowed. Both reads and writes are considered in these problems. The performance objective is to minimize the total data transfer cost. To address the placement problems, we first present the optimal solutions for a single object in a tree network without/with constraint on the number of replicas. Based on that, two schemes, namely, aggregate access (AGGA) and weighted popularity (WPOP), are proposed for the replication proxy placement problem. An optimal solution is described for the replica placement problem. The performance of the proposed placement schemes is evaluated with a set of carefully designed simulation experiments over a wide range of system parameters. The results give us several helpful intuitions in deploying transparent replication proxies in a practical system.

Evolutionary Algorithms for Allocating Data in Distributed Database Systems

A major cost in executing queries in a distributed database system is the data transfer cost incurred in transferring relations (fragments) accessed by a query from different sites to the site where the query is initiated. The objective of a data allocation algorithm is to determine an assignment of fragments at different sites so as to minimize the total data transfer cost incurred in executing a set of queries. This is equivalent to minimizing the average query execution time, which is of primary importance in a wide class of distributed conventional as well as multimedia database systems. The data allocation problem, however, is NP-complete, and thus requires fast heuristics to generate efficient solutions. Furthermore, the optimal allocation of database objects highly depends on the query execution strategy employed by a distributed database system, and the given query execution strategy usually assumes an allocation of the fragments. We develop a site-independent fragment dependency graph representation to model the dependencies among the fragments accessed by a query, and use it to formulate and tackle data allocation problems for distributed database systems based on query-site and move-small query execution strategies. We have designed and evaluated evolutionary algorithms for data allocation for distributed database systems.

An 8-Gb/s optical backplane bus based on microchannel interconnects: design, fabrication, and performance measurements

We describe the characteristics of a microchannel-based optical backplane including signal-to-noise ratio (SNR), interconnect distances, and data transfer rates. The backplane employs 250 /spl mu/m-spacing two-dimensional (2-D) vertical cavity surface emitting lasers (VCSELs) and a microlens array to implement 500 /spl mu/m-, 750 /spl mu/m-, and 1-mm optical beam arrays. By integrating the transmitter and a multiplexed polymeric hologram as a deflector/beam-splitter for the guided-wave optical backplane, the result demonstrates a multibus line architecture and its high-speed characteristics. Maximum interconnect distances of 6 cm and 14 cm can be achieved to satisfy 10/sup -12/ bit error rate (BER) using 2/spl times/2 beams of 500 /spl mu/m- and 1 mm-spacing array devices. The total data transfer rate of the developed backplane has shown 8 Gb/s from eye diagram measurements.

Cyberbroker: A lightweight web object collector

Centralized heavyweight collectors have been used to collect text-based web objects for building search engines. However, these collectors do not meet the need of collecting big web objects such as images and videos. We propose a lightweight mobile on-call object collector for the world wide web. Using this approach, we can deploy a collector easily on the site being indexed. We present a prototype system consisting of mobile Java collectors and search engines for collecting HTML/image objects. We provide analysis to find a lower bound in terms of total data transfer time required. This idealized scenario is complemented with another model based on implementation of the HTTP over TCP. The lower bound and the upper bound of the gains accrued are then validated using experiments.