Massive Data Transfer Research Articles

An efficient data transfer service for scientific applications in cloud environments

Recently, more and more data-intensive scientific applications have been deployed in cloud environments. Therefore, how to improve the efficiency of data transfer becomes an important issued that needs to be addressed. In this paper, we present an efficient data transfer framework which provides an integrated platform for data transfer, data scheduling and performance monitoring. Unlike those existing studies that focus on the utilisation of bandwidth resources, the proposed framework is implemented by integrating data transfer service and data scheduling service through a performance prediction service. In this way, it provides a flexible mechanism to enable a cloud system to improve the efficiency of data transfer. The implementation of the proposed framework has been deployed in a real-world cloud system, and experimental results have shown that in can significantly improve the efficiency of massive-data transfer comparing with many existing approaches.

CoMSA: compression of protein multiple sequence alignment files.

Bioinformatics databases grow rapidly and achieve values hardly to imagine a decade ago. Among numerous bioinformatics processes generating hundreds of GB is multiple sequence alignments of protein families. Its largest database, i.e. Pfam, consumes 40-230 GB, depending of the variant. Storage and transfer of such massive data has become a challenge. We propose a novel compression algorithm, CoMSA, designed especially for aligned data. It is based on a generalization of the positional Burrows-Wheeler transform for non-binary alphabets. CoMSA handles FASTA, as well as Stockholm files. It offers up to six times better compression ratio than other commonly used compressors, i.e. gzip. Performed experiments resulted in an analysis of the influence of a protein family size on the compression ratio. CoMSA is available for free at https://github.com/refresh-bio/comsa and http://sun.aei.polsl.pl/REFRESH/comsa. Supplementary data are available at Bioinformatics online.

Enhancing Data Transfer Performance Utilizing a DTN between Cloud Service Providers

The rapid transfer of massive data in the cloud environment is required to prepare for unexpected situations like disaster recovery. With regard to this requirement, we propose a new approach to transferring cloud virtual machine images rapidly in the cloud environment utilizing dedicated Data Transfer Nodes (DTNs). The overall procedure is composed of local/remote copy processes and a DTN-to-DTN transfer process. These processes are coordinated and executed based on a fork system call in the proposed algorithm. In addition, we especially focus on the local copy process between a cloud controller and DTNs and improve data transfer performance through the well-tuned mount techniques in Network File System (NFS)-based connections. Several experiments have been performed considering the combination of synchronous/asynchronous modes and the network buffer size. We show the results of throughput in all the experiment cases and compare them. Consequently, the best throughput in write operations has been obtained in the case of an NFS server in a DTN and an NFS client in a cloud controller running entirely in the asynchronous mode.

Designing hybrid graph model and algorithmic analysis of workflow decomposition in mobile distributed systems

The execution of data intensive workflow is challenging in the domain of mobile distributed systems due to the involvement of heterogeneous computing devices and dynamic network topology. Moreover, data intensive workflow execution requires frequent access to database servers for dataflow and control flow operations resulting in massive data transfer. The workflow decomposition and distribution are required to minimize data transfer and to reduce execution delay in mobile distributed systems. This paper proposes a hybrid graph model of workflow in mobile distributed systems relying on dynamic network topology and distributed database in order to facilitate workflow decomposition. The algebraic and numerical analysis of the hybrid graph model is represented providing inherent properties. Furthermore, we have presented an extensive analytical study of existing workflow decomposition approaches and have provided detailed analysis to determine the suitability of these approaches in mobile distributed systems. We have also proposed a novel framework and implementation directions for workflow decomposition based on multiple parameters for mobile distributed systems.

Resource allocation and sharing for heterogeneous data collection over conventional 3GPP LTE and emerging NB-IoT technologies

Tremendous growth in the volumes and diversity of data to be collected in future Internet of Things (IoT) applications is one of the key challenges for the networking infrastructures as they evolve from 4G+ to true 5G systems. Particularly, ubiquitous deployments of wireless video surveillance cameras force the IoT service providers to support massive multimedia data transfer together with more ‘conventional’ machine-type communications. Hence, the data services in a complex IoT network may become heterogeneous, since several categories of traffic need to be supported simultaneously, each having its own loads, quality-of-service requirements, and radio technology preferences. An emerging example of the latter is the in-band deployment mode of the recently standardized NarrowBand IoT (NB-IoT) technology, where the same spectrum is shared between 3GPP LTE-connected high-end equipment and NB-IoT-connected low-end devices. However, while all of the necessary technology enablers are already in place, the question of how to share the available bandwidth efficiently has not been addressed comprehensively. Targeting the indicated challenge, this paper introduces feasible strategies for resource sharing between the multimedia and sensory data in a hybrid LTE/NB-IoT wireless deployment as well as compares them within our rigorous analytical methodology. The conducted numerical study advocates for one of the allocation strategies – dynamic resource sharing with reservation – as the preferred solution for reliable collection of heterogeneous data in large-scale 5G-grade IoT deployments.

Improved Handover Decision Algorithm Using Multiple Criteria

The transfer of massive data between varied network positions links of network relies on data rate, as well as the traffic capacity of the network. Conventionally, a device that is mobile can be used to attain vertical handover functional by weighing in only an aspect, which refers to Received Signal Strength (RSS). The application of this particular criterion could lead to interruption in services, ineffective vertical handover, and a network load that is not balanced. Hence, this paper proposes an improvised vertical handover decision algorithm by integrating multi-criteria within a wireless network that is heterogeneous. The proposed algorithm comprised of three vertical handover decision algorithms, namely: mobile weight, network weight, and equal weight. Additionally, three technology interfaces were embedded in this study including Worldwide interoperability for Microwave Access (WiMAX), Wireless Local Area Network (WLAN), and Long-Term Evolution (LTE). As a result, the simulation outcomes demonstrated that the handover decision algorithm for network weight generated exceptional outputs, in comparison to mobile and equal weights, as well as the conventional network decision algorithm from the aspects of handover failure and handover number probabilities.

Towards Incidence Management in 5G Based on Situational Awareness

The fifth generation mobile network, or 5G, moves towards bringing solutions to deploying faster networks, with hundreds of thousands of simultaneous connections and massive data transfer. For this purpose, several emerging technologies are implemented, resulting in virtualization and self-organization of most of their components, which raises important challenges related to safety. In order to contribute to their resolution, this paper proposes a novel architecture for incident management on 5G. The approach combines the conventional risk management schemes with the Endsley Situational Awareness model, thus improving effectiveness in different aspects, among them the ability to adapt to complex and dynamical monitoring environments, and countermeasure tracking or the role of context when decision-making. The proposal takes into account all layers for information processing in 5G mobile networks, ranging from infrastructure to the actuators responsible for deploying corrective measures.

TCP-Adaptive in High Speed Long Distance Networks

With the development of high performance computing and increasing of network bandwidth, more and more applications require fast data transfer over high-speed long-distance networks. Research shows that the standard TCP Reno cannot fulfill the requirement of fast transfer of massive data due to its conservative congestion control mechanism. Some works have been proposed to improve the TCP throughput performance using more aggressive window increasing tactics and obtain substantial achievements. However, they cannot be strictly proved to be comprehensively suitable for high-speed complex network environments. In this paper, we propose TCP-Adaptive, an adaptive congestion control algorithm adjusting the increasing congestion window dynamically. The algorithm improves logarithmic detection procedure for available bandwidth in the flow path by distinguishing the first detection in congestion avoidance and retransmission timeout. On the other hand, an adaptive control algorithm is proposed to achieve better performance in high-speed long-distance networks. The algorithm uses round trip time (RTT) variations to predict the congestion trends to update the increments of congestion window. Simulations verify the property of TCP-Adaptive and show satisfying performance in throughput, RTT fairness aspects over high-speed long-distance networks. Especially in sporadic loss environment, TCP-Adaptive shows a significant adaptability with the variations of link quality

Object Identification in Binary Tomographic Images Using GPGPUs

The authors present a hybrid OpenCL CPU/GPU algorithm for identification of connected structures inside black and white 3D scientific data. This algorithm exploits parallelism both at CPU and GPGPU levels, but the work is predominantly done in GPUs. The underlying context of this work is the structural characterization of composite materials via tomography. The algorithm allows us to later infer location and morphology of objects inside composite materials. Moreover, execution times are very low thus allowing us to process large data sets, but within acceptable running times. Intermediate solutions are computed independently over a partition of the spatial domain, following the data parallelism paradigm, and then integrated both at GPU and CPU levels, using parallel multi-cores. The authors consistently explore parallelism both at the CPU level, by allowing the CPU stage to run in multiple concurrent threads, and at the GPU level with massive parallelism and concurrent data transfers and kernel executions.

Massive data transfer between supercomputers and grid clusters using SCP controlled by FTS

Data transfers between Supercomputing centers and Grid sites may be complicated since, in general, Supercomputing centers are not completely incorporated in the Grid environment. In particular, the Port d'Informació Científica (PIC) had to transfer data of cosmological simulations from the Barcelona Supercomputing Center (BSC) to Grid storage resources in order to post-process those data at PIC. The data connection is limited to the Secure Shell (SSH) protocol using the scp command. Besides a limitof bandwidth, we had to deal with a lack of automation and reliability of the transfers. PIC managed to control those scp transfers through gLite's File Transfer Service (FTS). Therefore, the discs containing the data at the BSC are mounted via SSH to a local Storage Resource Management (SRM) server at PIC. Once established the access to the data through the SRM server, the data can be transfered to any other SRM server, including the storage system at PIC, using grid protocols. Although we cannot improve the bandwidth, we reach a level of automation and reliability comparable to transfers from the Tier-0 to Tier-1 sites of the Large Hadron Collider (LHC) experiments.

Hybrid congestion control for high-speed networks

Nowadays, more and more applications require fast transfer of massive data over networks, and the emergence of high-speed networks provides an ideal solution to this challenge. Due to the limitations of the conservative congestion control algorithm, the standard TCP is no longer appropriate for high-speed networks to efficiently utilize the bandwidth resources. Therefore, several high-speed TCP variants have been suggested to conquer the problem. However, although these protocols perform successfully to improve the bandwidth utilization, they still have the weakness on the performance such as RTT-fairness, TCP-friendliness, etc. In this paper, we propose HCC TCP, a hybrid congestion control algorithm using the synergy of delay-based and loss-based approach for the adaptation to high speed and long distance network environment. The algorithm uses queuing delay as the primary congestion indicator, and adjusts the window to stabilize around the size which can achieve the full utilization of available bandwidth. On the other hand, it uses packet loss as the second congestion indicator, and a loss-based congestion control strategy is utilized to maintain high bandwidth utilization in the cases that the delay-based strategy performs inefficiently in the networks. The two approaches in the algorithm are dynamically transferred into each other according to the network status. We finally perform simulations to verify the properties of the proposed HCC TCP. The simulation results demonstrate HCC TCP satisfies the requirements for an ideal TCP variant in high-speed networks, and achieves efficient performance on throughput, fairness, TCP-friendliness, robustness, etc.

A survey of self-adaptive grids

Grid systems allow the execution of a class of highly demanding services and applications. These grids involve communication networks, and their links are essential resources for massive data transfers. However, the management of current grid systems requires intervention for efficient service provisioning. Moreover, this need increases with the increase in demand for grid services. Therefore, grid systems will become effective only when they are capable of self-managing resource allocation to cope with fluctuations in resource availability. At present, however, very few integrated self-adaptive mechanisms have been implemented in existing grid systems. The aim of this article is to provide a survey of existing mechanisms and suggest directions for enabling autonomic operation of grid systems.

Configuration of the virtual laboratory for fusion researches in Japan

A virtual laboratory system for nuclear fusion researches in Japan known as SNET run by the National Institute for Fusion Science has been in development for the past seven years. Twenty-one remote sites have participated in SNET, which reached a speed of 1 Gbps in April 2009. The SNET is a closed network system based on L2 and L3VPN provided by SINET3, which is a national academic network operated by the National Institute of Informatics. SNET has been successfully supporting the remote participation of various sizes and types of experimental equipments and has also been supporting the remote use of a supercomputer. In this paper, we describe the configuration of SNET, which is overcoming the challenges that arise in virtual laboratories; we mainly explain the remote participation in the experiment. Remarks about the remote participation regarding the ITER activity, massive data transfer, and GRID are also discussed. A data transfer experiment between Japan and France was performed, with the average throughput reaching 880 Mbps on 1 Gbps of bandwidth.

Measurement and performance issues of transport protocols over 10Gbps high-speed optical networks

With the integration of IP and optical technology, a high-speed optical network (of the order of 10Gbps) has emerged to support international research cooperation such as massive scientific data transfer and next generation Internet related research. Therefore, it is critical to explore the issues of measurement and evaluation on the performance of transport protocols over 10 Gbps high-speed optical networks. To the best of our knowledge, this is the first work that presents a measurement study on a variety of networking environments. The objectives of this paper are as follows: (i) determine the suitability of TCP parameters such as Jumbo Frame size, buffer sizes of a TCP sender and a receiver; (ii) evaluate TCP performance measurement tools and emulation tools over 10 Gbps high-speed optical networks; and (iii) compare performance of TCP variants with different metrics, such as throughput and fairness, by varying delays and randomized losses controlled with software emulators. The result shows that selection of emulation and performance measurement tools matters for the accurate measurement of TCP performance. The performance of TCP variants is highly impacted by the Linux TCP/IP stack tuning. Finally we present that overall and detailed performance, such as throughput and fairness, of each TCP variant is dependent on different network environments, such as packet loss rate and propagation delay.

Optical Network Control and Management for Grid Applications

PDF HTML阅读 XML下载 导出引用 引用提醒 支持网格应用的光网络控制和管理 DOI: 作者: 作者单位: 作者简介: 通讯作者: 中图分类号: 基金项目: Supported by International Cooperation and Communications Project of National Ministry of Science and Technology of China under Grant No.2006DFA1990 (国家科技部国际合作与交流项目); the National High-Tech Research and Development Plan of China under Grant No.2007AA01Z214 (国家高技术研究发展计划(863)) Optical Network Control and Management for Grid Applications Author: Affiliation: Fund Project: 摘要 | 图/表 | 访问统计 | 参考文献 | 相似文献 | 引证文献 | 资源附件 | 文章评论 摘要:在高能物理等科研领域有许多基于网格的科研项目,包括海量数据传输在内的网格应用对网络带宽、延迟等QoS指标等有很高的要求.传统的IP路由网络无法满足这些要求.面向连接的光网络能够较好地满足这些要求,但也面临着许多挑战.分析了科研网格应用的特点及其对光网络控制和管理的特殊要求,重点分析比较了几种现有的光网络控制和管理方式,归纳和总结了此领域尚待研究的问题,最后提出了Lambda Grid和OVPN等新的研究 方向. Abstract:There are many research projects based on grid in many research fields like high energy physics (HEP). Grid applications including massive data transfer have very high requirement for network on QoS parameters such as bandwidth and delay. The traditional IP routing network can not satisfy these requirements. However, the connection-oriented optical network has this capability, but it still faces many challenges. The authors firstly analyzed the features of grid applications and their special requirements for optical network control and manage- ment, and emphatically analyze several existing methods of optical network control and management. Subsequen- tly, current issues which are open are summarized. At last, some novel research issues such as Lambda Grid and OVPN are put forward. 参考文献 相似文献 引证文献

A basic R&D for an analysis framework distributed on wide area network

Present status is reported on several R&D issues related to world-wide analysis for forthcoming high-energy collider experiments. This study includes (1) high-density computer farm with effective configuration and management capability, (2) high throughput data storage mechanisms covering disk arrays and hierarchical storage architecture, and (3) high-performance massive data transfer techniques on large latency WAN. Understanding these basic technologies is necessary for the deployment of developing data grid computing.

Mass data acquisition systems in JT-60 data processing system

In the data processing system for the JT-60 tokamak, a unique mass data acquisition system with fast sampling, a transient mass data storage system (TMDS), has been used since 1988. It is composed of a minicomputer and 61 channels of 4/6 MB memory modules with a sampling rate up to 200 kHz and about 300 MB of data are transferred to a main computer by using a special LAN developed by Fujitsu Ltd. TMDS can handle a large amount of data, but cannot be enlarged in its capability, such as CPU power or the number of channels. To solve the problems of TMDS, a new fast VME data acquisition system (FDS), has been developed. It can acquire 6 MB of data per channel with a sampling rate of 200 kHz or 1 MHz and consists of a workstation with VMEbus memory modules. Up to now there are three FDSs with 24 channels. The minicomputer of TMDS has been replaced with a new system based on the technology of FDS. To cope with mass data transfer to a data server, they are connected with a gigabit ethernet switch.

Simulation and information assessment of TEM and STEM images

Small and moderate digital systems, especially those with computer-microscope interface capabilities and possibly equipped with image frame stores, are being commonly employed in electron microscopy laboratories. The complexity of such systems varies greatly, ranging from those composed of simply a personal computer with minimum interfacing hardwares such as digital to analog converters(DACs) and pulse counters (or ADCs), compact enough to be fitted into a small cart and transported from room to room, to those consisting of image processing systems controlled by mini- or super mini-computers with system-bus interface for massive data transfer, array processors for fast online data handling. Be it simple or complex, such a system allows signals from various sources to be collected and stored in digital form for on-line or off-line manipulation. Fig.1 is a highly schematic diagram of the system currently working in ASU HREM laboratory, on which this work was done.