Large Data Transfer Research Articles

A Network Management System for Handling Scientific Data Flows

Large scientific data transfers often occur at high rates causing increased burstiness in Internet traffic. To limit the adverse effects of these high-rate large-sized flows, which are referred to as $$\alpha $$? flows, on delay-sensitive audio/video flows, a network management system called Alpha Flow Traffic Engineering System (AFTES) is proposed for intra-domain traffic engineering. An offline approach is used in which AFTES analyzes NetFlow records collected by routers, extracts source---destination address prefixes of $$\alpha $$? flows, and uses these prefixes to configure firewall filters at ingress routers of a provider's network to redirect future $$\alpha $$? flows to traffic-engineered paths and isolated queues. The effectiveness of this scheme was evaluated through an analysis of 7 months of NetFlow data obtained from an ESnet router. For this data set, 91 % of bytes generated by $$\alpha $$? flows during high-rate intervals would have been directed had AFTES been deployed. The negative aspect of using address prefixes in firewall filters, i.e., the redirection of $$\beta $$β flows to $$\alpha $$?-flow paths/queues, was also quantified.

Sequences of large radius

A k-radius sequence over an n-element alphabet A is a sequence in which every two elements of A occur within distance k of each other, where the distance is defined as the difference of indices of terms. The problem of finding (for given k and n) such sequences of a small length, motivated by some problems occurring in large data transfer, has been investigated by several authors recently. In this paper we consider a slight modification of the problem. We change the definition of the distance between terms of the sequence by computing it as if the terms were placed on a circle. The length of the shortest such sequence is denoted by gk(n).We investigate sequences of radius linear in the alphabet size. For some values of c<1 and sufficiently large n we find the exact values of gcn(n) by providing direct constructions of optimal sequences. In the constructions we use Singer difference sets. Moreover, we give a nontrivial general lower bound for gcn(n).

Optimizing Ordered Throughput Using Autonomic Cloud Bursting Schedulers

Optimizing ordered throughput not only improves the system efficiency but also makes cloud bursting transparent to the user. This is critical from the perspective of user fairness in customer-facing systems, correctness in stream processing systems, and so on. In this paper, we consider optimizing ordered throughput for near real-time, data-intensive, independent computations using cloud bursting. Intercloud computation of data-intensive applications is a challenge due to large data transfer requirements, low intercloud bandwidth, and best-effort traffic on the Internet. The system model we consider is comprised of two processing stages. The first stage uses cloud bursting opportunistically for parallel processing, while the second stage (sequential) expects the output of the first stage to be in the same order as the arrival sequence. We propose three scheduling heuristics as part of an autonomic cloud bursting approach that adapt to changing workload characteristics, variation in bandwidth, and available resources to optimize ordered throughput. We also characterize the operational regimes for cloud bursting as stabilization mode versus acceleration mode, depending on the workload characteristics like the size of data to be transferred for a given compute load. The operational regime characterization helps in deciding how many instances can be optimally utilized in the external cloud.

Distributed MLEM: An Iterative Tomographic Image Reconstruction Algorithm for Distributed Memory Architectures

The processing speed for positron emission tomography (PET) image reconstruction has been greatly improved in recent years by simply dividing the workload to multiple processors of a graphics processing unit (GPU). However, if this strategy is generalized to a multi-GPU cluster, the processing speed does not improve linearly with the number of GPUs. This is because large data transfer is required between the GPUs after each iteration, effectively reducing the parallelism. This paper proposes a novel approach to reformulate the maximum likelihood expectation maximization (MLEM) algorithm so that it can scale up to many GPU nodes with less frequent inter-node communication. While being mathematically different, the new algorithm maximizes the same convex likelihood function as MLEM, thus converges to the same solution. Experiments on a multi-GPU cluster demonstrate the effectiveness of the proposed approach.

Complexity Analysis and Algorithm Design for Advance Bandwidth Scheduling in Dedicated Networks

An increasing number of high-performance networks provision dedicated channels through circuit switching or MPLS/GMPLS techniques to support large data transfer. The link bandwidths in such networks are typically shared by multiple users through advance reservation, resulting in varying bandwidth availability in future time. Developing efficient scheduling algorithms for advance bandwidth reservation has become a critical task to improve the utilization of network resources and meet the transport requirements of application users. We consider an exhaustive combination of different path and bandwidth constraints and formulate four types of advance bandwidth scheduling problems, with the same objective to minimize the data transfer end time for a given transfer request with a prespecified data size: 1) fixed path with fixed bandwidth (FPFB); 2) fixed path with variable bandwidth (FPVB); 3) variable path with fixed bandwidth (VPFB); and 4) variable path with variable bandwidth (VPVB). For VPFB and VPVB, we further consider two subcases where the path switching delay is negligible or nonnegligible. We propose an optimal algorithm for each of these scheduling problems except for FPVB and VPVB with nonnegligible path switching delay, which are proven to be NP-complete and nonapproximable, and then tackled by heuristics. The performance superiority of these heuristics is verified by extensive experimental results in a large set of simulated networks in comparison to optimal and greedy strategies.

Improving data transfer performance of web service workflows in the cloud environment

Web service data forwarding WSDF is a framework for centralised web service workflow, in which the intermediate result from a previous service is treated as a resource of the composite service and can be directly used by its subsequent service, without sending it back to the centralised control centre. To improve the data transfer performance of web service workflows in the cloud environment, we carried out a test of the WSDF framework in the ScienceCloud, provided by the Nimbus cloud infrastructure. The experiment showed that, in the cloud environment, the WSDF framework has significant performance advantage over normal web service framework for workflows with large data transfer and the improvement of performance agrees with the expected theoretical value.

High Performance Ethernet Packet Processor Core for Next Generation Networks

As the demand for high speed Internet significantly increasing to meet the requirement of large data transfers, real-time communication and High Definition ( HD) multimedia transfer over IP, the IP based network products architecture must evolve and change. Application specific processors require high performance, low power and high degree of programmability is the limitation in many general processor based applications. This paper describes the design of Ethernet packet processor for system-on-chip (SoC) which performs all core packet processing functions, including segmentation and reassembly, packetization classification, route and queue management which will speedup switching/routing performance making it more suitable for Next Generation Networks (NGN). Ethernet packet processor design can be configured for use with multiple projects targeted to a FPGA device the system is designed to support 1/10/20/40/100 Gigabit links with a speed and performance advantage. VHDL has been used to implement and simulated the required functions in FPGA.

MPEG compression of ultrasound RF channel data for a real-time software-based imaging system

Using software for beamforming in ultrasound systems provides high flexibility, and the large number of computations required in a software-based system can be performed in real time on a PC. However, the very large data transfer rate required from the ultrasound front-end to the PC host for real-time operation is a bottleneck which cannot be overcome without appropriate compression. Previous studies have examined JPEG compression of ultrasound RF channel data, but the schemes do not exploit temporal redundancy between adjacent frames. This study utilized MPEG technology to process the ultrasound RF data to increase the compression efficiency. Our results indicate that MPEG compression generally provides a better compression ratio than does JPEG compression. As an example, the compression ratio of MPEG compression in an 8-bit channel A/D data under 5 μm interframe displacement is smaller than 0.13, thus allowing real-time data transfer requirements to be met. Moreover, the compression efficiency for motions in different directions is shown to be highly dependent on the frame-to-frame correlation.

과학탐구활동을 위한 천문 관측 자료의 실시간 전송 및 영상 구현 실험 방법 개발

종래의 지구과학 교과서에서는 대중 과학잡지를 비롯한 각종 언론 매체에서 자주 다루는 최신 천문 현상들에 대한 내용이 매우 부실하였다. 2009 개정 과학과 교육과정에서는 이러한 내용들을 적극 수용하자는 것이 주요 개정 방향 중 하나이다. 추가로, 천문학과 물리학의 깊은 연관성에도 불구하고 그동안 지구과학 교과에서는 현대 물리학의 개념, 물리 교과에서는 천문학과의 연관성에 대한 내용이 많이 다루어지지 않았다. 따라서 2009 개정 과학과 교육과정에서는 과학의 융합 개념을 강조하고 있다. 위의 두 가지를 모두 포함하는 분야들 중 하나로서 고에너지 천체의 격변광 현상을 들 수 있다. 본 연구는 e-Science를 이용한 천체 관측의 실시간 자료 전송 및 영상화를 소개하였다. 그 첫 단계의 실험으로서 우리는 국가 연구 교육 시험망의 하나인 KOREN을 이용한 국내 최초의 한 일간 천문 관측 자료의 대용량 자료 전송 실험을 실시하였다. 또한 천문학 및 천체물리학 관측 분야에서 활발하게 진행되는 세계 각국의 e-Science 분야들을 소개하고, 과학탐구활동 및 과학 대중화 활동과 어떻게 연결되어 진행되는지 소개하였다. 아직 걸음마 단계인 국내 천문 분야의 e-Science에 있어서 우리의 실험이 가지는 과학적 및 교육적 의미를 짚어보고, 머지않아 상용화될 Gbps급 인터넷 환경에 있어서 이와 같은 실험의 과학탐구활동 및 과학 대중화 활동에 있어서의 활용을 전망해 보았다. Previous Earth Science textbooks have mostly lacked the latest astronomical phenomena frequently being reported in mass media such as popular science magazines. One of the main directions in the revision of the 2009 National Curriculum of Korea is to actively include those phenomena. Furthermore, despite a close link between astronomy and physics, the concept of modern physics has not been actively introduced in Earth Science textbooks and at the same time the linkage of physics to astronomy has rarely been studies in physics textbooks. Therefore, the concept of integration among different fields in science is emphasized in the new National Curriculum. Transient phenomena in the high energy astrophysical objects are examples that reflect such issue. The purpose of this study is to introduce transferring a real-time data and making imaging of astronomical observations using e-Science. As a first step, we performed the first experiment for a large data transfer of astronomical observation between Korea and Japan using KOREN, a National Research and Education Test Network. We introduce actively on-going fields of e-Science in observational activities of astronomy and astrophysics, and their close interrelationship with scientific inquiry activities and public outreach activities. We discuss our experiment in the scientific and educational aspects to the primitive e-Science activity in the Korean astronomical society and, in turn, provide a prospective view for its application to the scientific inquiry activities and public outreach activities in the upcoming commercial Gbps-level internet environments.

I-field™ Programs Enable Operational Excellence in a Challenging Environment—Pushing the Limits of Large Data Transfer for Real-Time Monitoring and Surveillance Operations in the San Joaquin Valley

Summary Chevron's San Joaquin Valley (SJV) Business Unit (BU) is a mature, heavy-oil, thermally enhanced asset consisting of large fields that have been producing for more than a century. To ensure continuing technology transformation, in 2003 SJV started its own i-field program, being among the first BUs to pursue Chevron's enterprisewide effort in building the Digital Oil Field of the Future. From its initiation, the program has been focusing on creating solutions to manage and optimize operations and enhance recovery. This paper showcases one of the most technologically advanced i-field heavy-oil solutions implemented in SJV thermal assets, involving a high-accuracy-tiltmeter program implemented by a partner service company to advance the understanding of diatomite operations and monitor the surface ground movement. In 2008, Chevron's SJV i-field team partnered with its Information Technology (IT) department to develop a state-of-the-art application called Integrated Diatomite Operations Tool (i-DOT). The product incorporated current field workflows for mapping the grid deformation with business data (well locations and status, steam volumes and dates) in relevant time. This was accomplished by establishing a live connection between Chevron and partner-service-company servers whereby terabytes of grid data would flow seamlessly into Chevron's SJV servers. Data-acquisition, data-processing, and data-storage efforts were accomplished through an integrated effort between IT, geographical information system (GIS), and PetroWeb development teams. Workflows were built through close collaboration between i-field engineers, petroleum engineers, earth scientists, IT professionals, and the operations group. The i-DOT solution has become the single point of operation monitoring and activity scheduling for multiple groups such as Operations, Asset Development, and the Drilling team. The benefits of implementing i-DOT solutions are significant and transformed our decision-making process in diatomite fields. This paper includes not only value creation but also lessons learned during development, deployment, and end-user acceptance.

An empirical comparison of Java remote communication primitives for intra-node data transmission

This paper presents a benchmarking suite that measures the performance of using sockets and eXtensible Markup Language remote procedure calls (XML-RPC) to exchange intra-node messages between Java virtual machines (JVMs). The paper also reports on an empirical study comparing sockets and XML-RPC with response time measurements from timers that use both operating system tools and Java language instrumentation. By leveraging packet filters inside the GNU/Linux kernel, the benchmark suite also calculates network resource consumption. Moreover, the framework interprets the response time results in light of memory subsystem metrics characterizing the behavior of the JVM. The empirical findings indicate that sockets perform better when transmitting small to very large objects, while XML-RPC exhibits lower response time than sockets with extremely large bulk data transfers. The experiments reveal trade-offs in performance and thus represent the first step towards determining if Java remote communication primitives can support the efficient exchange of intra-node messages.

Analytical Modeling of Network Throughput Prediction on the Internet

Predicting network throughput is important for network-aware applications. Network throughput depends on a number of factors, and many throughput prediction methods have been proposed. However, many of these methods are suffering from the fact that a distribution of traffic fluctuation is unclear and the scale and the bandwidth of networks are rapidly increasing. Furthermore, virtual machines are used as platforms in many network research and services fields, and they can affect network measurement. A prediction method that uses pairs of differently sized connections has been proposed. This method, which we call connection pair, features a small probe transfer using the TCP that can be used to predict the throughput of a large data transfer. We focus on measurements, analyses, and modeling for precise prediction results. We first clarified that the actual throughput for the connection pair is non-linearly and monotonically changed with noise. Second, we built a previously proposed predictor using the same training data sets as for our proposed method, and it was unsuitable for considering the above characteristics. We propose a throughput prediction method based on the connection pair that uses ν-support vector regression and the polynomial kernel to deal with prediction models represented as a non-linear and continuous monotonic function. The prediction results of our method compared to those of the previous predictor are more accurate. Moreover, under an unstable network state, the drop in accuracy is also smaller than that of the previous predictor.

A Programmable Video Platform and Its Application Mapping Framework Using the Target Application's SystemC Models

HD video applications can be represented with multiple tasks consisting of tightly coupled multiple threads. Each task requires massive computation, and their communication can be categorized as asynchronous distributed small data and large streaming data transfers. In this paper, we propose a high performance programmable video platform that consists of four processing element (PE) clusters. Each PE cluster runs a task in the video application with RISC cores, a hardware operating system kernel (HOSK), and task-specific accelerators. PE clusters are connected with two separate point-to-point networks: one for asynchronous distributed controls and the other for heavy streaming data transfers among the tasks. Furthermore, we developed an application mapping framework, with which parallel executable codes can be obtained from a manually developed SystemC model of the target application without knowing the detailed architecture of the video platform. To show the effectivity of the platform and its mapping framework, we also present mapping results for an H.264/AVC 720p decoder/encoder and a VC-1 720p decoder with 30 fps, assuming that the platform operates at 200MHz.

위성 통신용 다대역 안테나

본 논문에서 소개하는 위성 통신 안테나는 대용량 데이터 전송을 위한 위성 송수신 시스템에 응용되어 추가적인 급전부 교체나 다른 기구 형상 변경 없이 X, Ku와 Ka 3중 대역의 전파 신호를 동시에 송수신할 수 있다. 설계된 안테나는 오프셋 파라볼라 안테나 형태로 2중 대역(X/Ka 대역) 급전혼, 단일대역(Ku 대역) 급전혼, 주파수 선택막(FSS : frequency selective surface) 특성을 갖는 부반사판 및 파라볼라 주반사판으로 구성되어 있다. 주반사판의 초점에는 2중 대역 급전혼이 위치하며, FSS 부반사판으로 이미지 초점을 만들어 단일 대역 급전혼이 설치되어 운용된다. 본 안테나는 전기적 특성이 국내 위성 환경에 부합되도록 3차원 EM 시뮬레이터를 이용하여 설계하였고, 측정결과 X/Ku/Ka 대역에서 안테나 이득이 각각 31.6 dBi, 36.8 dBi, 40.8 dBi 이상, 교차편파는 각각 21.7 dB, 26.6 dB, 그리고 25.2 dB 이상의 특성으로 목표 성능을 만족함을 확인하였다. The design is presented for a SATCOM antenna capable of simultaneous multi-band (X/Ku/Ka-Band) communications without replacement of feed horns or change of other parts in the application as a ground satellite terminal for large data transfer. The antenna is the offset configuration and consists of a dual-band(X/Ka-band) feed horn, a single-band(Ku-band) feed horn, a frequency selective surface(FSS) sub-reflector and a parabolic main-reflector. The antenna has a main reflector defining a prime focus and a frequency selective surface sub-reflector defining an image focus. A dual-band feed and a single-band feed are provided at each of the prime focus and image focus. The antenna is designed using 3D EM simulator and the gains measured in X/Ku/Ka-band of the complete antenna assembly is more than 31.6dBi, 36.8dBi, 40.8dBi, and the cross polarization is 21.7dB, 26.6dB, 25.2dB, respectively.

A new framework for GLIF Interdomain Resource Reservation Architecture (GIRRA)

Many existing and emerging Scientific high-end applications (E-science) require end-to-end circuits interconnecting Grid resources for large data transfers. A few advanced networks, mainly National Research and Education Networks (NRENs), such as Surfnet, National Lambda Rail and Internet 2, now provide mechanisms for end-users to reserve and provision lightpaths via middleware referred to as Network Resource Mangers (NRMs). Although, some progress has been made in automated intra-domain lightpath services, inter-domain lightpath provisioning still requires manual intervention and presents several key challenges such as scalability of topology information exchanged, consistency and scalability of information model, security of access to the resources, hybrid networking and multi-layer lightpaths, and accounting and billing. In this paper, we describe a new architectural framework called Global Lambda Integrated Facility (GLIF) Interdomain Resource Reservation Architecture (GIRRA) with the goal to provide an integrated response to these challenges. We propose a new approach to model GLIF network domains and GOLEs as virtual switches and to describe their behavior, functionality, policy capabilities, and topology aggregation. We define an inter-domain path computation model to determine paths that meet constraints and access policy restrictions. We propose a security framework for authentication and authorization of users and a model for accounting and billing that aims to provide easy and secure access to the resources. Key aspects of the GIRRA solution are that it focuses on the inter-dependence between different challenges of inter-domain path provision, and it is built around already existing solutions for intra-domain resource provisioning.

Balancing internal supply chain logistics: a comparative analysis of manufacturing and service firm operations

With the advent of high speed and large volume data transfer, logistics and supply chain management techniques have made great changes in internal supply chains. The present study dealt with three large organisations, at least regionally headquartered in the Pittsburgh, PA (US) metropolitan area, within the service industry and how they strategically leverage incentives to promote SCM-related concepts to attract and retain internal and external customers. Each segment will begin with a brief synopsis of the current situation, followed by examples of how the company incorporated logistical SCM-related systems into their strategic plans. In terms of SCM initiatives, each company was documented and their relationships explored with various operational and financial performance metrics, including information sharing, product and service customisation, long-term relationships, hedging strategies, advanced planning systems, leveraging the internet, supply network structure, and distribution network structure.

The wandering token: Congestion avoidance of a shared resource

In a distributed system where scalability is an issue, the problem of enforcing mutual exclusion often arises in a soft form: the infrequent failure of the mutual exclusion predicate is tolerated, without compromising the consistent operation of the overall system. For instance this occurs when the operation subject to mutual exclusion requires massive use of a shared resource. We introduce a scalable soft mutual exclusion algorithm, based on token passing: one distinguished feature of our algorithm is that instead of introducing an overlay topology we adopt a random walk approach. The consistency of our proposal is evaluated by simulation, and we exemplify its use in the coordination of large data transfers in a backbone based network. This algorithm is studied in the frame of the CoreGRID Institute of Grid Information, Resource and Workflow Monitoring Services, in cooperation with the FORTH Institute, in Greece.

Design of hybrid type’s optical pickup actuator for system stability in high density reading and recording

Recently, there has been a trend to have large capacity and high data transfer rate in optical disk drive. Pickup actuator also should have high performance for satisfying this trend. Since moving magnet type actuator has more simple and robust structure than moving coil type one, we designed moving magnet type to accomplish high flexible mode frequency. However, the driving sensitivity of moving magnet type actuator was low because of its characteristics as following. At first, moving parts including magnets are heavy and electromagnetic (EM) circuits are not closed-loop owing to yoke composed of paramagnetic materials. Therefore, in order to increase the driving sensitivity without the degradation of the other dynamic characteristics, EM circuits were redesigned by adding solenoid coils on existing mechanism. As a result, the hybrid type actuator composed of moving magnet type and moving coil type was suggested to obtain high sensitivities and high flexible mode frequency. Design variables of EM circuits and structure parts were decided through parametric study and design of experiments procedure. Optimization algorithm using variable metric method was used to improve performance. Based on these results, the final model was presented.

UltraScience Net: High-Performance Network Research Test-Bed

The high-performance networking requirements for next generation large-scale applications belong to two broad classes: a) high bandwidths, typically multiples of 10Gbps, to support bulk data transfers, and b) stable bandwidths, typically at much lower bandwidths, to support computational steering, remote visualization, and remote control of instrumentation. Current Internet technologies, however, are severely limited in meeting these demands because such bulk bandwidths are available only in the backbone, and stable control channels are hard to realize over shared connections. The UltraScience Net (USN) facilitates the development of such technologies by providing dynamic, cross-country dedicated 10Gbps channels for large data transfers, and 150 Mbps channels for interactive and control operations. Contributions of the USN project are two-fold: Infrastructure Technologies for Network Experimental Facility: USN developed and/or demonstrated a number of infrastructure technologies needed for a national-scale network experimental facility. Compared to Internet, USN's data-plane is different in that it can be partitioned into isolated layer-1 or layer-2 connections, and its control-plane is different in the ability of users and applications to setup and tear down channels as needed. Its design required several new components including a Virtual Private Network infrastructure, a bandwidth and channel scheduler, and a dynamic signaling daemon. The control-plane employs a centralized scheduler to compute the channel allocations and a signaling daemon to generate configuration signals to switches. In a nutshell, USN demonstrated the ability to build and operate a stable national-scale switched network. Structured Network Research Experiments: A number of network research experiments have been conducted on USN that cannot be easily supported over existing network facilities, including test-beds and production networks. It settled an open matter by demonstrating that the performance of switched connections and Multiple Protocol Label Switching tunnels over routed networks are comparable. Furthermore, such connections can be easily peered, and the performance of the resultant hybrid connections is still comparable to the constituent pure connections. USN experiments demonstrated that Infiniband transport can be effectively extended to wide-area connections of thousands of miles, which opens up new opportunities for efficient bulk data transport. USN provided dedicated connections to Cray X1 supercomputer and helped diagnose TCP performance problems which might have been otherwise incorrectly attributed to traffic on shared connections. USN contributed to the development of transport methods for dedicated connections to other traffic. Recently, experiments were conducted to assess the performance of application acceleration devices that employ flow optimization and data compression methods to improve TCP performance.

CMS offline web tools

We describe a relatively new effort within CMS to converge on a set of web based tools, using state of the art industry techniques, to engage with the CMS offline computing system. CMS collaborators require tools to monitor various components of the computing system and interact with the system itself. The current state of the various CMS web tools is described along side current planned developments.The CMS collaboration comprises of nearly 3000 people from all over the world. As well as its collaborators, its computing resources are spread all over globe and are accessed via the LHC grid to run analysis, large scale production and data transfer tasks. Due to the distributed nature of collaborators effective provision of collaborative tools is essential to maximise physics exploitation of the CMS experiment, especially when the size of the CMS data set is considered. CMS has chosen to provide such tools over the world wide web as a top level service, enabling all members of the collaboration to interact with the various offline computing components. Traditionally web interfaces have been added in HEP experiments as an afterthought. In the CMS offline we have decided to put web interfaces, and the development of a common CMS web framework, on an equal footing with the rest of the offline development.Tools exist within CMS to transfer and catalogue data (PhEDEx and DBS/DLS), run Monte Carlo production (ProdAgent) and submit analysis (CRAB). Effective human interfaces to these systems are required for users with different agendas and practical knowledge of the systems to effectively use the CMS computing system. The CMS web tools project aims to provide a consistent interface to all these tools.