Data Flow System Research Articles

Gaia Data Flow System (GDFS) Project: the UK's contribution to Gaia data processing

AbstractGaia is an ESA cornerstone mission which will observe some billion stars in the galaxy enabling micro-arcsec astrometric catalogues to be constructed. In addition Gaia will produce high quality photometric and spectroscopic catalogues.The data processing tasks are large and complex. A European consortium has been formed - the Gaia Data Processing and Analysis Consortium (DPAC). This paper describes the form of the UK Gaia Data Flow System Project contribution to the DPAC.

Canadian Forces Evaluation of the EPINATO Health Surveillance System in Bosnia-Herzegovina

The Canadian Forces (CF) adopted the EPINATO surveillance system in 1996 to monitor disease and injury morbidity in deployed settings. The Directorate of Force Health Protection, CF Health Services Group initiated an evaluation of EPINATO in Task Force Bosnia-Herzegovina in August 2003. Two methods were used to assess coding reliability: a chart audit and Sick Parade Register review. Stakeholder interviews were conducted evaluating data flow, reporting structure, and key system attributes. Reliability (K, 95% confidence interval) was good in 4 of 24 categories--sexually transmitted diseases, K = 0.75 (0.50, 1.00); eye disorders, K = 0.51 (0.15, 0.88); ears/nose/ throat, K = 0.51 (0.33, 0.69); lower respiratory infections, K = 0.49 (95% confidence interval 0.34, 0.65)-but otherwise was poor. EPINATO is not an effective, reliable tool for CF deployment health surveillance. An improved health surveillance system is required to ensure disease and injury aberrations are detected and optimal preventive programs and policies are in place for deployed CF military members.

ATLAS TDAQ DataFlow network architecture analysis and upgrade proposal

The real-time operation of the ATLAS DataFlow system is highly dependent on the performance of the Gigabit Ethernet network interconnecting its components (/spl ap/800 end nodes). After examining the functional and performance requirements of the network, several design alternatives (with respect to traffic repartition on core devices and available concentration technologies) are presented and analyzed. We introduce the use of 10 Gigabit Ethernet as a flexible and simple technology for concentrating traffic. Network testing equipment as well as discrete event model simulations are used to assess the performance of various implementation options. Based on performance, fault tolerance and flexibility considerations a preferred architecture is proposed for implementation.

Sub Farm Interface of the ATLAS Dataflow System

Sub Farm Interface is the event builder of the ATLAS(A Toroidal LHC ApparatuS) Dataflow System. It receives event fragments from the Read Out System, builds full events and sends complete events to the Event Filter for high level event selection. This paper describes the implementation of the Sub Farm Interface. Furthermore, this paper introduces some issues on SFI(Sub Farm Interface) optimization and the monitoring service inside SFI.

OptionStream: An automated system for tracking derivative effects on equity prices

We present the design and development of a flow-based software system that became necessary for the study of a class of options-based price-estimators in the financial markets. Because a combination of factors-cost, reliability, uniformity and convention-made it virtually impossible to obtain historical options data, we developed a data-flow system to capture, process and analyze streaming data over a period of over a year. The system utilizes distributed processing nodes with checkpointing to process input data streams and reliably compute a variety of estimator updates for studying relationships between equity prices and the functions of corresponding option-related variables. The flow-based architecture is designed to support the high-volume data characteristics of the options market, along with the severely taxing computational requirements of hypothetical option-pricing so that experimental investigation of novel estimators becomes possible with current and historical data. Features such as high-volume stream-processing, intermediate checkpointing and load distribution make the design viable for more general streaming data processing applications. .

The base-line DataFlow system of the ATLAS trigger and DAQ

The base-line design and implementation of the ATLAS DAQ DataFlow system is described. The main components of the DataFlow system, their interactions, bandwidths, and rates are discussed and performance measurements on a 10% scale prototype for the final ATLAS TDAQ DataFlow system are presented. This prototype is a combination of custom design components and of multithreaded software applications implemented in C++ and running in a Linux environment on commercially available PCs interconnected by a fully switched gigabit Ethernet network.

VLTI Observing Preparation Sequence

Optical interferometers are instruments that combine the light coming from separated optical telescopes in order to get information with the spatial resolution of the telescope array. One of the major interferometer is the Very Large Telescope Interferometer (VLTI) . To observe an astronomical target with the VLTI, like with any other interferometers, one needs to carefully prepare the observations because of the numerous parameters to be fixed. In addition, the VLTI is being developed in the framework of the data flow system created for the Very Large telescopes (VLT) at the European Southern Observatory (ESO) . I recall the philosophy of the ESO/VLT data flow system and its adaptation to the VLTI data and then I describe the various steps required to prepare a VLTI observation. Finally, I illustrate the various steps of this preparation with an example focused in the observations of the young binary system Z CMa.

Dynamic Compilation of C++ Template Code

Generic programming using the C++ template facility has been a successful method for creating high‐performance, yet general algorithms for scientific computing and visualization. However, adding template code tends to require more template code in surrounding structures and algorithms to maintain generality. Compiling all possible expansions of these templates can lead to massive template bloat. Furthermore, compile‐time binding of templates requires that all possible permutations be known at compile time, limiting the runtime extensibility of the generic code. We present a method for deferring the compilation of these templates until an exact type is needed. This dynamic compilation mechanism will produce the minimum amount of compiled code needed for a particular application, while maintaining the generality and performance that templates innately provide. Through a small amount of supporting code within each templated class, the proper templated code can be generated at runtime without modifying the compiler. We describe the implementation of this goal within the SCIRun dataflow system. SCIRun is freely available online for research purposes.

Vertical Monitoring: a data acquisition application for the BaBar experiment

Online Dataflow (ODF) handles data acquisition and processing from the BaBar detector front end electronics through the delivery of complete events to a farm of UNIX nodes. To understand the system, a monitor application, Vertical Monitoring (VMON), has been developed. VMON multicasts UDP packets with performance information from each CPU of the Dataflow system. This information is currently received by a single client on one UNIX node and archived. A ROOT-based GUI is provided on the UNIX application side to control VMON, display the performance of the ODF system in real time and retrieve archived data. VMON is currently used successfully in BaBar to provide diagnostics in the dataflow system and help with projections of future performance.

Java support for data-intensive systems

Database system designers have traditionally had trouble with the default services and interfaces provided by operating systems. In recent years, developers and enthusiasts have increasingly promoted Java as a serious platform for building data-intensive servers. Java provides a number of very helpful language features, as well as a full run-time environment reminiscent of a traditional operating system. This combination of features and community support raises the question of whether Java is better or worse at supporting data-intensive server software than a traditional operating system coupled with a weakly-typed language such as C or C++.In this paper, we summarize and discuss our experience building the Telegraph dataflow system in Java. We highlight some of the pleasures of coding with Java, and some of the pains of coding around Java in order to obtain good performance in a data-intensive server. For those issues that were painful, we present concrete suggestions for evolving Java's interfaces to better suit serious software systems development. We believe these experiences can provide insight for other designers to avoid pitfalls we encountered and to decide if Java is a suitable platform for their system.

The NA48 event-building PC farm

The NA48 experiment at the CERN SPS aims to measure the parameter Re(/spl epsiv/'//spl epsiv/) of direct CP violation in the neutral kaon system with an accuracy of 2x10/sup -4/. Based on the requirements of: high event rates (up to 10 kHz) with negligible dead time; support for a variety of detectors with very wide variation in the number of readout channels; data rates of up to 150 MByte/s sustained over the beam burst; level-3 filtering and remote data logging in the CERN computer center; the collaboration has designed and built a modular pipelined data flow system with 40 MHz sampling rate. The architecture combines custom-designed components with commercially available hardware for cost effectiveness and flexibility. To increase the available data bandwidth and to add filtering and monitoring capabilities, the original custom-built event builder hardware has been replaced by a farm of 24 Intel PentiumII based PCs running the Linux operating system during the shutdown between the 1997 and 1998 data taking periods. During the data taking period 1998 the system has been successfully operated taking ca. 70 Terabyte of data.

Probabilistic data flow system with two-edge profiling

Traditionally optimization is done statistically independent of actual execution environments. For generating highly optimized code, however, runtime information can be used to adapt a program to different environments. In probabilistic data flow systems runtime information on representative input data is exploited to compute the probability with what data flow facts may hold. Probabilistic data flow analysis can guide sophisticated optimizing transformations resulting in better performance. In comparison classical data flow analysis does not take runtime information into account. All paths are equally weighted irrespectively whether they are never, heavily, or rarely executed. In this paper we present the best solution what we can theoretically obtain for probabilistic data flow problems and compare it with the state-of-the-art one-edge approach. We show that the differences can be considerable and improvements are crucial. However, the theoretically best solution is too expensive in general and feasible approaches are required. In the sequel we develop an efficient approach which employs two-edge profiling and classical data flow analysis. We show that the results of the two-edge approach are significantly better than the state-of-the-art one-edge approach.

A High-Level Dataflow System

This paper presents a new dataflow graph model, where only data tokens are allowed to flow. First we introduce a High-Level Dataflow System (HLDS) to describe a formal dataflow graph model, then we present a homogeneous HLDS (hHLDS) that formally describes our proposal. In this proposal the dataflow graph is obtained by employing only actors with homogeneous I/O conditions, that is, each actor, which executes an elemental operation, is characterised by having one output and two input arcs. Even though no control tokens are allowed, i.e. no T-gate, merge, and switch actors are present in this model, it is always possible to obtain dataflow graphs, which represent any programming structure and whose behaviour is well-behaved. As homogeneous I/O conditions are a severe restriction to represent the flow of a computation and the token flow in such dataflow graphs is completely asynchronous, proof is given to guarantee their determinacy. The main advantage of this representation is that it maps directly to hardware through a one-to-one correspondence between actors of the model and Functional Units of a dataflow machine.

Design of cache memories for dataflow architecture

Abstract The recent advance in dataflow processing — to combine the dataflow paradigm with the control-flow paradigm — has brought out many new challenging issues. This hybrid organization has made it possible to study and adapt familiar control-flow concepts such as cache memories within the framework of the dataflow architecture. The concept of cache memory has proven its effectiveness in the von Neumann architecture due to the spatial and temporal localities which govern the organization of the conventional programming execution. A dataflow paradigm, does not informally support locality, since the execution sequence is enforced only by the availability of operands. However, dataflow programs can be reordered based on various criteria to enhance the locality of instruction references. This can be achieved by: (i) careful partitioning of a dataflow program into vertical layers of data dependent instructions; and (ii) proper distribution and allocation of the recurrence portions of the dataflow program. Enhancing the locality of data references in the dataflow architecture is a more challenging problem. This paper studies the design of instruction, data (operand), and I-Structure cache memories using the Explicit Token Store (ETS) model of dataflow system. The performance results obtained using various benchmark programs are presented and analyzed.

Health and Safety in Australian Agriculture - Linking Research, Data and Action

ABSTRACT Almost every week the rural media provide farming communities around Australia with information about health and safety hazards on farms. The results of the increasing research and data collection are being made broadly available. But are we just adding yet another worry to already stressed farmers, farm workers and farm families by this approach? Until recently, there was little information or support available to farmers to assist them to identify the hazards on their own workplace, to assess the relative risk to health and safety of the many hazards identified, or to plan adoption of effective solutions to reduce risk based on the cash flow of the farming enterprise. Managing Farm Safety is a program which has been developed through integration of data findings, research findings, farmer responsibilities under OHS Occupational Health and Safety legislation, farm management practicalities, farm cash flow and farmer aspirations. The program will be incorporated into rural education, training and extension services. The services to farmers and farm families include access to training in farm OHS management, provision of a management tool or kit which includes and on-farm audit, record of education and training for employees, record of farm chemicals usage, and information about risk reduction solutions are available, their relative effectiveness and the cost of each option. The program will be provided in commodity specific format as more of the major industry groups in Australian agriculture move to adopt the approach. This paper describes the data flow system used in the development and maintenance of this program.

Modeling, design, and performance analysis of a parallel hybrid data/command driven architecture system and its scalable dynamic load balancing circuit

Addresses a proposed parallel hybrid dataflow architecture, a scalable dynamic load balancing circuit for the proposed architecture, and performance analysis, first, of the load balancing circuit and, second, the architecture using the load balancing circuit. The contributions and focus of this paper are: (1) that it first describes the requirements for and the framework of a parallel, medium to coarse grain, hybrid token controlled dataflow architecture. This paper only deals with real-time applications of the architecture. (2) The paper next describes the dynamic load balancing strategy for the hybrid dataflow architecture and a resulting mathematical model of the load balancing function required by the load balancing strategy. (3) The organization, design, and implementation of a basic digital circuit suitable for VLSI implementation which implements the mathematical model of the load balancing function required by the architecture is next presented. This circuit implements a control token mapping function and is therefore called a "token mapper". (4) It is next shown that the basic dynamic load balancing circuit (token mapper) design is scalable therefore allowing the hybrid dataflow architecture to be scalable. (5) The performance of the dynamic load balancing circuit is then analyzed at both the circuit and architectural systems level. A parallel simulation of the proposed parallel hybrid dataflow architecture employing its presented dynamic load balancing circuit was developed for two example applications and used for architectural system level performance analysis. Analysis of simulation results verified correct operation of the proposed hybrid dataflow system architecture and its dynamic load balancing circuit.

Parallel architectures: Cache memories for dataflow systems

Cache memory — so effective in traditional control-flow architecture — has the potential to enhance dataflow system performance as well. The authors explore the recent trend in combining dataflow and control-flow processing, which offers new alternatives in computer architecture design, and analyze cache memory's application to the dataflow environment.

Cheops: a reconfigurable data-flow system for video processing

The Cheops Imaging System is a compact, modular platform for acquisition, processing, and display of digital video sequences and model-based representations of moving scenes, and is intended as both a laboratory tool and a prototype architecture for future programmable video decoders. Rather than using a large number of general-purpose processors and dividing up image processing tasks spatially, Cheops abstracts out a set of basic, computationally intensive stream operations that may be performed in parallel and embodies them in specialized hardware. We review the Cheops architecture, describe the software system that has been developed to perform resource management, and present the results of some performance tests.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Performance Studies of Id on the Monsoon Dataflow System

In this paper, we examine the performance of Id, an implicitly parallel language, on Monsoon, an experimental dataflow machine. One of the precepts of our work is that the Id run-time system and compiled Id programs should run on any number of Monsoon processors without change. Our experiments running Id programs on Monsoon show that speedups of more than 7 are easily achieved on 8 processors for most of the applications that we studied. We explain the sources of overhead that limit the speedup of each of our benchmark programs. We also compare the performance of Id on a single Monsoon processor with C/Fortran on a DEC Station 5000 (MIPS R3000 processor), to establish a baseline for the efficiency of Id execution on Monsoon. We find that the execution of Id programs on one Monsoon processor takes up to three times as many cycles as the corresponding C or Fortran programs executing on a MIPS R3000 processor. We identify the sources of inefficiency on Monsoon and suggest improvements, where possible. In many cases, however, improving single processor performance will reduce parallel processor performance.

Block scheduling of iterative algorithms and graph-level priority scheduling in a simulated data-flow multiprocessor

Iterative methods for solving linear systems are discussed. Although these methods are inherently highly sequential, it is shown that much parallelism could be exploited in a data-flow system by scheduling the iterative part of the algorithms in blocks and by looking ahead across several iterations. This approach is general and will apply to other iterative and loop-based problems. It is also demonstrated by simulation that relying solely on data-driven scheduling of parallel and unrolled loops results in low resource utilization and poor performance. A graph-level priority scheduling mechanism has been developed that greatly improves resource utilization and yields higher performance.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>