Complex Data Flow Research Articles

Ophidia: Toward Big Data Analytics for eScience

This work introduces Ophidia, a big data analytics research effort aiming at supporting the access, analysis and mining of scientific (n-dimensional array based) data. The Ophidia platform extends, in terms of both primitives and data types, current relational database system implementations (in particular MySQL) to enable efficient data analysis tasks on scientific array-based data. To enable big data analytics it exploits well-known scientific numerical libraries, a distributed and hierarchical storage model and a parallel software framework based on the Message Passing Interface to run from single tasks to more complex dataflows. The current version of the Ophidia platform is being tested on NetCDF data produced by CMCC climate scientists in the context of the international Coupled Model Intercomparison Project Phase 5 (CMIP5).

Fluχ: a quality-driven dataflow model for data intensive computing

Today, there is a growing need for organizations to continuously analyze and process large waves of incoming data from the Internet. Such data processing schemes are often governed by complex dataflow systems, which are deployed atop highly-scalable infrastructures that need to manage data efficiently in order to enhance performance and alleviate costs.Current workflow management systems enforce strict temporal synchronization among the various processing steps; however, this is not the most desirable functioning in a large number of scenarios. For example, considering dataflows that continuously analyze data upon the insertion/update of new entries in a data store, it would be wise to assess the level of modifications in data, before the trigger of the dataflow, that would minimize the number of executions (processing steps), reducing overhead and augmenting performance, while maintaining the dataflow processing results within certain coverage and freshness limit.Towards this end, we introduce the notion of Quality-of-Data (QoD), which describes the level of modifications necessary on a data store to trigger processing steps, and thus conveying in the level of performance specified through data requirements. Also, this notion can be specially beneficial in cloud computing, where a dataflow computing service (SaaS) may provide certain QoD levels for different budgets.In this article we propose F l u χ, a novel dataflow model, with framework and programming library support, for orchestrating data-based processing steps, over a NoSQL data store, whose triggering is based on the evaluation and dynamic enforcement of QoD constraints that are defined (and possibly adjusted automatically) for different sets of data. With F l u χ we demonstrate how dataflows can be leveraged to respond to quality boundaries that bring controlled and augmented performance, rationalization of resources, and task prioritization.

Managing data dependencies in service compositions

Composing services into service-based systems requires the design of coordination logic, which describes all service interactions realizing the composition. Coordination can be defined as the management of dependencies; in a services context we can discriminate between ‘control flow’ that manages sequence dependencies and ‘data flow’ for managing data dependencies. Current research fails to address the management of data dependencies in a systematic way and mostly treats it as subordinate to sequence dependencies. In this article a ‘data flow’ pattern language is presented that provides a systematic way of designing the data flow aspects of a coordination scenario, orthogonally to the way in which the control flow is designed. Starting from a set of fundamental and basic building blocks, each data dependency will yield a data flow design that takes a set of design criteria (e.g. loose coupling, data confidentiality, etc.) into account. The pattern language is evaluated in three ways. First, it is shown that every potential coordination scenario for managing a data dependency can be composed by the set of patterns. Second, the pattern language was applied in a real-life insurance case to show how it can guide the design of complex data flows. Third, the patterns were implemented in a tool that provides configurable model-to-code transformations for automatically generating BPEL coordination scenarios. In this tool both the data flow and control flow can be designed separately using different sets of patterns.

Management of complex data flows in the ASDEX Upgrade plasma control system

Establishing adequate technical and physical boundary conditions for a sustained nuclear fusion reaction is a challenging task. Phased feedback control and monitoring for heating, fuelling and magnetic shaping is mandatory, especially for fusion devices aiming at high performance plasmas. Technical and physical interrelations require close collaboration of many components in sequential as well as in parallel processing flows. Moreover, handling of asynchronous, off-normal events has become a key element of modern plasma performance optimisation and machine protection recipes.The manifoldness of plasma states and events, the variety of plant system operation states and the diversity in diagnostic data sampling rates can hardly be mastered with a rigid control scheme. Rather, an adaptive system topology in combination with sophisticated synchronisation and process scheduling mechanisms is suited for such an environment. Moreover, the system is subject to real-time control constraints: response times must be deterministic and adequately short.Therefore, the experimental tokamak device ASDEX Upgrade employs a discharge control system DCS, whose core has been designed to meet these requirements. In the paper we will compare the scheduling schemes for the parallelised realisation of a control workflow and show the advantage of a data-driven workflow over a managed workflow. The data-driven workflow as used in DCS is based on signals connecting process outputs and inputs. These are implemented as real-time streams of data samples. Consequently, real-time signal management forms the foundation of DCS. The paper explains the principal features such as tagged samples, signal groups, algorithmic blocks and processes as well as scheduling schemes which allow DCS control applications to be defined as self-contained modular building blocks glued together by a software framework.By virtue of this sound foundation, DCS is a mature but still evolving system for reliable, distributed control of an entire tokamak device coordinating and monitoring 20 diagnostic systems, 14 magnetic power supplies, 5 heating systems with a total power of more than 25MW, 8 gas fuelling channels, a pellet injector and a killer gas gun.

Spatial Data-mining Technology Assisting in Petroleum Reservoir Modeling

Abstract Petroleum reservoir modeling integrates multi- discipline research works which involve much more data type, huge data and complex data-flow. It is very important to bring the spatial data-mining (SDM) method into reservoir modeling. According to the characterization of reservoir modeling procedure, spatial data-mining can be used in reservoir geological research and ranking reservoir modeling. Geo-statistical method, spatial analysis method, visualizing method and non-linear method are the most effective methods for reservoir modeling based on the type of reservoir knowledge. Spatial-overlapping method is used for studying the relationship between sedimentary micro-facies and oil production with statistical method. The distribution knowledge of sedimentary micro-facies is built by geo-statistical method. The visualization method is used for checking the exactness of reservoir model. The application in Wenxi I Block of Tuha oil field is shown that spatial data-mining technology not only improves the efficient of reservoir modeling, but also verifies the knowledge obtained from each researching stage so as to guarantee the integrality and exactness of reservoir knowledge.

A New Data Mining Method based on Multidimensional-Data Flow

Data mining, this is the whole fish bone drive a new term, is an abstract concept and improve the multidimensional-data bones application process data mining technology is applied to process optimization and structure of the data mining. End-to-end DMDF figure including complex data flow and different processing composition and improve the level of breeding. DMDF provide comprehensive platform and mixed methods to support the whole life cycle of comprehensive method data mining. Data pretreatment, data classification, association rules mining and prediction of the foundation and contact the whole life cycle of the data mining process. DMDF support several different mining components from the strategic level, tactics level of abstraction level, and then design a data mining system to realize the reasonable process execution buildings. Is a new direction DMDF structure of the data mining process.

A Hardware-Efficient Multi-Resolution Block Matching Algorithm and its VLSI Architecture for High Definition MPEG-Like Video Encoders

High throughput, heavy bandwidth requirement, huge on-chip memory consumption, and complex data flow control are major challenges in high definition integer motion estimation hardware implementation. This paper proposes an efficient very large scale integration architecture for integer multi-resolution motion estimation based on optimized algorithm. There are three major contributions in this paper. First, this paper proposes a hardware friendly multi-resolution motion estimation algorithm well-suited for high definition video encoder. Second, parallel processing element (PE) array structure is proposed to implement three-level hierarchical motion estimation, only 256PEs are enough for one reference frame real-time high definition motion estimation by efficient PE reuse. Third, efficient on-chip reference pixel buffer sharing mechanism between integer and fractional motion estimation is proposed with almost 50% SRAM saving and memory bandwidth reduction. The proposed multi-resolution motion estimation algorithm reached a good balance between complexity and performance with rate distortion optimized variable block size motion estimation support. Also, we have achieved moderate logic circuit and on-chip SRAM consumption. The proposed architecture is well-suited for all MPEG-like video coding standards such as H.264, audio video coding standard, and VC-1.

A profile-based tool for finding pipeline parallelism in sequential programs

Traditional static analysis fails to auto-parallelize programs with a complex control and data flow. Furthermore, thread-level parallelism in such programs is often restricted to pipeline parallelism, which can be hard to discover by a programmer. In this paper we propose a tool that, based on profiling information, helps the programmer to discover parallelism. The programmer hand-picks the code transformations from among the proposed candidates which are then applied by automatic code transformation techniques. This paper contributes to the literature by presenting a profiling tool for discovering thread-level parallelism. We track dependencies at the whole-data structure level rather than at the element level or byte level in order to limit the profiling overhead. We perform a thorough analysis of the needs and costs of this technique. Furthermore, we present and validate the belief that programs with complex control and data flow contain significant amounts of exploitable coarse-grain pipeline parallelism in the program’s outer loops. This observation validates our approach to whole-data structure dependencies. As state-of-the-art compilers focus on loops iterating over data structure members, this observation also explains why our approach finds coarse-grain pipeline parallelism in cases that have remained out of reach for state-of-the-art compilers. In cases where traditional compilation techniques do find parallelism, our approach allows to discover higher degrees of parallelism, allowing a 40% speedup over traditional compilation techniques. Moreover, we demonstrate real speedups on multiple hardware platforms.

Recent Bioinformatics Advances in the Analysis of High Throughput Flow Cytometry Data

Recent Bioinformatics Advances in the Analysis of High Throughput Flow Cytometry Data

The Daily Commute: An Analysis of the Geography of the Labour Market using 2006 Census Data

This paper explores the major commuting areas within New Zealand and how commuting patterns have evolved between 1996 and 2006. It focuses primarily on the new insights that mapping and visualisation methods can bring to the analysis and understanding of complex flow data. In particular, we discuss some approaches to delineating labour market areas based on commuter inflow statistics and demonstrate the advantages that spider flow maps bring to the visualisation and understanding of commuting flows between areas. Spider flow maps are based on origin-destination information from the 2006 Census, but the paper also includes an historical perspective, examining changes in, the number and proportion of people commuting between areas and using different modes of transport used for commuting. Although our focus is on the advantages that these new methods can bring to the analysis of commuting data, some interesting findings arise. Both the number of commutes and the distance travelled by commuters has increased markedly since 1996, particularly around the largest cities of the Auckland. Wellington and Christchurch. Labour markets centered on these cities go well beyond territorial authority boundaries and. particularly in the Auckland case, are becoming increasingly polycentric. Data visucalisation makes the exploration of these patterns much more accessible.

Workflow-Based Data Parallel Applications on the EGEE Production Grid Infrastructure

Setting up and deploying complex applications on a Grid infrastructure is still challenging and the programming models are rapidly evolving. Efficiently exploiting Grid parallelism is often not straight forward. In this paper, we report on the techniques used for deploying applications on the EGEE production Grid through four experiments coming from completely different scientific areas: nuclear fusion, astrophysics and medical imaging. These applications have in common the need for manipulating huge amounts of data and all are computationally intensive. All the cases studied show that the deployment of data intensive applications require the development of more or less elaborated application-level workload management systems on top of the gLite middleware to efficiently exploit the EGEE Grid resources. In particular, the adoption of high level workflow management systems eases the integration of large scale applications while exploiting Grid parallelism transparently. Different approaches for scientific workflow management are discussed. The MOTEUR workflow manager strategy to efficiently deal with complex data flows is more particularly detailed. Without requiring specific application development, it leads to very significant speed-ups.

Concepts for Visualization of Multidirectional Phase-contrast MRI of the Heart and Large Thoracic Vessels

Multidirectional phase-contrast magnetic resonance imaging allows the acquisition of time-resolved velocity fields (vectors) of cardiac and vascular blood flow. Its unique ability to provide vectorial flow information promises to give new insights into hemodynamic physiology. However, up to now appropriate and standardized procedures and software tools are missing to take advantage of all the information contained in the data. The objective of this work is to present a new versatile software tool and to demonstrate its practical value for the examination of multidirectional blood flow. An exemplary selection of data sets from healthy volunteers, patients with cardiovascular pathologies, and healthy domestic pigs has been acquired using a phase-contrast magnetic resonance imaging sequence based on FLASH (fast low angle shot) that encodes velocity as field of three-dimensional vectors. For data processing, we have developed a software tool that integrates the whole workflow, including noise filtering, interactive visualization, and flow quantification. Using the software tool visualization of complex flow data is easily generated within 5 minutes; interactive exploration of the data is possible in real-time. Exemplary physiologic and pathologic flow patterns were visualized in an intuitive manner. The visual results suggest valuable diagnostic information; its significance, however, must be further evaluated together with the development of more specific data processing. Multidirectional phase-contrast magnetic resonance imaging is a valuable tool for assessment of cardiac and vascular hemodynamics. With the development of tools that offer standardized and thus comparable visualizations it may be integrated into the clinical routine in the near future.

Overview: Knowledge Management and Training (December 2007)

The topic of knowledge management has matured considerably over recent years, evolving from abstract discussion about the translation of raw data into useful information on to knowledge while presenting an exposé of the differences between explicit and tacit knowledge. These dialogues, more often than not, lacked cohesion as well as a clear impression of the effect on the industry's bottom line. In reviewing the various papers for this year's feature, I have to remark about the superior quality and high degree of relevance of the various publications. One can witness a foundation of data acquisition and management in a continuum, which spans from the large volumes of data that comprise the static domain of the shared Earth model to the capacity to collect data on a continuous basis along the full profile of the wellbore. The latter is a clear demonstration of the great strides in technology made by the providers of data-gathering systems. Until recently, many knowledge-management initiatives were hindered by an environment that was unable to make immediate and efficient use of all the data being acquired to benefit the operation of the asset. A mainstay in the downstream operation of refinery and pipeline systems, the upstream E&P industry has evolved slowly in accepting the value of real-time data. Clearly, we are now at an advanced stage of integration of technology and information that does, indeed, drive decision making on a continuous basis. Coupling the advances described above with the ascribing of value to specific information provides added capability in making decisions that are bottom-line driven in an industry that depends increasingly on marginal assets as we focus on sustainability of current and future reserves. The papers featured this month provide an interesting wealth of information on the state of play of data-gathering technology, integration initiatives, operational deployment of digital oilfield environments, and the use of mentoring as a means of capturing and retaining knowledge. These are all excellent ingredients in a recipe for the continuing success of integrating knowledge management into mainstream E&P operations. Knowledge Management and Training additional reading available at the SPE eLibrary: www.spe.org SPE 106916 "From Sensors to Models to Visualization—Handling the Complex Data Flow" by Ø. Kolnes, SINTEF Petroleum Research, et al. SPE 103287 "Sustained Competitive Advantage Through Structured Mentoring" by G. Arango, SPE, Hughes Christensen, et al. SPE 108175 "Value-of-Information Applications in Unconventional Resource Plays" by Patrick E. Leach, Decision Strategies, et al.

Identifying Candidate Normal and Leukemic B Cell Progenitor Populations with Hierarchical Clustering of 6-Color Flow Cytometry Data - A Better View.

6-color flow cytometry allows multiparameter analysis of high numbers of single cells. It is an excellent tool for the characterization of a wide range of hematopoietic populations and for monitoring minimal residual disease. However, analysis of complex flow data is challenging. Gating populations on 28 two-parameter plots is extremely tedious and does not reflect the multidimensionality of the data. Here, we describe a novel approach, employing hierarchical clustering (HCA) and support vector machine (SVM) learning in analyzing flow data. This approach provides a new perspective for looking at flow data and promises better identification of rare and novel subpopulations that escape classic analysis. Our aim was to identify normal and leukemic B cell progenitor/stem cell populations in normal (n=6) and ALL (n=10) bone marrow. Samples were labelled with fluorochrome-conjugated antibodies to 6 CD markers (CD 10, 19, 22, 34, 38, 117) and 104 to 106 events were acquired (FACSCanto, BD Biosciences). To analyze flow data with HCA we developed a new algorithm, better suited for the ellipsoid nature of cell populations than other current HCA metrics. Data exported from DiVa software were externally compensated and Hyperlog transformed to achieve a logarithmic-like scale that displayed zero and negative values. Normalized data were then subjected to HCA employing a scale-invariant Mahalanobis distance measurement for merging clusters. This reflects the extended ellipsoid shape of the populations (here: 8 dimensional ellipsoids). We developed a new adaptive linkage algorithm that smoothly shifts from the Euclidean distance (when clusters are too small to compute Mahalanobis distance) to Mahalanobis distance measurement. This allowed us to build the hierarchy from single events, yet to retain the advantage of Mahalanobis measurement for larger clusters. To build classifiers we used SVM employing polynomial kernel. All work was carried out in MATLAB (MathWorks, Inc.). The resulting hierarchical tree combined with the heatmap of the CD marker expression allows visualization of hierarchically clustered data with all 8 parameters displayed in a single plot (!) as compared to 28 traditional two-parameter plots. HCA has big advantage of providing populations homogenous in their expression pattern of all parameters (without the need for complex sub or back gating). We were able to identify populations corresponding to the different stages of B-cell development. In a normal control bone marrow we could detect the following candidate B-lineage progenitor populations: CD34+117+38+10−22−19− (0.94% of total) progenitor/stem cells, CD34+117−38+10+22+19med (0.26% of total) pro-B cells, CD34−117−38+10+22+19+ (2.77% of total) small pre-B cells (lower FCS values), CD34−117−38+10+22+19+ (1.09% of total) large pre-B cells (higher FCS values) and CD34−117−38lo10−22+19+ (5.94% of total) (immature) B cells. In 10 diagnostic or relapse samples HCA clearly identified the main leukemic population. HCA is able to visualize otherwise “hidden” populations. This was exemplified by a distinct CD38+B-lin− population that overlapped with other populations in all 28 two-parameter plots (most likely T cells). We have built a classifier able to find established populations across samples and in large datasets (106 events) for which HCA would be computationally too demanding. In summary, we show the advantages of using hierarchical clustering analysis for large complex multiparameter flow cytometry datasets.

A practical dynamic single assignment transformation

This paper presents a novel method to construct a dynamic single assignment (DSA) form of array intensive, pointer free C programs. A program in DSA form does not perform any destructive update of scalars and array elements; that is, each element is written at most once. As DSA makes the dependencies between variable references explicit, it facilitates complex analyses and optimizations of programs. Existing transformations into DSA perform a complex data flow analysis with exponential analysis time, and they work only for a limited class of input programs. Our method removes irregularities from the data flow by adding copy assignments to the program, so that it can use simple data flow analyses. The presented DSA transformation scales very well with growing program sizes and overcomes a number of important limitations of existing methods. We have implemented the method and it is being used in the context of memory optimization and verification of those optimizations. Experiments show that in practice, the method scales well indeed, and that added copy operations can be removed in case they are unwanted.

AN ANALYTICAL METHOD FOR PARALLELIZATION OF RECURSIVE FUNCTIONS

Programming with parallel skeletons is an attractive framework because it encourages programmers to develop efficient and portable parallel programs. However, extracting parallelism from sequential specifications and constructing efficient parallel programs using the skeletons are still difficult tasks. In this paper, we propose an analytical approach to transforming recursive functions on general recursive data structures into compositions of parallel skeletons. Using static slicing, we have defined a classification of subexpressions based on their data-parallelism. Then, skeleton-based parallel programs are generated from the classification. To extend the scope of parallelization, we have adopted more general parallel skeletons which do not require the associativity of argument functions. In this way, our analytical method can parallelize recursive functions with complex data flows.

AN ANALYTICAL METHOD FOR PARALLELIZATION OF RECURSIVE FUNCTIONS

Programming with parallel skeletons is an attractive framework because it encourages programmers to develop efficient and portable parallel programs. However, extracting parallelism from sequential specifications and constructing efficient parallel programs using the skeletons are still difficult tasks. In this paper, we propose an analytical approach to transforming recursive functions on general recursive data structures into compositions of parallel skeletons. Using static slicing, we have defined a classification of subexpressions based on their data-parallelism. Then, skeleton-based parallel programs are generated from the classification. To extend the scope of parallelization, we have adopted more general parallel skeletons which do not require the associativity of argument functions. In this way, our analytical method can parallelize recursive functions with complex data flows.

An Analytical Method for Parallelization of Recursive Functions

Programming with parallel skeletons is an attractive framework because it encourages programmers to develop efficient and portable parallel programs. However, extracting parallelism from sequential specifications and constructing efficient parallel programs using the skeletons are still difficult tasks. In this paper, we propose an analytical approach to transforming recursive functions on general recursive data structures into compositions of parallel skeletons. Using static slicing, we have defined a classification of subexpressions based on their data-parallelism. Then, skeleton-based parallel programs are generated from the classification. To extend the scope of parallelization, we have adopted more general parallel skeletons which do not require the associativity of argument functions. In this way, our analytical method can parallelize recursive functions with complex data flows.

An architecture for control and monitoring of discrete events systems

In a first part this work presents a hierarchical and modular architecture for real-time control and monitoring of Discrete Events Systems like Flexible Manufacturing Systems (F.M.S.). The first characteristic of this architecture is based on a splitting up procedure to build a hierarchical structure allowing to manage the complexity of the process. The second characteristic concerns the organization of this structure to drive the process. On the one hand, control system, monitoring system and model exhibiting the process capabilities are distributed in each node of this hierarchical structure. So each of these nodes is a `control and monitoring module'. On the other hand, a further process model more complete than the one included in each control and monitoring module is added to the hierarchical structure. The last contribution of this work introduces an inter-levels communication mechanism suitable for real-time control and monitoring requirements. The addition of a supervisor distributed in each control and monitoring module allows to manage the complex data flow taking part in the inter-levels communication.

Modular architecture for high performance implementation of 2-dimensional fast Fourier transform

The paper proposes new structural schemes of high-throughput processors executing 2-dimensional fast Fourier transforms (FFTs) by the column–row method and the vector-radix algorithm. These structures offer a simplified system of data commutation (eliminating cross-commutations of complex data flows), and a simple ordering of input array without preliminary accumulation. These structures also eliminate overlap of input arrays and matrix-transposition devices during computation.