Dataflow Programming Research Articles

A two-level formal model for Big Data processing programs

• A high-level model for iterative and non-iterative large-scale data processing programs. • Formalizing data flow with Petri Nets and data processing operations with Monoid Algebra. • Application of the model to formalize mutation operators for mutation testing in data flow programs. • Proposal of new mutation operators for iterative data flow programs. This paper proposes a model for specifying data flow-based parallel data processing programs agnostic of target Big Data processing frameworks. The paper focuses on the formal abstract specification of non-iterative and iterative programs, generalizing the strategies adopted by data flow Big Data processing frameworks. The proposed model relies on Monoid Algebra and Petri Nets to abstract Big Data processing programs in two levels: a higher level representing the program data flow and a lower level representing data transformation operations (e.g., filtering, aggregation, join). We extend the model for data processing programs proposed in [1] , for modeling iterative data processing programs. The general specification of these programs implemented by data flow-based parallel programming models is essential given the democratization of iterative and greedy Big Data analytics algorithms. Indeed, these algorithms call for revisiting parallel programming models to express iterations. The paper gives a comparative analysis of the iteration strategies proposed by Apache Spark, DryadLINQ, Apache Beam, and Apache Flink. It discusses how the model achieves to generalize these strategies.

An Interactive Interpreter for Two Dimensional Lucid

We present an initial draft of the “Luminous" interactive interpreter for a new dialect of the functional dataflow language Lucid. Luminous is not the first implementation of Lucid but it is novel in several ways. First, the dialect is two dimensional (has a space as well as a time dimension) with special space operators. And it performs dimensional analysis to determine, for each program variable, which dimensions may be required to evaluate a variable and which are irrelevant. This information is vital for producing output and for avoiding expensive redundancy in the cache. Also, Luminous is interactive: the user can demand the value of an arbitrary expression and immediately see the results in an appropriate form, depending on the dimensionality of the expression. The user can also alter existing definitions and re-demand the value of expressions, to see the effect of the change. Unlike previous Lucid implementation, Luminous operates directly on the source - it doesn’t build a parse tree. The source is ‘dismantled’ into elementary equations in a bottom-up procedure based on program transformation which skips building a parse tree. Dismantling is a non deterministic distributable dataflow procedure. This could become very important when (in the future) we deal with programs millions of lines long, when parsing is a major problem. The final result of dismantling is an (unordered) set of ‘atomic’ equations – so called because they cannot be simplified, each consisting of a simple variable equated to an expression consisting of a single operator applied to arguments, which are also simple variables (or literals). Converting a program to a set of atomic equations (another first for Luminous) vastly simplifies evaluation and static analysis. We discuss shortcomings of the current interpreter, the most glaring of which is the lack of user defined stream transformations.

Web-Based Abulyatama Alumni Information System

Tracer study is a study of tracer traces of graduates or alumni carried out to alumni after graduation. The obstacle faced by STMIK Abulyatama in conducting a tracer study is at the time of data collection, currently collecting data manually by distributing questionnaires and telephone interviews with alumni and companies. Therefore, it is necessary to develop a complete E-Tracer study service, which can provide information for higher education institutions for the benefit of curriculum improvements, trace alumni in detail and determine user satisfaction of college graduates. The research objectives are; (1) To find out the alumni information system using a web-based programming language, (2) Designing an alumni management information system using Data Flow Diagrams (DAD) and programming languages ​​such as HTML, CSS, JQuery, JSON, PHP and MySQL as databases, and (3) To improve the existing system to be even better, by building an information system that can improve performance on alumni data collection at Abulyatama University. The stages in the research consist of; literature study, data collection, systems analysis, system development, and report generation. Based on the results of the alumni information system analysis, it can be seen that until now it is still done manually. The author can conclude several things, namely; (1) With a Web-Based Alumni Information System, it is hoped that it will make work easier and accurate and more time-efficient information can be obtained, (2) The use of a manual system has limitations in the data search process, because the data is still stored in archive form, ( 3) Difficulty in updating the latest data for publication to the public, and (4) With this application, it is expected to simplify and speed up the administrative process at Abulyatama University.

Support tools for functional programming distance learning and teaching

The cloud parallel programming system CPPS being under development at the A.P. Ershov Institute of Informatics Systems SB RAS is based on the Cloud Sisal language which carries on the traditions of previous versions of the Sisal language while remaining a functional data-flow language focused on writing large scientific programs and expanding their capabilities by supporting cloud computing. In this paper, the online environment of the CPPS system which allows a user on any device with Internet access to develop and execute functional programs in the Cloud Sisal language is considered.

Handling Iterations in Distributed Dataflow Systems

Over the past decade, distributed dataflow systems (DDS) have become a standard technology. In these systems, users write programs in restricted dataflow programming models, such as MapReduce, which enable them to scale out program execution to a shared-nothing cluster of machines. Yet, there is no established consensus that prescribes how to extend these programming models to support iterative algorithms. In this survey, we review the research literature and identify how DDS handle control flow, such as iteration, from both the programming model and execution level perspectives. This survey will be of interest for both users and designers of DDS.

Verified Lustre Normalization with Node Subsampling

Dataflow languages allow the specification of reactive systems by mutually recursive stream equations, functions, and boolean activation conditions called clocks. Lustre and Scade are dataflow languages for programming embedded systems. Dataflow programs are compiled by a succession of passes. This article focuses on the normalization pass which rewrites programs into the simpler form required for code generation. Vélus is a compiler from a normalized form of Lustre to CompCert’s Clight language. Its specification in the Coq interactive theorem prover includes an end-to-end correctness proof that the values prescribed by the dataflow semantics of source programs are produced by executions of generated assembly code. We describe how to extend Vélus with a normalization pass and to allow subsampled node inputs and outputs. We propose semantic definitions for the unrestricted language, divide normalization into three steps to facilitate proofs, adapt the clock type system to handle richer node definitions, and extend the end-to-end correctness theorem to incorporate the new features. The proofs require reasoning about the relation between static clock annotations and the presence and absence of values in the dynamic semantics. The generalization of node inputs requires adding a compiler pass to ensure the initialization of variables passed in function calls.

DFlow: A Data Flow Analysis Tool for C/C++

AbstractAbstract syntax trees (ASTs), control flow graphs (CFGs), and data flow analysis (DFA) are prerequisites for static and dynamic analysis and vulnerability detection for programs; thus, obtaining them is significant. Recently, many tools related to generating ASTs, CFGs, and DFA have been proposed. However, most tools can only construct ASTs, very few can construct ASTs and CFGs, and almost none can construct all three. The vast majority of AST, CFG, and DFA tools are for other languages (e.g., Java and Python), and while a few are for C/C++, they are implemented in other languages, creating complex working environments, and overreliance on other language‐related libraries. To address these shortcomings, we present a DFA tool, DFlow, for C/C++. First, a lexical/grammatical analyzer generated by Flex and Bison is used to analyze the program. Second, an AST is constructed from the results; then, a CFG is obtained from the analysis results and the information from the AST. Finally, based on the AST and CFG, DFA is performed, and the vulnerabilities of simple programs are determined. We test some common vulnerable code and common weakness enumeration slicing code, which show the effectiveness of DFlow in program data flow analysis and vulnerability checking. The results show that our tool can implement ASTs, CFGs, and DFA, and we add some rules to the tool for vulnerability detection. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

The System for Transforming the Code of Dataflow Programs into Imperative

Functional dataflow programming languages are designed to create parallel portable programs. The source code of such programs is translated into a set of graphs that reflect information and control dependencies. The main way of their execution is interpretation, which does not allow to perform calculations efficiently on real parallel computing systems and leads to poor performance. To run programs directly on existing computing systems, you need to use specific optimization and transformation methods that take into account the features of both the programming language and the architecture of the system. Currently, the most common is the Von Neumann architecture, however, parallel programming for it in most cases is carried out using imperative languages with a static type system. For different architectures of parallel computing systems, there are various approaches to writing parallel programs. The transformation of dataflow parallel programs into imperative programs allows to form a framework of imperative code fragments that directly display sequential calculations. In the future, this framework can be adapted to a specific parallel architecture. The paper considers an approach to performing this type of transformation, which consists in allocating fragments of dataflow parallel programs as templates, which are subsequently replaced by equivalent fragments of imperative languages. The proposed transformation methods allow generating program code, to which various optimizing transformations can be applied in the future, including parallelization taking into account the target architecture.

Stochastic gradient descent‐based support vector machines training optimization on Big Data and HPC frameworks

SummarySupport vector machines (SVM) is a widely used machine learning algorithm. With the increasing amount of research data nowadays, understanding how to do efficient training is more important than ever. This article discusses the performance optimizations and benchmarks related to providing high‐performance support for SVM training. In this research, we have focused on a highly scalable gradient descent‐based approach to implementing the core SVM algorithm. In providing a scalable solution, we have designed optimized high‐performance computing and dataflow‐oriented SVM implementations. A high‐performance computing approach means the algorithm is implemented with the bulk synchronous parallel (BSP) model. In addition, we analyzed the language level optimizations and math kernel optimizations on a prominent HPC modeling programming language (C++) and dataflow modeling programming language (Java). In the experiments, we compared the performance of classic HPC models, classic dataflow models, and hybrid models designed on classic HPC and dataflow programming models. Our research illustrates a scientific approach in designing the SVM algorithm at scale in classic HPC, dataflow, and hybrid systems.

Multi-task Ada code generation from synchronous dataflow programs on multi-core: Approach and industrial study

• Proposing a new multi-task code generation method for the synchronous language SIGNAL. • Proposing a common multi-task structure, called Virtual Multi-Tasks (VMT), for different platform targets of our compiler. • The formal syntax and the operational semantics of VMT are mechanized in the proof assistant Coq. • Concurrent JobQueue is adopted for implementing the platform-dependent multi-task code to provide fine-grained parallelization. • A real-world aerospace industrial case study is used to show the feasibility of the method presented in the paper. The growing trend to use multi-core processors to get more performance is increasingly present in safety-critical systems. Synchronous dataflow programming is naturally well-suited to parallel execution, thanks to the fact that all data dependencies are always explicit. MiniSIGNAL is a multi-task code generation tool for the synchronous dataflow language SIGNAL. The existing MiniSIGNAL code generation strategies mainly consider coarse-grained parallelism based on Ada multi-task model. However, when we applied it to industrial case studies, this code generation scheme has revealed inefficient: architecture aspects of the target platform have to be taken into account to achieve fine-grained parallelism. To generate more efficient target code from industrial cases, this paper presents a new multi-task code generation method for MiniSIGNAL. Starting at the level of synchronous clocked guarded actions (S-CGA) which is an intermediate language for the compilation process of MiniSIGNAL, the transformation consists of two parts: at the platform-independent level, transforming the S-CGA representation to an abstract multi-task structure (called Virtual Multi-Tasks, VMT); at the platform-dependent level, adopting the thread pool pattern concurrent JobQueue to support fine-grained parallel Ada code generation from the VMT structure. Moreover, the formal syntax and the operational semantics of VMT are mechanized in the proof assistant Coq. Finally, the effectiveness of our approach is illustrated by an application of the real-world Guidance, Navigation and Control system.

PDKit: A data science toolkit for the digital assessment of Parkinson's Disease.

PDkit is an open source software toolkit supporting the collaborative development of novel methods of digital assessment for Parkinson's Disease, using symptom measurements captured continuously by wearables (passive monitoring) or by high-use-frequency smartphone apps (active monitoring). The goal of the toolkit is to help address the current lack of algorithmic and model transparency in this area by facilitating open sharing of standardised methods that allow the comparison of results across multiple centres and hardware variations. PDkit adopts the information-processing pipeline abstraction incorporating stages for data ingestion, quality of information augmentation, feature extraction, biomarker estimation and finally, scoring using standard clinical scales. Additionally, a dataflow programming framework is provided to support high performance computations. The practical use of PDkit is demonstrated in the context of the CUSSP clinical trial in the UK. The toolkit is implemented in the python programming language, the de facto standard for modern data science applications, and is widely available under the MIT license.

A Systematic Review of Binary Program Vulnerabilities Feature Extraction and Discovery Strategy Generation Methods

At present, the cyber security situation has turned increasingly serious across the globe. The cyberspace has become the fifth domain of various nations following the four domains of land, sea, sky and outer space. It is gradually obvious for the cyber attacks to collectivize and nationalize. Under this circumstance, as the first line of defense in maintaining cyber security defense, the control on the vulnerabilities is extremely significant. In consideration of the problem of low discovering efficiency in current vulnerability fuzzy testing discovery methods, this paper intends to explore the binary program vulnerability feature extraction and vulnerability discovery guiding strategy generation methods. Then, the extraction of the program control flow and data flow information and slicing operation are obtained according to program logic. Depending on a potential assessment model of binary program vulnerabilities based on the main hierarchy analysis method, the paper achieves a reasonable assessment of the potential of vulnerabilities in different slices of the program. Finally, in accordance with the evaluation results, a generation method for the vulnerability discovery path is proposed to improve the efficiency of binary program vulnerability discovery.

Transparent Synchronous Dataflow

Dataflow programming is a popular and convenient programming paradigm in systems modelling, optimisation, and machine learning. It has a number of advantages, for instance the lacks of control flow allows computation to be carried out in parallel as well as in distributed machines. More recently the idea of dataflow graphs has also been brought into the design of various deep learning frameworks. They facilitate an easy and efficient implementation of automatic differentiation, which is the heart of modern deep learning paradigm. [abstract abridged]

Probeware for the Modern Era: IoT Dataflow System Design for Secondary Classrooms

Sensor systems have the potential to make abstract science phenomena concrete for K-12 students. Internet of Things (IoT) sensor systems provide a variety of benefits for modern classrooms, creating the opportunity for global data production, orienting learners to the opportunities and drawbacks of distributed sensor and control systems, and reducing classroom hardware burden by allowing many students to “listen” to the same data stream. To date, few robust IoT classroom systems have emerged, partially due to lack of appropriate curriculum and student-accessible interfaces, and partially due to lack of classroom-compliant server technology. In this article, we present an architecture and sensor kit system that addresses issues of sensor ubiquity, acquisition clarity, data transparency, reliability, and security. The system has a dataflow programming interface to support both science practices and computational data practices, exposing the movement of data through programs and data files. The IoT Dataflow System supports authentic uses of computational tools for data production through this distributed cloud-based system, overcoming a variety of implementation challenges specific to making programs run for arbitrary duration on a variety of sensors. In practice, this system provides a number of unique yet unexplored educational opportunities. Early results show promise for Dataflow as a valuable learning technology from research conducted in a high school classroom.

КОМПЛЕКС СРЕДСТВ ТРАНСЛЯЦИИ ПРОГРАММ НА ЯЗЫКЕ C В ПРОГРАММЫ НА ЯЗЫКЕ ПОТОКА ДАННЫХ COLAMO

КОМПЛЕКС СРЕДСТВ ТРАНСЛЯЦИИ ПРОГРАММ НА ЯЗЫКЕ C В ПРОГРАММЫ НА ЯЗЫКЕ ПОТОКА ДАННЫХ COLAMO

Model Design of Construction Crane and Motion Regulation Using Hardware Control and Programming

In this paper, the methodology of development and regulation of the physical small model of Construction Crane is presented. The type of crane is Luffing Jib Crane. The model with its main parts is developed with 3D printing using moulded plastic materials. It is regulated for motion using microcontroller Arduino Nano, Motor drivers, and Servo Motors. Motion control is programmed with LabView Software. The motion processes that will be studied are lifting, lowering, and rotation of the Jib. The purpose is to create and analyse small and scaled model of Construction Crane that can be programmed and regulated to carry loads. This paper is a contribution to Crane’s design and regulation and it is a multidisciplinary engineering project from the fields of mechanical engineering design, materials selection, 3d printing, electronics hardware, systems control, data flow programming and software applications.

Methodologies for Synthesizing and Analyzing Dynamic Dataflow Programs in Heterogeneous Systems for Edge Computing

Methodologies for Synthesizing and Analyzing Dynamic Dataflow Programs in Heterogeneous Systems for Edge Computing

OmpSs@FPGA framework for high performance FPGA computing

This article presents the new features of the OmpSs@FPGA framework. OmpSs is a data-flow programming model that supports task nesting and dependencies to target asynchronous parallelism and heterogeneity. OmpSs@FPGA is the extension of the programming model addressed specifically to FPGAs. OmpSs environment is built on top of Mercurium source to source compiler and Nanos++ runtime system. To address FPGA specifics Mercurium compiler implements several FPGA related features as local variable caching, wide memory accesses or accelerator replication. In addition, part of the Nanos++ runtime has been ported to hardware. Driven by the compiler this new hardware runtime adds new features to FPGA codes, such as task creation and dependence management, providing both performance increases and ease of programming. To demonstrate these new capabilities, different high performance benchmarks have been evaluated over different FPGA platforms using the OmpSs programming model. The results demonstrate that programs that use the OmpSs programming model achieve very competitive performance with low to moderate porting effort compared to other FPGA implementations.

Оптимизация графов потока управления в промежуточных представлениях языка функционально-потокового параллельного программирования

Functional dataflow programming languages are intended for the development of architecture-independent parallel programs and support the control of computations based on data availability. Due to the fact that at present parallel computing systems are very widespread, and their programming in imperative languages is associated with portability problems, the development of architecturally independent parallel programming tools is an urgent task. When such a program is translated, intermediate representations are formed as the information graph and the corresponding control graph. During program execution, data readiness signals are transmitted along the arcs of the control graph. An explicit selection of the control graph allows us not only to change the computational control strategies and ensure the adaptation of the program to the architecture features, but also to apply specific methods for optimizing control dependencies. The paper proposes transformation methods that provide optimization of the control graph. When generating a control graph from an informational one, redundant arcs are introduced into it, the removal of which does not affect the result of the program, but leads to its more efficient execution. It is shown that in dataflow programs, in addition to control dependencies inherent in other programming languages, additional ones associated with the implementation features of deferred or conditional computations described by delayed lists arise. A formal description of redundant dependencies of various types is given, as well as an effective algorithm for their identification. The developed approach can be applied to such dataflow programming languages as PIFAGOR and Smile.

A Scalable Platform for Distributed Object Tracking Across a Many-Camera Network

Advances in deep neural networks (DNN) and computer vision (CV) algorithms have made it feasible to extract meaningful insights from large-scale deployments of urban cameras. Tracking an object of interest across the camera network in near real-time is a canonical problem. However, current tracking platforms have two key limitations: 1) They are monolithic, proprietary and lack the ability to rapidly incorporate sophisticated tracking models, and 2) They are less responsive to dynamism across wide-area computing resources that include edge, fog, and cloud abstractions. We address these gaps using Anveshak, a runtime platform for composing and coordinating distributed tracking applications. It provides a domain-specific dataflow programming model to intuitively compose a tracking application, supporting contemporary CV advances like query fusion and re-identification, and enabling dynamic scoping of the camera network's search space to avoid wasted computation. We also offer tunable batching and data-dropping strategies for dataflow blocks deployed on distributed resources to respond to network and compute variability. These balance the tracking accuracy, its real-time performance, and the active camera-set size. We illustrate the concise expressiveness of the programming model for four tracking applications. Our detailed experiments for a network of 1000 camera-feeds on modest resources exhibit the tunable scalability, performance, and quality trade-offs enabled by our dynamic tracking, batching, and dropping strategies.