Large Data Flow Research Articles

Methods and Models for Analyzing Methane Sorption Capacity of Coal Based on Its Physicochemical Characteristics

The authors study the influence of physicochemical parameters on methane adsorption capacity of coal and offer the analytical method for the methane adsorption capacity for three-phased condition of methane. It is found that in the depth interval to 300 m below surface, methane adsorption capacity measured in lab can exceed natural gas content of coal obtained from geological exploration data by 30%, and the change in the thermodynamic condition of coal–methane system brings irreversible physicochemical consequences in terms of the altered ratios of physical states of the main components. There is no linear connection between natural gas content of a coal bed and its methane adsorption capacity with respect to occurrence depth. The application of Big Data in treatment and interpretation of large data flows is described. The theoretical data predicted using the proposed method and the experimental data on methane content of Kuzbass coal agree.

Real Time Processing System of Massive Data Stream in Internet of Things Based on Apache Storm

The data of Internet of Things has the characteristics of real time and large scale. For the data, real-time processing is usually necessary, especially in some security scenarios, this paper tries to design a system which can support the real-time processing of large data flow. This paper firstly analyzes the characteristics of the data flow in the Internet of Things, and by using the results of the analysis, I puts forward the key technologies that can be used in the system and the main problems faced. Finally, I describe the architecture of the system I designed in detail from three aspects of data access, data processing and data storage.

Structural health monitoring: Closing the gap between research and industrial deployment

There has been a large volume of research on structural health monitoring since the 1970s but this research effort has yielded relatively few routine industrial applications. Structural health monitoring can include applications on very different structures with very different requirements; this article splits the subject into four broad categories: rotating machine condition monitoring, global monitoring of large structures (structural identification), large area monitoring where the area covered is part of a larger structure, and local monitoring. The capabilities and potential applications of techniques in each category are discussed. Condition monitoring of rotating machine components is very different to the other categories since it is not strictly concerned with structural health. However, it is often linked with structural health monitoring and is a relatively mature field with many routine applications, so useful lessons can be read across to mainstream structural health monitoring where there are many fewer industrial applications. Reasons for the slow transfer from research to practical application of structural health monitoring include lack of attention to the business case for monitoring, insufficient attention to how the large data flows will be handled and the lack of performance validation on real structures in industrial environments. These issues are discussed and ways forward proposed; it is concluded that given better focused research and development considering the key factors identified here, structural health monitoring has the potential to follow the path of rotating machine condition monitoring and become a widely deployed technology.

SAND: Smart and Adaptable Networking Design Using Virtual Slicing over Software-Defined Network

The importance of reliable and adaptable networks has become increasingly relevant with the escalation of connectivity in our lives. The growth of streaming of entertainment and development of always online software has created an environment of large data flows that need to be handled efficiently. Historically this problem has been solved with hardware-based load-balancers. These solutions often times are expensive and lack flexibility and scalability. With the use of Software-Defined Networking, a more dynamic solution can be created to meet network load balancing needs. We propose a Smart and adaptable network design seeking to utilize network resources more efficiently by identifying traffic patterns and analyzing network metric to dynamically build virtual slices. With this design, we were able to solve the aforementioned issues through minimizing packet loss, maximizing network link utilization, and efficiently reduce the load on the controller.

A Stepwise Spatio-Temporal Flow Clustering Method for Discovering Mobility Trends

Massive flows that represent the individual level of movements and communications can be easily obtained in the age of big data. Generalizing spatial and temporal flow patterns from such data is essential to demonstrate spatial connections and mobility trends. Clustering approaches provide effective methods to handle data sets that contain massive individual-level flows. However, existing flow clustering studies obscure the geometric properties of flow data, such as direction and length, which significantly indicate movement trends. In addition, temporal information is often ignored because previous approaches have mainly focused on the perspective of spatial clusters of flow data, resulting in a loss of temporal patterns. In this paper, we introduce new spatial and temporal similarity measurements between flows and propose a new clustering approach of flow data based on a stepwise strategy. This method can identify clusters from distinct flow distributions and discover significant spatio-temporal trends from large flow data. Simulated experiments with synthetic flows and a case study using Beijing taxi trip data are conducted to validate the usefulness of the proposed method.

Comprehensive approach for optimization traffic routing and using network resources in a virtual data center

Today, telecommunication networks are the basis for deploying different types of cloud applications and services in data centers. The growing use of the cloud computing concept to provide access to applications and services every year increases the amount of converged network traffic. Existing approaches for traffic routing which using in the software defined-network don’t provide enough flexible solutions, able to adapt to changes in traffic flows in real time. At peak load, this leads to an overabundance of traffic to specific physical network nodes which are not prepared to handle a large data flow. One of the approaches that make it possible to solve this problem is the organization of adaptive traffic routing. In this paper are describes the developed approaches for effective control of traffic flow in the virtual data center. Our approaches are using the methods of the data mining and machine learning to more accurately classify and identify traffic flows of cloud applications and network services. This information helps for optimization traffic routes in real time. Another element of the proposed complex solution is solving the task of identifying routes in the virtual network of a data center. Within the framework of the research, the problem of route identification has been decomposed into two consecutive subproblems. The first part is aimed at the classification of communication channels in a virtual data center for a number of features that are characteristic of all superimposed networks. At the second part, based on the received data, the routes are clustered in accordance with the QoS requirements imposed by the current traffic flows. The received data are combined with information about the current flows of the circulating network and are fed to the input of the neural network to decide on the choice of a suitable route. Thus, the solution of the task of adaptive routing allows determining the priorities when creating the bandwidth of traffic flows, as well as establishing rules for managing the network of the data center.

DEVELOPING METHODOLOGY FOR «SMART» SEARCH FOR POLITICAL PROCESS MARKERS IN SOCIAL MEDIA

The paper describes the methodological framework for political process studies based on social media markers. The empirical basis of the paper is a number of authors’ studies carried out using Predictor Mining, an interdisciplinary approach combining Data Mining and Social Computing. Predictor Mining is a computer-assisted technique which allows extracting data from unstructured chaotic large data flow for further analysis and prediction of political protests and their dynamics. A special attention is paid to the core principles of predictive mining related to the creation of protest markers of online user behavior in Russia and abroad. The article describes how Data Mining and Social Computing techniques help to evaluate the risks and possibilities to resist the technologies of mobilization of protest settings in the modern Russian and post-Soviet countries. The authors present the results of a comparative analysis of the specifics of mass political protest in Russia compared with Ukrainian and Moldavian cases. The following measurements related to the cases were made by the authors: analysis of active nihilistic and passive nihilistic data flows in the Ukrainian social media discourse (2014, computer-assisted classification of data flow); analysis of protest-related tag cloud in the Russian social media (2011—2017, discourse analysis); analysis of the use of social media as a tool for mobilizing political protest in Russia (2011—2017; cybermetrics); defining the structure of user roles and the density of social ties in the Russian and Ukrainian VKontakte protest communities (2017, building social graphs); analysis of social and political protest of collective actors in Russia (Manezh square, Kondopoga, Pugachev, Rostovon- Don, Biryulevo, Mineralnye Vody) (2006—2014, triangulation involving event analysis, discourse analysis and cybermetrics). Using a combination of different approaches to data acquisition and analysis such as survey techniques, content analysis, online and offline, qualitative and quantitative, traditional and Big Data techniques which are in fact a methodological experiment, the authors draw certain conclusions and propose recommendations related to the search for political protest markers in the web. As to the prospects of Predictor Mining, the authors consider that there is a need to elaborate a method that could encompass preliminary assessment, scenario planning and forecasting of regional and national social and political processes using the online traces.

Development of a numerical code to simulate the hydrodynamic energy potential, applied at Bou Ismail bay

The knowledge of wave energy propagation in the shallow water allow us to understand the functioning of marine ecosystems and to give answers about several costal phenomena. In the Algerian coast, the waves are omnipresent with an energy that is stripped under different physical forms. The classification of areas with high hydrodynamic energy potential requires a large data flow, concerning the physical hydrodynamic parameters produced by this energy. In this study we have developed a numerical code based on various mathematical theories and equations describing the physical state of the wave, including the formulas of MUNK, Coastal Engineering Research Center, 'CERC' and of the French Central Hydraulics Laboratory, 'LCHF', based on the output data of the SWAN model. This code allowed us to measure several wave’s physical parameters sufficiently well in order to understand and quantify the energies transmitted along the coast. A spatial analysis of the obtained results allowed us to classify seven coastal stations with an average wave energy exceeding 20 kW/m in the stormy days.

ANALYSIS OF QUALITY OF DEVELOPED SOFTWARE USED IN OIL PROCESSING AND PETROCHEMICAL INDUSTRY

The oil and gas industry is Russia is the leading industry in terms of cash inflows to the budget. Information technologies are actively being introduced into this sphere in order to minimize the cost of oil production. It should be noted that for any large oil refinery, not only quality output is important, but also the storage and transmission of large data flows. Progress does not stand still, science and technology are developing, new tasks are emerging. In this connection, it is necessary to create new software and improve the old one. Each software product has the right, inherent only to it, features and features that can provide certain advantages or disadvantages. In connection with the sharp advantage and a lot of modern software systems with the simultaneous fulfillment of the requirements imposed on their quality and safety. To ensure high efficiency and quality of software and software complexes, international standards are developed, developed with the participation of representatives of the leading companies in the software development industry. The key point of the article is checking the developed software tools for semantic errors, the description of programs in accordance with GOST 28806-90 Quality of software. Terms and Definitions. The importance of timely software verification on various errors, as they can cause significant damage, far exceeding the positive effect of their use. The article considers the importance of the correct use of IT terms and expressions. Static and dynamic analyzes of the software source code are considered. The advantages and disadvantages for each of the types of analysis that are presented in the table are considered. The advantages and disadvantages for each of the types of analysis that are presented in the table are considered. A static analysis of two software products was performed using the PVS-Studio analyzer. Based on the analysis, the relevant conclusions are drawn.

An energy-efficient virtual machine placement and route scheduling scheme in data center networks

The increasing requirements of big data analytics and complex scientific computing impose significant burdens on cloud data centers. As a result, not only the computation but also the communication expenses in data centers are greatly increased. Previous work on green computing in data centers mainly focused on the energy consumption of the servers rather than the communication. However, for those emerging applications with big data-flows transmission, more energy consumption could be consumed by communication links, switching and aggregation elements. To this end, based on data-flows’ transmission characteristics, we proposes a novel Job-Aware Virtual Machine Placement and Route Scheduling (JAVPRS) scheme to reduce the energy consumption of data center networks (DCN) while still meeting as many network QoS (Quality of Service) requirements as possible. Our proposed scheme focuses on not just migrating large data flows, but also integrating small data flows to improve the utilization rate of the communication links. With more idle switches turned off, DCN’s energy consumption will thus be reduced. Besides the data flows’ migration and integration, the Traffic Engineering (TE) technique is also applied to decrease the transmission delay and increase the network throughput. To evaluate the performance of our proposed scheme, a number of simulation studies are performed. Compared to the selected benchmarks, the simulation results showed that JAVPRS can achieve 22.28%–35.72% energy saving while reducing communication delay by 5.8%–6.8% and improving network throughput by 13.3%.

Large areas surface deformation analysis through a cloud computing P-SBAS approach for massive processing of DInSAR time series

We present in this work a methodology for computing surface deformation time series and mean velocity maps of large areas. Our approach relies on the availability of a multi-temporal set of synthetic aperture radar (SAR) data collected from ascending and descending orbits over an area of interest, and also permits us to estimate the vertical and horizontal (East-West) components of the Earth's surface deformation. The adopted methodology is based on an advanced cloud computing implementation of the differential SAR interferometry (DInSAR) Parallel Small Baseline Subset (P-SBAS) processing chain which allows the unsupervised processing of large SAR data volumes, from the raw data (level-0) imagery up to the generation of the corresponding DInSAR time series and maps. The solution presented, which is highly scalable, has been tested on ascending and descending ENVISAT SAR archives comprising approximately 400GB of data, which have been acquired over a large area of southern California (US) that extends over about 90,000km2. Such an input dataset has been processed in parallel by exploiting 280 computing nodes of the Amazon Web Services Cloud environment. The overall processing lasted about 8h and cost approximately $1900 USD. Moreover, to produce the final mean deformation velocity maps of the vertical and horizontal (East-West) displacement components of the whole investigated area, we also took advantage of the information available from external GPS measurements that permit us to account for possible regional trends not easily detectable by DInSAR and to refer the P-SBAS measurements to an external geodetic datum.The results presented clearly demonstrate the effectiveness of the proposed approach that paves the way to the extensive use of the available ERS-1/2 and ENVISAT SAR data archives. Furthermore, the proposed methodology can be particularly suitable to deal with the very large data flow provided by the Sentinel-1 constellation, thus permitting the extension of the DInSAR analyses at a nearly global scale.

Leveraging big data in population health management

Population health management takes into account many determinants of health, including medical care, social and physical environments and related services, genetics, and individual behavior. Many different types of data may be used to guide population health management programs and to estimate program value. In addition to the variety of data required for these programs, big population health program data are characterized by large volume, high velocity, and inconsistent data flows. This manuscript describes how big data analytics have been used to craft a population health program to help improve the lives of about four million older adults who have an AARP® Medicare Supplement Insurance plan (i.e., a Medigap plan). Plan enrollees have access to a wellness program, holistic care coordination programs, two telephone-based advice lines, concierge support for insurance and medical care needs, and a program designed to help reduce unnecessary emergency room (ER) visits. During 2009–2011, these program components led to several improvements in health care. For example, increased duration in care coordination was associated with fewer hospital readmissions, and participants were significantly more likely to have recurring physician office visits and recommended laboratory tests. Participants in ER decision support reduced their ER visits by 1299 visits per 1000 insureds, compared with a reduction of 1121 visits per 1000 insureds for individuals who did not participate in the program. Better depression management helped reduce depression symptoms in 59 % of participants engaged in that program. Big data analytics of member data suggested the need for a wellness program feature, which began in 2014. Analytics of disease management services offered in 2009–2011 helped to combine and refocus these program features to enhance effectiveness in later years. Using big data to help manage and evaluate a population health program has led to several improvements in health care. Program management, reporting, and evaluation processes generated additional data which, when analyzed, continues to refine program implementation and quality. Future improvements to this program may include enhanced integration of social service programs that will generate their own data streams for analyses designed to further improve health and wellbeing.

The Tunisian Mediterranean coastline: potential threats from urban discharges Sfax-Tunisian Mediterranean coasts

There have been growing concerns over the alarming threats to marine and terrestrial ecosystems from the improper discharge of effluents into water bodies worldwide, including Tunisia. Despite the large flow of data on the environmental pollution of various sites, little data is currently available on the physicochemical, geochemical conditions of the seawater of the Gulf of Gabes (South East Tunisia). Accordingly, the present study was undertaken to investigate and evaluate the physicochemical, biochemical and geochemical properties and attributes of the water and sediment samples from the coastal dumping sites of the Boudriere PK4, Ouady Ezzit, Ouady Maou, Sidi Salem, Sidi Mansour, Thyna and Sfax harbour. The results revealed that the marine environment in the title sites was alarming, suffering from heavily loaded pollutants from various sources that are predominantly discharged without preliminary or inadequate treatment. The total suspended matter (TSM) values recorded in the waters along the coastal sites were higher than the standard norms. In fact, more than 4350 mg/l of TSM were recorded at the coast, and the biochemical oxygen demand (BOD5) and chemical oxygen demand (COD) values exceeded the standard limits, reaching 300 mg/L and 1050 mg/L, respectively. The total oxidizable matter, representing BOD5 and COD, also exceeded 1360 mg/L. Low dissolved oxygen (DO) concentrations fluctuating between 0 and 2.95 mg/L were observed, except for some oueds. This reflected high levels of DO depletion and a tendency towards anoxic conditions. High hydrocarbon loads were also observed. Heavy metal pollution was equally alarming. Overall, the results indicated that the serious organic and mineral contamination of the Tunisian Mediterranean coastline calls for urgent action from state authorities and the public to remediate polluted sites and better control non-complying effluent discharge activities.

The data acquisition system for a fixed target experiment at NICA complex at JINR and its connection to the ATLAS TileCal readout electronics

Today's large-scale science projects have always encountered challenges in processing large data flow from the experiments, the ATLAS detector records proton-proton collisions provided by the Large Hadron Collider (LHC) at CERN every 50 ns which results in a total data flow of 10 Pb/s. These data must be reduced to the science data product for further analysis, thus a very fast decisions need to be executed, to modify this large amounts of data at high rates. The capabilities required to support this scale of data movement is development and improvement of high-throughput electronics. The upgraded LHC will provide collisions at rates that will be at least 10 times higher than those of today due to it's luminosity by 2022. This will require a complete redesign of the read-out electronics and Processing Units (PU) in the Tile-calorimeter (TileCal) of the ATLAS experiment. A general purpose, high-throughput PU has been developed for the TileCal at CERN, by using several ARM-processors in cluster configuration. The PU is capable of handling large data throughput and apply advanced operations at high rates. This system has been proposed for the fixed target experiment at NICA complex to handle the first level processes and event building. The aim of this work is to have a look at the architecture of the data acquisition system (DAQ) of the fixed target experiment at the NICA complex at JINR, by compiling the data-flow requirements of all the subcomponents. Furthermore, the VME DAQ modules characteristics to control, triggering and data acquisition will be described in order to define the DAQ with maximum readout efficiency, no dead time and data selection and compression.

A practical approach for clustering large data flows of malicious URLs

Over the last couple of years there has been a substantial increase of malicious attacks that are using the Internet as an infection vector. One solution to counter this problem is to implement a filter at the network connection level. Due to the large amount of data that has to be filtered in real-time, any practical approach has to consider both memory usage and performance limitations in order to deliver a fast response time. This paper presents a cloud-based mechanism that can be used to filter large amounts of network traffic with respect to both memory and response time limitations. The algorithms have been tested on data flows of more than 750 million of URLs/day. We will address different practical problems, such as storage, computation time and large data flow clustering. In the end we will also present different statistical results that we obtained over a period of 2 months.

Deep Packet Inspection Based on Many-Core Platform

With the development of computer technology, network bandwidth and network traffic continue to increase. Considering the large data flow, it is imperative to perform inspection effectively on network packets. In order to find a solution of deep packet inspection which can appropriate to the current network environment, this paper built a deep packet inspection system based on many-core platform, and in this way, verified the feasibility to implement a deep packet inspection system under many-core platform with both high performance and low consumption. After testing and analysis of the system performance, it has been found that the deep packet inspection based on many-core platform TILE_Gx36 [1] [2] can process network traffic of which the bandwidth reaches up to 4 Gbps. To a certain extent, the performance has improved compared to most deep packet inspection system based on X86 platform at present.

Modern approaches to processing large hyperspectral and multispectral aerospace data flows

We consider approaches to processing large hyperspectral and multispectral imaging flows produced in aerospace monitoring for solving a wide range of problems of management of natural resources, environmental security, prevention of natural disasters and technogenic accidents, as well as problems of real economy, and basic and applied sciences. We analyze the specific features of the phases of hyperspectral data analysis and describe a software and hardware system that uses new and improved methods and algorithms for processing large flows of hyperspectral and other aerospace data and has a high-performance computer. This system contains different types of software for identifying the types of given objects by solving inverse problems of remote sensing as well as by analyzing their qualitative and quantitative characteristics, combined multiparameter processing of hyperspectral aerospace data, tracking the local changes including those related to changes in meteorological conditions and vegetation periods, detecting and identifying the types of small objects on the basis of analysis of individual parts of the image, detecting and identifying heat sources, etc. We bring examples of processing of hyperspectral and multispectral satellite images with the help of software and hardware tools developed.

Dataflow Graph Partitioning for Area-Efficient High-Level Synthesis with Systems Perspective

Area efficiency in datapath synthesis is a widely accepted goal of high-level synthesis. Applications represented by their dataflow graphs are synthesized using resource sharing principles to reduce the area. However, existing resource sharing algorithms focus on absolute area reduction and maximal resource sharing. This kind of a design approach leads to constraints on how often, in terms of number of clock cycles, a new set of input data can be fed to an application. It also leads to very large multiplexers in case of very big dataflow graphs with hundreds of nodes. An adaptive dataflow graph partitioning algorithm is proposed that partitions a graph taking into account a user-defined constraint on how often a new set of input data (generally referred to as data initiation interval) is available. At the same time, a resource sharing algorithm is applied to such partitions in order to reduce area. Multiple design points are generated for a given dataflow graph with different area and time measures to enable a designer to make decisions. We demonstrate our graph partitioning algorithm using synthetically generated large dataflow graphs and on some benchmark applications.

OpenCyto: an open source infrastructure for scalable, robust, reproducible, and automated, end-to-end flow cytometry data analysis.

Flow cytometry is used increasingly in clinical research for cancer, immunology and vaccines. Technological advances in cytometry instrumentation are increasing the size and dimensionality of data sets, posing a challenge for traditional data management and analysis. Automated analysis methods, despite a general consensus of their importance to the future of the field, have been slow to gain widespread adoption. Here we present OpenCyto, a new BioConductor infrastructure and data analysis framework designed to lower the barrier of entry to automated flow data analysis algorithms by addressing key areas that we believe have held back wider adoption of automated approaches. OpenCyto supports end-to-end data analysis that is robust and reproducible while generating results that are easy to interpret. We have improved the existing, widely used core BioConductor flow cytometry infrastructure by allowing analysis to scale in a memory efficient manner to the large flow data sets that arise in clinical trials, and integrating domain-specific knowledge as part of the pipeline through the hierarchical relationships among cell populations. Pipelines are defined through a text-based csv file, limiting the need to write data-specific code, and are data agnostic to simplify repetitive analysis for core facilities. We demonstrate how to analyze two large cytometry data sets: an intracellular cytokine staining (ICS) data set from a published HIV vaccine trial focused on detecting rare, antigen-specific T-cell populations, where we identify a new subset of CD8 T-cells with a vaccine-regimen specific response that could not be identified through manual analysis, and a CyTOF T-cell phenotyping data set where a large staining panel and many cell populations are a challenge for traditional analysis. The substantial improvements to the core BioConductor flow cytometry packages give OpenCyto the potential for wide adoption. It can rapidly leverage new developments in computational cytometry and facilitate reproducible analysis in a unified environment.

Mapping Large Spatial Flow Data with Hierarchical Clustering

AbstractIt is challenging to map large spatial flow data due to the problem of occlusion and cluttered display, where hundreds of thousands of flows overlap and intersect each other. Existing flow mapping approaches often aggregate flows using predetermined high‐level geographic units (e.g. states) or bundling partial flow lines that are close in space, both of which cause a significant loss or distortion of information and may miss major patterns. In this research, we developed a flow clustering method that extracts clusters of similar flows to avoid the cluttering problem, reveal abstracted flow patterns, and meanwhile preserves data resolution as much as possible. Specifically, our method extends the traditional hierarchical clustering method to aggregate and map large flow data. The new method considers both origins and destinations in determining the similarity of two flows, which ensures that a flow cluster represents flows from similar origins to similar destinations and thus minimizes information loss during aggregation. With the spatial index and search algorithm, the new method is scalable to large flow data sets. As a hierarchical method, it generalizes flows to different hierarchical levels and has the potential to support multi‐resolution flow mapping. Different distance definitions can be incorporated to adapt to uneven spatial distribution of flows and detect flow clusters of different densities. To assess the quality and fidelity of flow clusters and flow maps, we carry out a case study to analyze a data set of 243,850 taxi trips within an urban area.