Continuous Data Flow Research Articles

Adopting Cloud Computing in Aviation Industry for Flight Safety

Flight Data Recorder (FDR) an aircraft component mounted on board, registers and records all different operations and conditions of aircraft during flight. Video animation and reconstruction of the flight progress can then be visualized by investigators to show the last moments before the accident happened after data retrieval from FDR. There might be difficulties in some cases where it is hard to rescue FDR from accident scene, or if found, data cannot be retrieved due to exposure to extreme heat. Thus, an alternative source of data is highly crucial; therefore cloud computing technology linkage to aircraft systems in the Real Time for data network environment to maintain continuous flow of data and safety has been stressed upon. This research examined how concerned stakeholders in the global aviation industry can improve significantly on aviation safety by linking and saving the aircraft’s data communications to cloud computing on the ground to facilitate the tracking and monitoring of aircrafts in real time, in case the aircraft crashes and loses or suffers damages on its black box component.

Finding tendencies in streaming data using Big Data frequent itemset mining

The amount of information generated in social media channels or economical/business transactions exceeds the usual bounds of static databases and is in continuous growing. In this work, we propose a frequent itemset mining method using sliding windows capable of extracting tendencies from continuous data flows. For that aim, we develop this method using Big Data technologies, in particular, using the Spark Streaming framework enabling distributing the computation along several clusters and thus improving the algorithm speed. The experimentation carried out shows the capability of our proposal and its scalability when massive amounts of data coming from streams are taken into account.

Online bagging for recommender systems

AbstractEnsemble methods have been successfully used in the past to improve recommender systems; however, they have never been studied with incremental recommendation algorithms. Many online recommender systems deal with continuous, potentially fast, and unbounded flows of data—big data streams—and often need to be responsive to fresh user feedback, adjusting recommendations accordingly. This is clear in tasks such as social network feeds, news recommender systems, automatic playlist completion, and other similar applications. Batch ensemble approaches are not suitable to perform continuous learning, given the complexity of retraining new models on demand. In this paper, we adapt a general purpose online bagging algorithm for top‐N recommendation tasks and propose two novel online bagging methods specifically tailored for recommender systems. We evaluate the three approaches, using an incremental matrix factorization algorithm for top‐N recommendation with positive‐only user feedback data as the base model. Our results show that online bagging is able to improve accuracy up to 55% over the baseline, with manageable computational overhead.

Comparative Study of Different Cost Functions Between Neighbors for Optimizing Energy Dissipation in WSN

Lifetime of a battery is one of the major concerns in wireless sensor networks (WSNs). The traditional network algorithm chooses the data transmission path based on the distance between sender and receiver. For example, in directed diffusion, the nodes closer to the sink node are more active so that the continuous data flow of the network is maintained. However, in such a scenario, there is an energy hole problem because the nodes closer to the sink reduce their energy since they are more active. But if the residual energy and shortest hop counts can be taken together, then the problem can be overcome. We have analyzed different transmission cost functions with respect to approximated uniform energy dissipation directed diffusion algorithm. The final goal of the paper is to find out the suitable cost function to resist the early death of the first node of a network. We introduce dynamic priority variable, which is a tradeoff between total energy consumption and uniform energy dissipation.

Multi-label classification from high-speed data streams with adaptive model rules and random rules

Multi-label classification is a methodology that tries to solve classification problems where multiple classes are associated with each data example. Data streams pose new challenges to this methodology caused by the massive amounts of structured data production. In fact, most of the existent batch mode methods may not support this condition. Therefore, this paper proposes two multi-label classification methods based on rule and ensembles learning from continuous flow of data. These methods are derived from a multi-target regression algorithm. The main contribution of this work is the rule specialization for subsets of class labels, instead of the usual local (individual models for each output) or a global (one model for all outputs) methods. Prequential evaluation was conducted where global, local and subset operation modes were compared against other online classifiers found in the literature. Six real-world data sets were used. The evaluation demonstrated that the subset specialization presents competitive performance, when compared to local and global approaches and online classifiers found in the literature.

Real-Time Sentiment Analysis of Twitter Streaming data for Stock Prediction

In this study, an attempt has been made for making financial decisions such as stock market prediction, to predict the potential prices of a company’s stock and to serve the need of this, Twitter data 1 2 has been considered for scoring the impression that is carried for a particular firm. Streaming data proves to be a perennial source of data analysis collected in real-time. Streaming data basically deals with the continuous flow of data which carries information from sources like websites, mobile phone applications, server logs, social websites, trading floors, etc. The major characteristics of such data being its accessibility and availability, help in proper analysis and prediction of user behavior in a ceaseless manner. The classifying model made out of historical data can be relentlessly honed to give even more accurate results since its outcome is always compared to the next tick of the clock. Spark streaming has been considered for the processing of humongous data and data ingestion tools like Twitter API and Apache Flume have been further implemented for analysis.

ECONOMIC SECURITY AS AN ELEMENT OF STRATEGIC MANAGEMENT SYSTEM: ACCOUNTING AND ANALYTICAL ASPECT

The purpose of the paper is to implement the strategic management system at an enterprise, many information resources are required. Being at the crossroads of accounting, analysis, control, and coordination, the accounting and analytical support plays a special role in the enterprise’s strategic management – it connects, integrates, and coordinates all these functions, transferring the enterprise management to a qualitatively new level. Methodology. The accounting and analytical support serves as a self-regulation mechanism at the enterprise providing feedback in the management system. The effectiveness of strategic management is measured by the enterprise response to changes in the environment, by time limits of data flow. The data flow speed is determined by the maximum amount of information received for the shortest period of time and indicates the economic system efficiency, ability to respond flexibly, timely, and adequately to changes in the external and internal environment. The enterprise management effectiveness depends on the enterprise managers’ knowledge of the market, the conditions of its operation, the number of competitors, similar goods, development dynamics, etc. The purpose of the article is to justify expediency of the logistic approach applied to the organization of the enterprise’s accounting and analytical support, harmonization of principles of its development, determination of an element for analysis of the necessary information about the enterprise resources, competition, and changes in the internal and external environment. Results. To achieve this goal, an issue of an integrated approach to the creation of the accounting and analytical support as a key tool for the formation of a data flow in the enterprise’s strategic development system was considered. Theoretical approaches to the definition of “accounting and analytical support” are generalized and the main directions of the accounting and analytical support development, which can be used at modern enterprises, are described herein. The accounting and analytical support is proved to be a continuous data flow aimed at a single goal – to provide complete, reliable, relevant information on all levels of management in different activities. Specific characteristics of data flows (purposefulness, speed, discreteness, intensity, capacity, multifunctionality) are proved to increase the accounting and analytical support efficiency. Practical implications. All flow processes at an enterprise form an information system, consisting of: 1) information infrastructure (information and technology); 2) information support (means of information collection, processing, storage and transmission – information and communication resources). The data flow is believed to be one of the elements in the information space, which is considered from different aspects. For this purpose, the accounting and analytical support was offered to form on the principles of the logistic approach. Elements of the accounting and analytical support system are proposed to harmonize on modern principles, grouped in three directions: from general scientific principles of development strategy formation to development of requirements for the information support quality. Also, a key component is revealed to be required to coordinate and assess the entire system efficiency. Value/originality. The economic security system was proposed as this component, and the directions for the accounting and analytical support for economic security under strategic development were explored. The sequence of information formation and control measures in the economic security system is determined herein.

Advancing beyond the system: telemedicine nurses\u2019 clinical reasoning using a computerised decision support system for patients with COPD \u2013 an ethnographic study

BackgroundTelemedicine is changing traditional nursing care, and entails nurses performing advanced and complex care within a new clinical environment, and monitoring patients at a distance. Telemedicine practice requires complex disease management, advocating that the nurses’ reasoning and decision-making processes are supported. Computerised decision support systems are being used increasingly to assist reasoning and decision-making in different situations. However, little research has focused on the clinical reasoning of nurses using a computerised decision support system in a telemedicine setting. Therefore, the objective of the study is to explore the process of telemedicine nurses’ clinical reasoning when using a computerised decision support system for the management of patients with chronic obstructive pulmonary disease. The factors influencing the reasoning and decision-making processes were investigated.MethodsIn this ethnographic study, a combination of data collection methods, including participatory observations, the think-aloud technique, and a focus group interview was employed. Collected data were analysed using qualitative content analysis.ResultsWhen telemedicine nurses used a computerised decision support system for the management of patients with complex, unstable chronic obstructive pulmonary disease, two categories emerged: “the process of telemedicine nurses’ reasoning to assess health change” and “the influence of the telemedicine setting on nurses’ reasoning and decision-making processes”. An overall theme, termed “advancing beyond the system”, represented the connection between the reasoning processes and the telemedicine work and setting, where being familiar with the patient functioned as a foundation for the nurses’ clinical reasoning process.ConclusionIn the telemedicine setting, when supported by a computerised decision support system, nurses’ reasoning was enabled by the continuous flow of digital clinical data, regular video-mediated contact and shared decision-making with the patient. These factors fostered an in-depth knowledge of the patients and acted as a foundation for the nurses’ reasoning process. Nurses’ reasoning frequently advanced beyond the computerised decision support system recommendations. Future studies are warranted to develop more accurate algorithms, increase system maturity, and improve the integration of the digital clinical information with clinical experiences, to support telemedicine nurses’ reasoning process.

Sacbe: A building block approach for constructing efficient and flexible end-to-end cloud storage

Abstract End-to-end solutions enable users to protect their data, before sending them to the cloud, from confidentiality violations, service outages and vendor lock-in incidents. These solutions however require the integration and orchestration of multiple applications that affect the end-user service experience. This paper presents Sacbe, an approach for building efficient and flexible end-to-end cloud storage based on building blocks (BB), which are logical representations of independent applications encapsulated into containers. The developers can build structures such as pipelines, stacks and/or clusters of applications by chaining BBs through I/O interfaces. These structures enable users to move/process data/metadata in continuous dataflows from their devices to the cloud and enables organizations to build cloud storage services. We implemented a complete realization of an end-to-end cloud storage solution, which includes pipelines of BBs running on client-side for end-users to ensure in-house the confidentiality and reliability of data as well as stacks and clusters of BBs to build authentication, sharing, and storage services in a private cloud. This prototype was evaluated through controlled experimentation and a case study based on a satellite imagery, which revealed the feasibility of end-to-end solutions built with Sacbe as the end-user service experience was significantly improved in comparison with other solutions.

Data-driven decomposition of long-term echosounder time series from ocean observatories

Recent advances in technology have produced a deluge of acoustic data that offer opportunities to study ecological processes at scales that are not possible previously. A prominent example is the continuous data flow from the fleet of upward-looking echosounders installed by the Ocean Observatories Initiatives (OOI) at diverse global locations. However, it is unclear if conventional echosounder analysis routines are effective in analyzing these data sets, due to the generally scarce ground-truth resources and limited empirical knowledge at many locations. In this study, we explored the use of signal decomposition methods in discovering daily patterns of marine organism activities in long-term echosounder time series. Using non-negative matrix factorization (NMF), we show that the echograms can be decomposed into a weighted combination of discrete components, each with acoustic features that can be exploited for inferring the underlying biological assemblage. In addition, the component weights provide an a...

AdOn HDP-HMM: An Adaptive Online Model for Segmentation and Classification of Sequential Data.

Recent years have witnessed an increasing need for the automated classification of sequential data, such as activities of daily living, social media interactions, financial series, and others. With the continuous flow of new data, it is critical to classify the observations on-the-fly and without being limited by a predetermined number of classes. In addition, a model should be able to update its parameters in response to a possible evolution in the distributions of the classes. This compelling problem, however, does not seem to have been adequately addressed in the literature, since most studies focus on offline classification over predefined class sets. In this paper, we present a principled solution for this problem based on an adaptive online system leveraging Markov switching models and hierarchical Dirichlet process priors. This adaptive online approach is capable of classifying the sequential data over an unlimited number of classes while meeting the memory and delay constraints typical of streaming contexts. In this paper, we introduce an adaptive "learning rate" that is responsible for balancing the extent to which the model retains its previous parameters or adapts to new observations. Experimental results on stationary and evolving synthetic data and two video data sets, TUM Assistive Kitchen and collated Weizmann, show a remarkable performance in terms of segmentation and classification, particularly for sequences from evolutionary distributions and/or those containing previously unseen classes.

A new and updated resource for codon usage tables

BackgroundDue to the degeneracy of the genetic code, most amino acids can be encoded by multiple synonymous codons. Synonymous codons naturally occur with different frequencies in different organisms. The choice of codons may affect protein expression, structure, and function. Recombinant gene technologies commonly take advantage of the former effect by implementing a technique termed codon optimization, in which codons are replaced with synonymous ones in order to increase protein expression. This technique relies on the accurate knowledge of codon usage frequencies. Accurately quantifying codon usage bias for different organisms is useful not only for codon optimization, but also for evolutionary and translation studies: phylogenetic relations of organisms, and host-pathogen co-evolution relationships, may be explored through their codon usage similarities. Furthermore, codon usage has been shown to affect protein structure and function through interfering with translation kinetics, and cotranslational protein folding.ResultsDespite the obvious need for accurate codon usage tables, currently available resources are either limited in scope, encompassing only organisms from specific domains of life, or greatly outdated. Taking advantage of the exponential growth of GenBank and the creation of NCBI’s RefSeq database, we have developed a new database, the High-performance Integrated Virtual Environment-Codon Usage Tables (HIVE-CUTs), to present and analyse codon usage tables for every organism with publicly available sequencing data. Compared to existing databases, this new database is more comprehensive, addresses concerns that limited the accuracy of earlier databases, and provides several new functionalities, such as the ability to view and compare codon usage between individual organisms and across taxonomical clades, through graphical representation or through commonly used indices. In addition, it is being routinely updated to keep up with the continuous flow of new data in GenBank and RefSeq.ConclusionGiven the impact of codon usage bias on recombinant gene technologies, this database will facilitate effective development and review of recombinant drug products and will be instrumental in a wide area of biological research. The database is available at hive.biochemistry.gwu.edu/review/codon.

Redblock: a tool for online deduplication on large datasets

Online data deduplication aims to identify records that represent the same purpose on a continuous data flow environment. It must be able to process a range of information with high effectiveness and no delays. The purpose of this paper is to introduce a developed tool entitled Redblock, for real-time data deduplication, using a distributed platform for online processing combined with an Inverted Index. During the experimental evaluation, Redblock managed to provide good preliminary results in terms of efficiency and effectiveness in a database.

Dynamic data science and official statistics

AbstractMany of the challenges and opportunities of data science have to do with dynamic factors: a growing volume of administrative and commercial data on individuals and establishments, continuous flows of data and the capacity to analyze and summarize them in real time, and the necessity for resources to maintain them. With its emphasis on data quality and supportable results, the practice of Official Statistics faces a variety of statistical and data science issues. This article discusses the importance of population frames and their maintenance; the potential for use of multi‐frame methods and linkages; how the use of large scale non‐survey data may shape the objects of inference; the complexity of models for large data sets; the importance of recursive methods and regularization; and the benefits of sophisticated spatial visualization tools in capturing spatial variation and temporal change. The Canadian Journal of Statistics 46: 10–23; 2018 © 2017 Statistical Society of Canada

The Crowd in Requirements Engineering: The Landscape and Challenges

Crowd-based requirements engineering (CrowdRE) could significantly change RE. Performing RE activities such as elicitation with the crowd of stakeholders turns RE into a participatory effort, leads to more accurate requirements, and ultimately boosts software quality. Although any stakeholder in the crowd can contribute, CrowdRE emphasizes one stakeholder group whose role is often trivialized: users. CrowdRE empowers the management of requirements, such as their prioritization and segmentation, in a dynamic, evolved style through collecting and harnessing a continuous flow of user feedback and monitoring data on the usage context. To analyze the large amount of data obtained from the crowd, automated approaches are key. This article presents current research topics in CrowdRE; discusses the benefits, challenges, and lessons learned from projects and experiments; and assesses how to apply the methods and tools in industrial contexts. This article is part of a special issue on Crowdsourcing for Software Engineering.

Getting the Data Flowing: Lessons Learned from Existing Reporting Systems in the Forestry Sector in Indonesia for REDD+ MRV.

In the context of REDD+, Measurement, Reporting and Verification (MRV) is one way to manage forest change information. A national carbon and non-carbon database will be used in REDD+ to negotiate compensation schemes with the international community. Much of this data will be collected at the local level, thus a reporting system that can integrate these locally collected data into the national database is crucial. In this paper we compare and draw lessons from three existing local to national reporting systems that include the participation of local communities: 1) the government extension services, 2) the government owned forestry company, and 3) a private logging company in Indonesia, and provide recommendations for REDD+ reporting systems. The results suggest that the main desired conditions for effective data flow are: benefits to motivate local participation, based on contributions to reporting activities; simple data format and reporting procedures to allow local participation in the reporting process, and to support data aggregation at the national level; a facilitator to mediate data aggregation at the village level to ensure data consistency, completeness and accuracy; and a transparent and clear data flow. Under these conditions, continuous, accountable and consistent data flow from the local level will reach the national level where it can be fully utilized.

Driving Profiles Computation and Monitoring for Car Insurance CRM

Customer segmentation is one of the most traditional and valued tasks in customer relationship management (CRM). In this article, we explore the problem in the context of the car insurance industry, where the mobility behavior of customers plays a key role: Different mobility needs, driving habits, and skills imply also different requirements (level of coverage provided by the insurance) and risks (of accidents). In the present work, we describe a methodology to extract several indicators describing the driving profile of customers, and we provide a clustering-oriented instantiation of the segmentation problem based on such indicators. Then, we consider the availability of a continuous flow of fresh mobility data sent by the circulating vehicles, aiming at keeping our segments constantly up to date. We tackle a major scalability issue that emerges in this context when the number of customers is large—namely, the communication bottleneck—by proposing and implementing a sophisticated distributed monitoring solution that reduces communications between vehicles and company servers to the essential. We validate the framework on a large database of real mobility data coming from GPS devices on private cars. Finally, we analyze the privacy risks that the proposed approach might involve for the users, providing and evaluating a countermeasure based on data perturbation.

A Flexible Infrastructure for Dynamic Power Control of Electric Vehicle Battery Chargers

This paper proposes a flexible infrastructure for dynamic power control (FIDPC) of electric vehicle (EV) battery chargers. This infrastructure dynamically adjusts the EV battery charger current, according to the power demand of the home wherein the vehicle is plugged. An infrastructure was implemented to validate this proposal. This infrastructure is composed by an EV battery charger and a communication system based on a radio-frequency (RF) interface. The battery charger has nominal power of 3.6 kVA and operates with sinusoidal current and unitary total power factor, whereas the RF interface provides continuous data flow to the battery charger with information on the home total current consumption (root-mean-square value). Experimental tests were performed under realistic conditions to validate the concept behind the proposed FIDPC. These tests served to assess the behavior of the EV battery charger with dynamic power control on a single-phase 230-V 16-A 50-Hz residential electrical installation. The experimental results confirm the quick time response of the FIDPC, even when working under heavy-home-load variations.

A dynamic prime number based efficient security mechanism for big sensing data streams

Big data streaming has become an important paradigm for real-time processing of massive continuous data flows in large scale sensing networks. While dealing with big sensing data streams, a Data Stream Manager (DSM) must always verify the security (i.e. authenticity, integrity, and confidentiality) to ensure end-to-end security and maintain data quality. Existing technologies are not suitable, because real time introduces delay in data stream. In this paper, we propose a Dynamic Prime Number Based Security Verification (DPBSV) scheme for big data streams. Our scheme is based on a common shared key that updated dynamically by generating synchronized prime numbers. The common shared key updates at both ends, i.e., source sensing devices and DSM, without further communication after handshaking. Theoretical analyses and experimental results of our DPBSV scheme show that it can significantly improve the efficiency of verification process by reducing the time and utilizing a smaller buffer size in DSM.

Survey of Object-Based Data Reduction Techniques in Observational Astronomy

Abstract Dealing with astronomical observations represents one of the most challenging areas of big data analytics. Besides huge variety of data types, dynamics related to continuous data flow from multiple sources, handling enormous volumes of data is essential. This paper provides an overview of methods aimed at reducing both the number of features/attributes as well as data instances. It concentrates on data mining approaches not related to instruments and observation tools instead working on processed object-based data. The main goal of this article is to describe existing datasets on which algorithms are frequently tested, to characterize and classify available data reduction algorithms and identify promising solutions capable of addressing present and future challenges in astronomy.