Large Data Streams Research Articles

Data Integrity Problems in High-Volume High-Velocity Data Ingestion

In the era of bigdata, and never ending data push from IoT devices, the IT infrastructure are built to be scalable to handle the huge batch loads or continuous streaming live data. However, the big question is how can be establish the data integrity. How can we make sure no data is lost from Origin till the destination passing through numerous touch points enroute ? How can we ensure the quality with continuous inflow ? Should the inflow be suspended to perform the DQ checks? Or should it be a totally independent parallel activity. Let’s explore. Key words: Quality Data Management, Data Pipelines, AutoScalling, Large Scale Streaming Data, Performance, IoT, Data Cleansing

Time dimension feature extraction and classification of high-dimensional large data streams based on unsupervised learning

In order to solve the problem of low accuracy of time dimension feature extraction and classification of high-dimensional large data streams, this paper proposes a time dimension feature extraction and classification algorithm of high-dimensional large data streams based on unsupervised learning. Analyze the trend of high-dimensional data flow changes under machine learning, and achieve dimensionality reduction of high-dimensional large traffic time dimensional data through local save projection. Analyze the spatial relationship between feature attributes and feature space, segment and fit high-dimensional big data streams and time dimensional feature data streams, further segment time dimensional sequences using sliding windows, and complete feature extraction through discrete dyadic wavelet transform. According to the clustering algorithm, cluster the time dimension feature data stream, calculate the cosine similarity of the feature data, model the time dimension feature stream of training samples, use the feature classification function to minimize the classification loss, and use unsupervised learning to achieve the final classification task. The test results show that this method can improve the temporal feature extraction and classification accuracy streams.

Task allocation algorithm for distributed large data stream group computing in the era of digital intelligence

In the era of digital intelligence, this paper studies the task allocation algorithm of distributed large data stream group computing, and reasonably allocates the task of group computing to meet the needs of massive computing and analysis of distributed large data stream. According to the idea of swarm intelligence perception and crowdsourcing platform, the task allocation model of distributed large data stream group computing is constructed to realize the task allocation of group computing. A distributed large data stream group computing task model and a user model are constructed, user attributes are initialized by using the accuracy of the answers submitted by users, the possibility that users can participate in the group computing task is predicted by a logistic regression algorithm, so that user candidate sequences participating in the computing task can be obtained, and the accuracy of the user’s real topics and corresponding topics can be grasped by capturing the candidate users’ real topics and evaluating the accuracy algorithm. Select the users who meet the subject area, update the candidate user sequence, and filter the users again on the basis of fully considering the factors such as information gain, user integrity and cost, so as to get the final user sequence and complete the task allocation of group computing. Experiments show that this method can solve the problem of distributed large data flow group computing task allocation, achieve high accuracy, reduce the cost, and effectively improve the information gain.

LIQUID NEURAL NETWORKS: PRINCIPLE OF WORK AND AREAS OF APPLICATION

The article deals with the architecture of liquid neural networks (LNN) and their potential in modern technologies. Thanks to the constant development of algorithms and hardware, neural networks are becoming more and more powerful and efficient, which opens up new opportunities for their application. The authors describe the principle of operation of liquid neural networks, which includes the process of learning and inference, which allows effective use of the natural dynamics of the system to solve various tasks, including classification, prediction, and control. We note that the concept of LNNs arose as an attempt to overcome some of the limitations and problems faced by traditional neural networks. The study considers the basic concepts and principles of LNNs and their application potential in various fields, from robotics to medicine and industry. The study also determines the main advantages and disadvantages of LNNs compared to traditional models. It is possible to use them to process a large stream of data, such as video, audio, or sensory data from various sensor types, allowing robots to receive information about their environment and make decisions based on that data. In medical diagnostics and image processing, liquid neural networks can significantly contribute to the quality and efficiency of diagnostic procedures. LNNs can enable the implementation of automatic control systems that monitor and regulate parameters of production processes or adapt to changes in the environment and optimise parameters to achieve maximum productivity and product quality. The field of LNN lacks standards and is limited to using performance metrics. Establishing standards and objective metrics will allow researchers and engineers to understand and compare different LNN implementations. Although LNNs are relatively efficient in terms of power consumption, their implementation at the hardware level may require new technologies and architectures to optimise performance. As a result, the study outlines the prospects for the further development of this technology. Keywords: liquid neural networks, artificial intelligence, adaptive control, learning efficiency, application potential.

The mechanism for improving the management of the maintenance and repair process using neural network technology

The system of preventive maintenance for various types of equipment is considered. To identify the need for managerial influence on the processes of maintenance and repair of the product, the coefficient of variation of the maintenance (repair) cycle is proposed. This coefficient is formed by the values of operating time or work between stops for maintenance or repair. To determine its numerical value, it is recommended to use such concepts as "maintenance (repair) cycle" and "frequency of maintenance (repair)". The values of the coefficient characterizing the stability of the process of repair and maintenance of the product are determined. These values of the coefficient are proposed to be used to justify the need to analyze the causes of failures or, in other words, the instability of the repair and maintenance process. The main managerial influences on the process of maintenance and repair are determined, which can be taken as a result of the analysis of the causes of failures. These include: the decision to search for a technical service provider, expanding the competencies of employees, updating the material base. Modern industry 4.0 tools allow you to get the required level of quality when working with a large data stream. The use of neural network technology in various technical service management tasks is due to some advantages over classical methods of statistical processing of results. The experiments of various authors are presented in which it is shown that neural network technologies are able to interpret graphic information, justify its use in technical diagnostics at the stage of analyzing diagnostic information in the form of oscillograms, which allows the use of complex diagnostic equipment in the absence of experts. A neural network simulation was carried out to recognize defects in bearing assemblies. As a result, the created neural network recognized the type of defect in accordance with the matrices of input signals and the output matrix of types of defects, which makes it possible to develop a mechanism for improving technical service using neural networks as a tool for analyzing diagnostic information in the absence of experts in this field.

Aquatic Soundscape Recordings Reveal Diverse Vocalizations and Nocturnal Activity of an Endangered Frog.

AbstractAutonomous sensors provide opportunities to observe organisms across spatial and temporal scales that humans cannot directly observe. By processing large data streams from autonomous sensors with deep learning methods, researchers can make novel and important natural history discoveries. In this study, we combine automated acoustic monitoring with deep learning models to observe breeding-associated activity in the endangered Sierra Nevada yellow-legged frog (Rana sierrae), a behavior that current surveys do not measure. By deploying inexpensive hydrophones and developing a deep learning model to recognize breeding-associated vocalizations, we discover three undocumented R. sierrae vocalization types and find an unexpected temporal pattern of nocturnal breeding-associated vocal activity. This study exemplifies how the combination of autonomous sensor data and deep learning can shed new light on species' natural history, especially during times or in locations where human observation is limited or impossible.

An Optimal Random Forest Model for Enhancing Decision-Making in Improving Students’ Performance via Educational Data Mining

The need for measures to enhance students’ performance, especially in Science and Math as a panacea for development in this 4th industrial revolution has become a global call. The large stream and heterogeneous nature of educational data make it difficult to use traditional statistical methods for data analysis to support decisions. However, Educational data mining has been effective and efficient in addressing this issue but much focus of researchers has been on only students' academic records to determine performance. This study sought to propose a Random Forest model to predict and enhance students’ performance. The study adopted a 5-stage mining process to mine psycho-socio-economic demographics educational data with Mutual information, Chi-squared test, Featurewiz, RandomizedSearch CV, and GridSearch CV to optimize the model’s performance. The study’s outcome revealed the key factors affecting students’ performance and that the model was enhanced by a 10.4% increment in precision and f1-score and 9.1% recall value, 7.1secs (62%) improvement in execution time and 78.7% improvement in Root Mean Square Error. This outcome remains a contribution to guiding decision-making in the educational setting and a basis for further studies on model optimization.

The influence of current results in a event­oriented data collection system

The article discusses synchronous and asynchronous procedures in the implementation of the microservices management algorithm in an adaptive system for processing large data flows when collecting information on the main event-oriented approach in the implementation of the architecture of a software system that processes information in real-time. This approach is important when processing large volumes of data from heterogeneous information sources, especially when the task is to minimize the total processing time of large data streams and reduce the total number of calls to data sources. The proposed approach makes it possible, based on its adaptability, to manage the selection of the composition and the number of calls of microservices to the sources, by the events that occur during the collection of information. Thus, it is possible to determine the choice of sources based on the assessment of the effectiveness of obtaining relevant information from them. This is especially important when processing large data flows from heterogeneous information sources when the task is to minimize the total time of collection and processing of large data flows. In turn, this poses the task of minimizing the number of requests to information sources to obtain a sufficient number of data units that are relevant to the search query. The creation of effective big data processing systems requires constant development of approaches to the architecture of building software applications. The event-oriented microservice architecture of the system makes it possible to adapt the operation of the system to the loads on individual microservices and the efficiency of their work by forming and responding to relevant events based on the analysis of relevant events that occur during the collection and initial processing of the received data. Depending on the specific task, it is possible to use both a synchronous and an asynchronous microservice management algorithm. The article provides an analysis of the effectiveness of obtaining relevant data depending on the degree of consideration in the evaluation of results and the formation of events, both the depth of the previous history of the results of requests and the size of their quantitative impact. The use of event-oriented microservice architecture can be especially effective when developing various information and analytical systems that need, according to user requests, to access various sources of information, analyze their data for relevance according to the request, and process large data streams.

Business Management Model of Private and Public News Channels in Pakistan: A Case Study of GEO News and PTV News

Purpose: The chief purpose of this study was designed to explore the phenomenal changes of business management models of the private and public news channels in Pakistan. The rationale behind this objective was to explore changing business approaches as adopted by the private news channel like Geo News and the public news channel like PTV News. The advent of the social media has altogether changed the landscape for the traditional media including newspapers and news channels, therefore, it seemed quite appropriate to explore new strategies for enhancing business for news channels in Pakistan. 
 Design/Methodology/Approach: The study employed in-depth interviews qualitative method to collect qualitative data from different patterns of data for major themes. Qualitative Content Analysis was partially employed to minimize large stream of data into small chunk and analyze data accordingly.
 Findings: The study found out that the state-run news channel like PTV News was solely relying for financial support on the federal and provincial governments. Accordingly, the PTV News was providing absolute coverage to public departments and enhancing image of the ministers of the ruling political party and their ministries to enhance their public trust. However, the private news channel like GEO News has to make all-out efforts on advertising agencies, real estate sectors, multi-national companies, national and international corporations to meet its financial besides enhancing business of the Channel. Reciprocally, the private sector providing huge funding to GEO News has some sort of influence on its editorial content and news coverage thus compromising freedom of express to some extent. 
 Implications/Originality/Value: The study has drawn a visible line between financial expenditure and its impact on news coverage and editorial policy. PTV News was providing absolute coverage to public departments due to financial dependency, whereas the GEO News was providing coverage to all political parties and other public and private sector with slight difference. However, the freedom of expression of the GEO News was compromised to some extent because of financial dependency.

Opportunities and Challenges of Remote Sensing, Geospatial Data, and Machine Learning in Obtaining Accessibility and Location Information for Sustainable Development in Indonesia

With the advancement of technologies so does the data collection method which creates a large, rapid, and diverse stream of data. Statistic Indonesia (BPS) has also encouraged to utilize this by starting to collect geospatial information on respondents and public facilities. To keep up with this a change needs to be made in processing methods to accommodate massive, high-dimensional, and multiform data collected in different forms such as machine learning. This progression also opens up a new opportunity for tackling various statistical data problems such as accessibility and location data. Remote sensing is one of the big data sources that undergoes a lot of changes shown in the high spatial and temporal resolution satellite imagery availability, together with the BPS geotagging data shows great promise in classifying land use and geospatial analysis. Even so, there are still some challenges in remote sensing as well as other geospatial data utilization. The goals of this review paper are to study the opportunities and challenges in utilizing remote sensing, geospatial data, and machine learning for accessibility and location information. In this paper, we explore the possibilities and limitations in its implementation into SDGs indicators that involve accessibility and location such as indicators 9.1.1, 11.1.1, 11.2.1, 11.3.1, and 11.7.1 including other variables needed for the calculation like access to public facilities. Moreover, our experiment using geotagging data shows potential in improving proportion estimation when compared to using a simple ratio. Our DEGURBA following the UN definition using machine learning LULC for dasymetric mapping also provides more insight compared to the existing data. We can conclude that there are great opportunities in applying remote sensing and other geospatial data to monitor the accessibility and location to further sustainable development in Indonesia.

Hierarchical AI enables global interpretation of culture plates in the era of digital microbiology

Full Laboratory Automation is revolutionizing work habits in an increasing number of clinical microbiology facilities worldwide, generating huge streams of digital images for interpretation. Contextually, deep learning architectures are leading to paradigm shifts in the way computers can assist with difficult visual interpretation tasks in several domains. At the crossroads of these epochal trends, we present a system able to tackle a core task in clinical microbiology, namely the global interpretation of diagnostic bacterial culture plates, including presumptive pathogen identification. This is achieved by decomposing the problem into a hierarchy of complex subtasks and addressing them with a multi-network architecture we call DeepColony. Working on a large stream of clinical data and a complete set of 32 pathogens, the proposed system is capable of effectively assist plate interpretation with a surprising degree of accuracy in the widespread and demanding framework of Urinary Tract Infections. Moreover, thanks to the rich species-related generated information, DeepColony can be used for developing trustworthy clinical decision support services in laboratory automation ecosystems from local to global scale.

Online Machine Learning and Surrogate-Model-Based Optimization for Improved Production Processes Using a Cognitive Architecture

Cyber-Physical Systems (CPS) play an essential role in today’s production processes, leveraging Artificial Intelligence (AI) to enhance operations such as optimization, anomaly detection, and predictive maintenance. This article reviews a cognitive architecture for Artificial Intelligence, which has been developed to establish a standard framework for integrating AI solutions into existing production processes. Given that machines in these processes continuously generate large streams of data, Online Machine Learning (OML) is identified as a crucial extension to the existing architecture. To substantiate this claim, real-world experiments using a slitting machine are conducted, to compare the performance of OML to traditional Batch Machine Learning. The assessment of contemporary OML algorithms using a real production system is a fundamental innovation in this research. The evaluations clearly indicate that OML adds significant value to CPS, and it is strongly recommended as an extension of related architectures, such as the cognitive architecture for AI discussed in this article. Additionally, surrogate-model-based optimization is employed, to determine the optimal hyperparameter settings for the corresponding OML algorithms, aiming to achieve peak performance in their respective tasks.

Mining Discriminative Itemsets Over Data Streams Using Efficient Sliding Window

In this paper, we present an efficient novel method for mining discriminative itemsets over data streams using the sliding window model. Discriminative itemsets are the itemsets that are frequent in the target data stream, and their frequency in the target stream is much higher in comparison to their frequency in the rest of the streams. The problem of mining discriminative itemsets has more challenges than mining frequent itemsets, especially in the sliding window model, as during the window frame sliding, the algorithms have to deal with the combinatorial explosion of itemsets in more than one data stream, for the transactions coming in and going out of the sliding window. We propose a single scan algorithm using two novel in-memory data structures for mining discriminative itemsets in a combination of offline and online sliding windows. Offline processing is used for controlling the generation of many unpromising itemsets. Online processing is used for getting more up-to-date and accurate online answers between two offline slidings. The discovered discriminative itemsets are accurately updated in the offline sliding window periodically, and the mining process is continued in the online sliding between two periodic offline slidings. The extensive empirical analysis shows that the proposed algorithm provides efficient time and space complexities with full accuracy. The algorithm can handle large, fast-speed, and complex data streams.

РОЗРОБКА ІНТЕЛЕКТУАЛЬНОЇ СИСТЕМИ НАВІГАЦІЇ ТА ПРОСТОРОВОЇ ОРІЄНТАЦІЇ ДЛЯ ЛЮДЕЙ З ВАДАМИ ЗОРУ

A prototype of an information and communication navigation and spatial orientation system for people with visual impairments has been developed based on a mobile device. The prototype has several advantages, including portability, convenience, and ease of use. This is achieved through a harmonious combination of optimized software modules that provide data collection, navigation, intelligent analytics, and image processing within a single smartphone. To implement the prototype, stages of development of the information and communication system were previously conducted, subject area analysis was carried out, software and hardware options were researched, and an architecture was designed based on an algorithm for asynchronous processing of a large stream of data. The result of the work is a verified basic prototype of the system, ready for use in testing conditions and data collection for further research.

Multi‐Dimensional Characterization of Battery Materials

AbstractDemand for low carbon energy storage has highlighted the importance of imaging techniques for the characterization of electrode microstructures to determine key parameters associated with battery manufacture, operation, degradation, and failure both for next generation lithium and other novel battery systems. Here, recent progress and literature highlights from magnetic resonance, neutron, X‐ray, focused ion beam, scanning and transmission electron microscopy are summarized. Two major trends are identified: First, the use of multi‐modal microscopy in a correlative fashion, providing contrast modes spanning length‐ and time‐scales, and second, the application of machine learning to guide data collection and analysis, recognizing the role of these tools in evaluating large data streams from increasingly sophisticated imaging experiments.

Mapping floods from remote sensing data and quantifying the effects of surface obstruction by clouds and vegetation

Floods are one of the most devastating natural calamities affecting millions of people and causing damage all around the globe. Flood models and remote sensing imagery are often used to predict and understand flooding. An increasing number of earth observation satellites are producing data at a rate that far outpaces our ability to manually extract meaningful information from it, motivating a surge in research on automatic feature detection in satellite imagery using machine learning and deep learning algorithms to automate flood mapping so that information from large streams of data can be extracted in near-real time and used for disaster response at landscape scale. The development of such an algorithm is predicated on exposure to training datasets that are representative of the full range of diversity in the spatial and spectral signature of surface water as it is sampled by space-based instruments. To address these needs, we developed a semantically labeled dataset of high-resolution multispectral imagery (Maxar WorldView 2/3) strategically sampled to be representative of North American surface water variability along five spatiotemporal strata: latitude, topographic complexity, land use, and day of year. This dataset was utilized to train a convolutional neural network (CNN) to automatically detect inundation extents using the Deep Earth Learning, Tools, and Analysis (DELTA) framework, an open source TensorFlow/Keras interpreter for satellite imagery. Our research objective was to demonstrate the out-of-sample accuracy of our trained CNN at landscape scale. The model performed well, with 98% precision and 94% recall for the water class during validation. We then evaluated the accuracy of our satellite-derived flood maps from trained machine learning model against a hydraulic model. For this, we compared predicted inundation extents against the USGS Flood Inundation Mapping (FIM) Program's flood map library at 17 different locations, where the FIM library provides flood inundation extents based on hydraulic models built for river reaches and corresponding to stage measurements at a nearby USGS gaging site. Compared to the hydraulic model, we estimated the underprediction of flood inundation by optical remote sensing data in our areas of interest to be 62%. We used land use data from National Land Cover Database (NLCD) and cloud masks to estimate that 79% of underprediction was due to these obstructions, with 74% belonging to vegetation, 9% to clouds, and 4% to both. A significant amount of inundation is missed when only optical remote sensing data is considered, and we suggest the use of flood models along with remote sensing data for getting the most realistic flood inundation extents.

Freight Operational Characteristics Mined from Anonymous Mobile Sensor Data

Effective transportation performance measurement (TPM) benefits from ubiquitous transportation system monitoring both spatially and temporally. In the context of freight-oriented TPM, traditional devices such as inductive loops, cameras, manual counts, and so forth, may fail to provide comprehensive and high-resolution coverage, providing, for example, only volume counts for a small subset of links across a large network with no indication of trip linkages. New sources of big data from mobile sensors including on-board global positioning system (GPS) devices allow more comprehensive network coverage and insights into trip chaining behaviors. However, to gain actionable insights into system performance from large and noisy streams of mobile sensor data, it is necessary to mine it for relevant operational characteristics of the trucks it represents. Such characteristics include stop locations, stop duration, stop time of day, trip length, and trip duration. To address this methodological need, this paper presents three heuristic algorithms: “stop identification,”“path identification,” and “trip identification.” To address the issue of determining relevant operational characteristics, a multinomial logit (MNL) model approach is applied to determine the commodity carried based on the outputs of the heuristic algorithms. The MNL model is novel in that it relates operational characteristics to commodity carried thus filling a critical data gap that currently limits the development of advanced freight forecasting models. The set of models developed in this paper allow large-scale GPS data to be used for freight planning while maintaining levels of data anonymity that allow such data to be shared with public agencies.

Fault Tolerant Gamma-Minus Network

In large-scale supercomputers thousands of processors are connected together to their respective memory modules which are controlled by several control units connected in parallel. Large data streams have to be communicated between these processors and memories through interconnection networks. Multistage interconnection network (MIN) is an efficient way to provide these communications at a very reasonable cost. For such large systems MIN employed should be highly reliable and fast to meet the desired specifications of these high speed switch fabrics. In this paper two new architectures of MIN are proposed which are named as fault-tolerant Gamma-Minus (FTGM-1 & FTGM-2) networks. These proposed architectures are multipath MIN with totally disjoint paths and are highly reliable and fault tolerant. The routing algorithm proposed for these structures is simple distance tag routing with non-backtracking which overcomes rerouting overheads and hence improve communication delays. The proposed MIN ensures multiple fault tolerance at each stage including input and output stage with minimal horizontal distance between source node to destination node. For validating the results, a comparison has been presented between existing MINs with these proposed designs. The proposed MINs outperform the existing MIN in terms of reliability, fault tolerance and up to some extent cost as well.

Knowledge mining of unstructured information: application to cyber domain

Information on cyber-related crimes, incidents, and conflicts is abundantly available in numerous open online sources. However, processing large volumes and streams of data is a challenging task for the analysts and experts, and entails the need for newer methods and techniques. In this article we present and implement a novel knowledge graph and knowledge mining framework for extracting the relevant information from free-form text about incidents in the cyber domain. The computational framework includes a machine learning-based pipeline for generating graphs of organizations, countries, industries, products and attackers with a non-technical cyber-ontology. The extracted knowledge graph is utilized to estimate the incidence of cyberattacks within a given graph configuration. We use publicly available collections of real cyber-incident reports to test the efficacy of our methods. The knowledge extraction is found to be sufficiently accurate, and the graph-based threat estimation demonstrates a level of correlation with the actual records of attacks. In practical use, an analyst utilizing the presented framework can infer additional information from the current cyber-landscape in terms of the risk to various entities and its propagation between industries and countries.

Big Data Analytics: Deep Content-Based Prediction with Sampling Perspective

The world of information technology is more than ever being flooded with huge amounts of data, nearly 2.5 quintillion bytes every day. This large stream of data is called big data, and the amount is increasing each day. This research uses a technique called sampling, which selects a representative subset of the data points, manipulates and analyzes this subset to identify patterns and trends in the larger dataset being examined, and finally, creates models. Sampling uses a small proportion of the original data for analysis and model training, so that it is relatively faster while maintaining data integrity and achieving accurate results. Two deep neural networks, AlexNet and DenseNet, were used in this research to test two sampling techniques, namely sampling with replacement and reservoir sampling. The dataset used for this research was divided into three classes: acceptable, flagged as easy, and flagged as hard. The base models were trained with the whole dataset, whereas the other models were trained on 50% of the original dataset. There were four combinations of model and sampling technique. The F-measure for the AlexNet model was 0.807 while that for the DenseNet model was 0.808. Combination 1 was the AlexNet model and sampling with replacement, achieving an average F-measure of 0.8852. Combination 3 was the AlexNet model and reservoir sampling. It had an average F-measure of 0.8545. Combination 2 was the DenseNet model and sampling with replacement, achieving an average F-measure of 0.8017. Finally, combination 4 was the DenseNet model and reservoir sampling. It had an average F-measure of 0.8111. Overall, we conclude that both models trained on a sampled dataset gave equal or better results compared to the base models, which used the whole dataset.