Incoming Data Stream Research Articles

SkyMapper optical follow-up of gravitational wave triggers: Alert science data pipeline and LIGO/Virgo O3 run

Abstract We present an overview of the SkyMapper optical follow-up programme for gravitational-wave event triggers from the LIGO/Virgo observatories, which aims at identifying early GW170817-like kilonovae out to $\sim200\,\mathrm{Mpc}$ distance. We describe our robotic facility for rapid transient follow-up, which can target most of the sky at $\delta<+10\deg $ to a depth of $i_\mathrm{AB}\approx 20\,\mathrm{mag}$ . We have implemented a new software pipeline to receive LIGO/Virgo alerts, schedule observations and examine the incoming real-time data stream for transient candidates. We adopt a real-bogus classifier using ensemble-based machine learning techniques, attaining high completeness ( $\sim98\%$ ) and purity ( $\sim91\%$ ) over our whole magnitude range. Applying further filtering to remove common image artefacts and known sources of transients, such as asteroids and variable stars, reduces the number of candidates by a factor of more than 10. We demonstrate the system performance with data obtained for GW190425, a binary neutron star merger detected during the LIGO/Virgo O3 observing campaign. In time for the LIGO/Virgo O4 run, we will have deeper reference images allowing transient detection to $i_\mathrm{AB}\approx 21\,\mathrm{mag}$ .

A system for off-line validation of medical data in DICOM archive

The problem of consistency of medical data in Hospital Data Management Systems is considered in the context of correctness of medical images itself to minimize possible harm from spurious DICOM files. The approach should be considered as an addition to other securing techniques like watermarking, encrypting, testing conformance with the standard. To achieve amenable accuracy in practice two aspects are taken into account: correctness of periodicity and correctness of image data (time series) itself for considered modality. This paper proposes an architecture of an information system and network filter integrated to it to provide facilities for analysis and alert management. The architecture is designed to perform the analysis of incoming data streams within components working in a clustered manner, providing horizontal scalability and fault tolerance to be BigData-ready.

Differences in the perception of digital information of aviation operators depending on the degree of extraversion

Considers the influence of the degree of extraversion on the perception of digital information by aviation operators. The analysis of the results of solving the Schulte tables by the test subjects on the NS-Psychotest hardware complex based on the recording of the eye track, performed using a stationary eye tracking device RED250mobile eye tracking device was carried out. While performing the search function, the number of saccadic movements, their amplitude, and the search time for a given digital value were registered. It has been revealed, that introverts during realization eyes search function perform less saccadic movements and spend less time on it than extraverts do. Significant differences have been found during resolving search tasks depending on degree of extraversion. Some interconnections of extension the latent period of the saccade with complication of solving process the cognitive problem were also found. It was found out, that increase in speed of saccadic movements of eyes leads to low efficiency of results of search task execution. Results of conducted research emphasize value of individual approach to medical flight service taking into account psychological features of flight crew in conditions of rapid progress in aviation technologies and means of visualisation of flight information. The obtained data reveal some features of information perception by operators of complex ergatic systems, the study of which in the future will help to maintain the reserves of attention in a continuous stream of incoming data, and thereby reduce the load on the visual analyzer and increase the reliability of professional activities of flight crew.

Predictive Maintenance for Injection Molding Machines Enabled by Cognitive Analytics for Industry 4.0.

The exploitation of big volumes of data in Industry 4.0 and the increasing development of cognitive systems strongly facilitate the realm of predictive maintenance for real-time decisions and early fault detection in manufacturing and production. Cognitive factories of Industry 4.0 aim to be flexible, adaptive, and reliable, in order to derive an efficient production scheme, handle unforeseen conditions, predict failures, and aid the decision makers. The nature of the data streams available in industrial sites and the lack of annotated reference data or expert labels create the challenge to design augmented and combined data analytics solutions. This paper introduces a cognitive analytics, self- and autonomous-learned system bearing predictive maintenance solutions for Industry 4.0. A complete methodology for real-time anomaly detection on industrial data and its application on injection molding machines are presented in this study. Ensemble prediction models are implemented on the top of supervised and unsupervised learners and build a compound prediction model of historical data utilizing different algorithms’ outputs to a common consensus. The generated models are deployed on a real-time monitoring system, detecting faults in real-time incoming data streams. The key strength of the proposed system is the cognitive mechanism which encompasses a real-time self-retraining functionality based on a novel double-oriented evaluation objective, a data-driven and a model-based one. The presented application aims to support maintenance activities from injection molding machines’ operators and demonstrate the advances that can be offered by exploiting artificial intelligence capabilities in Industry 4.0.

Magnet-Free Nonreciprocity [Scanning the Section

Electromagnetic waves, from radio signals to light, generally obey a strict symmetry in transmission: if they can travel from point A to point B, they can also travel backward from B to A with the same properties. This general symmetry, rooted in the fact that Maxwell’s equations are symmetric with respect to time and known as reciprocity, has profound implications on a variety of technologies and device functionalities. On the one hand, we count on reciprocity when we operate an antenna, in a variety of measurement schemes, and in analytical/numerical techniques to study the response of electromagnetic systems. On the other hand, reciprocity inherently limits the degree of control in manipulating electromagnetic waves and hence the overall functionality of many devices. For instance, breaking reciprocity enables the possibility of transmitting a signal without having to worry about backreflections, which may harm sensitive equipment or interfere with incoming data streams, and of transmitting and receiving signals through the same transducer at the same frequency and at the same time—the holy grail of full-duplex communication schemes. Fields as diverse as radar technologies, wireless communications, and optical and quantum computing all require nonreciprocal signal routing to enable reception of signal echoes while transmitting, data transmission without worrying about jammers, and to guarantee one-way streams of data that can be robustly processed.

Framework for opinion as a service on review data of customer using semantics based analytics

At Opinion mining plays a significant role in representing the original and unbiased perception of the products/services. However, there are various challenges associated with performing an effective opinion mining in the present era of distributed computing system with dynamic behaviour of users. Existing approaches is more laborious towards extracting knowledge from the reviews of user which is further subjected to various rounds of operation with complex procedures. The proposed system addresses the problem by introducing a novel framework called as Opinion-as-a-Service which is meant for direct utilization of the extracted knowledge in most user friendly manner. The proposed system introduces a set of three sequential algorithm that performs aggregated of incoming stream of opinion data, performing indexing, followed by applying semantics for extracting knowledge. The study outcome shows that proposed system is better than existing system in mining performance.

Edge-Computing-Enabled Unmanned Module Defect Detection and Diagnosis System for Large-Scale Photovoltaic Plants

The power efficiency of photovoltaic (PV) modules is highly correlated with their health status. Under dynamically changing environments, PV defects could spontaneously form and develop into fatal faults during the daily operation of PV power plants. To facilitate defect detection with less human intervention, a nondestructive, contactless, and automatical visual inspection system with the help of unmanned aerial vehicles and edge computing is proposed in this article. During the processing of the incoming data stream, the system may collect some new, unknown, and unlabeled defects that have not been identified yet in the existing database. To distinguish them from the existing ones, a deep embedded restricted cluster algorithm is designed to identify the unknown and unlabeled PV module defects in an unsupervised manner. Limited by the resources of edge devices and the availability of images of PV defects for training, we developed an online solution combined with deep learning, data argumentation, and transfer learning to properly address the issues of running resource-hungry applications on edge devices and lack of training samples faced by the deep learning approaches used in the field. In addition, pointwise convolution layers are introduced into the network to reduce the parameters and the size of the model. With the reduction of the network depth of the deep convolutional neural network model and the features transferred from the learned defects, the resource consumption of our proposed approach is significantly reduced, and thus can be used on a wide range of edge devices to complete defect detection in a timely manner with high accuracy. The experimental results clearly demonstrate the practicality and effectiveness.

A real-time peak-detection approach for nuclear detection and its implementation on an FPGA

Detecting a pulse correctly is a key process in nuclear detection. Because the radiation emission is a random process, it is hard to design a suitable peak-detection approach in FPGA. The error detection will influence the final energy spectrum and flood histogram. In order to improve the result of nuclear detection, this paper proposes a novel method for nuclear signal peak-detection, which can improve both the effective counting rate and the quality of pulses in real-time. The main method is to establish a normalized reference pulse regardless of waveform through the least squares method. By calculating the loss between the incoming data stream and normalized reference pulse, this algorithm retains the pulses whose loss is below the threshold. We select the threshold based on statistical methods. The algorithm is implemented on field programmable gate array (FPGA) successfully, and this process is able to work in real-time. The result shows that the effective counting rate can improve about 19.8% and more than 99% pile-up and error pulses will be suppressed. By analyzing reserved pulses, the energy spectrum and flood histogram could be well rectified. The energy resolution increases 11% compared with traditional algorithm. Furthermore, due to this new algorithm, the low-energy threshold can be lower.

Differentiated capacity extension method for system of data storage with multilevel structure

Subject of Research. The paper presents a method for differentiated capacity extension of the data warehouse. The method is built on a predictive model of time series with an estimate of volume for the traffic storage. The effect of the incoming data stream structure on the choice of the prediction model is considered. Methods. The storage system is presented in the form of a matrix specifying the number of storage levels and the number of carriers/volumes at each level. The matrix elements are metadata of the recorded files that are stored on the corresponding carriers/volumes of multilevel data storage system. The matrix visualizes the data storage state in the form of patterns. Patterning is performed by systematic slices of matrix values. Periodic analysis of the data warehouse state patterns gives the possibility to evaluate the time to reach the maximum value of the carrier capacity. The predictive model, which is the basis of the method for data warehouse differentiated capacity extension, takes into account the structure of the incoming data stream. In the presence of a self-similar structure of traffic for storage, a predictive model of auto-regression and an integrated moving average is implemented. For traffic without a self-similar structure, a general linear predictive model of the time series at known past values is implemented. The prediction model is applied separately for each storage carrier/volume. Main Results. Structure features of the traffic arriving for storage are given. Self-similarity properties are verified on the example of LTE-traffic, demonstrating the presence of “heavy-tailed” distributions. The prediction results for volume of traffic arriving for storage are obtained by the autoregressive model and the integrated moving average. The predictive and real values of the traffic volume are given, as well as the prediction error value. A technique for differentiated capacity extension of the data storage system is developed, which establishes a sequence of steps for analysis of patterns and the structure of traffic arriving for storage. Practical Relevance. The method for differentiated capacity extension of the data storage takes into account the multilevel organization of storage and the structure of the incoming data stream, which provides organizing a differentiated capacity extension in accordance with the characteristics of the files and ensuring the requirements for guaranteed storage time.

Use of Factors Related to the Consumption of Fast Moving Consumer Goods in Business Intelligence System for Managing Orders to Suppliers in Retail Chain

The study examines some aspects related to current trends in the modeling of business intelligent systems (BIS) specializing in retail chains for Fast Moving Consumer Goods (FMCG ) . Current concepts related to factors that influence business processes and their application in business intelligent order management systems in retail chains for FMCG are presented. The aim of the present study is to investigate the factors that have the strongest influence on the consumption of FMCG in retail chains and to derive the values that support business-intelligent processes for automated order management to suppliers. The studied factors presented in the presentation also include consideration of the structure of the incoming data streams, their extraction and their application in practice.

Investigation of Parallel Data Processing Using Hybrid High Performance CPU

The paper investigates parallel data processing in a hybrid CPU+GPU(s) system using multiple CUDA streams for overlapping communication and computations. This is crucial for efficient processing of data, in particular incoming data stream processing that would naturally be forwarded using multiple CUDA streams to GPUs. Performance is evaluated for various compute time to host-device communication time ratios, numbers of CUDA streams, for various numbers of threads managing computations on GPUs. Tests also reveal benefits of using CUDA MPS for overlapping communication and computations when using multiple processes. Furthermore, using standard memory allocation on a GPU and Unified Memory versions are compared, the latter including programmer added prefetching. Performance of a hybrid CPU+GPU version as well as scaling across multiple GPUs are demonstrated showing good speed-ups of the approach. Finally, the performance per power consumption of selected configurations are presented for various numbers of streams and various relative performances of GPUs and CPUs.

An Efficient Approach for Storage of Big Data Streams in Distributed Stream Processing Systems

Besides, centralized managing, processing and querying, the storage is one of the important components of a big data management. There is always a huge requirement of storing immense volumes of heterogeneous data in different formats. In big data steam processing applications, the storage is given a priority and always plays a big role in historical data analysis. During stream processing, some of the incoming data and the intermediate results are always a good source of future samples. These samples can be used for the future evaluation to eliminate the numerous mistakes of storing and maintaining the big data streams. Hence, a big data stream application requires an efficient support for storage of historical queries. The researchers, scientist and academicians are working hard to develop a sophisticated mechanism that is needed for storage to keep the most useful data for the future references by means of stream archive storage. However, a stream processing system can’t store the whole incoming stream data for future references. A technique is needed to get rid of the expired data and free the space for more incoming data in an archive storage. Hence keeping in view, the storage space limitation, integration issues and its associated cost, we try to optimize the stream archive storage and free more space for future data. The proposed enhanced algorithm will help to delete the obsolete data (retention or expired) and free the space for the new incoming data in a distributed platform. Our paper presents an Enhanced Time Expired Algorithm (ETEA) for stream archived storage in a distributed environment for removing the obsolete data based on time expiration and providing a space for the new incoming data for historical data analysis during the skew time (Hot Spots).We also evaluated the efficiency of our algorithm using the skew factor. The experimental results show that our approach is 98% efficient and fast than other conventional techniques.

Amplitude/Polarization Shift Keying Based and Logic Gate in Polarization Dependent C-Rich SiC Ring Waveguide

The TE/TM-mode spectral-separated C-rich SiC ring waveguide is employed to demonstrate polarization-dependent amplitude/polarization shift keying (ASK/PolSK) “AND” logic gate. The plasma-enhanced chemical vapor deposition-grown C-rich SiC film exhibits a nonstoichiometric SiC nano-cluster phase structure with a C/Si atomic ratio of 1.28, which provides an efficient nonlinearity of γ = 4.3 × 102 W−1m−1 and n2 = 2.44 × 10−12 cm2/W to demonstrate the “AND” gating between incoming ASK and PolSK data streams. By carefully designing the polarization preference of the ring waveguide structure, the TE/TM-mode-dependent transmission combs are spectrally separated because of the high birefringence of the waveguide optically triggered by the incoming pulsed stream. This type of a polarization dependent spectral notch can be utilized to perform the polarization-dependent all-optical AND gating, which was done in the current study by inserting a dual-format modulated pump and a continuous-wave probe into a ring waveguide. Output of the probe can be cross-wavelength modulated to provide the exact “AND” gating between the ASK and PolSK streams only when the pump is set to the ON state and the polarization of the pump is parallel to the corresponding notch of the ring waveguide throughput. This type of C-rich SiC ASK/PolSK AND logic gate is fully compatible with silicon-based complementary metal-oxide-semiconductor (CMOS) technology, which also has the advantages of low power consumption and enables polarization engineering for future photonic data logics.

Design and Implementation of an Advanced Radar Signal Processor with Waveform Generator and BIST Unit on a Single FPGA Chip

Field Programmable Gate Arrays (FPGAs) are best suited for signal processing applications that require real-time processing. Therefore, it is preferred over DSP processors in implementing radar receivers that processes incoming continuous stream of data. In the past, implementing complex arithmetic operations in floating point representation was a monopoly on DSP processors and the designers had to work around the sequential nature of the DSP processor to make it suitable for real-time applications by using buffered and multi-clocking designs. But nowadays, the great advances in FPGA design technology minimized this design effort since it is capable of performing these complex algorithms in real time. This paper represents the design and implementation of an advanced radar signal processor for binary phase-coded pulsed radar incorporating a time range side lobes suppression technique on a single FPGA chip. The proposed hardware design includes a waveform generator, an advanced signal processing unit, and a built-in self-test (BIST). Design aspects and hardware details for each part is introduced thoroughly.

Experimental Analysis on Processing of Unbounded Data

Processing of unordered and unbounded data is the prime requirement of the current businesses. Large amount of rapidly generated data demands the processing of the same without the storage and as per the timestamp associated with it. It is difficult to process these unbounded data with batch engine as the existing batch systems suffer from the delay intrinsic by accumulating entire incoming records in a group prior to process it. However windowing can be useful when dealing with unbounded data which pieces up a dataset into fixed chunks for processing with repeated runs of batch engine. Contrast to batch processing, stream handling system aims to process information that is gathered in a little timeframe. In this way, stream data processing ought to be coordinated with the flow of data. In the real world the event time is always skewed with the processing time which introduce issues of delay and completeness in incoming stream of data. In this paper, we presented the analysis on the watermark and trigger approach which can be used to manage these unconventional desires in the processing of unbounded data.

FAIR.ReD: Semantic knowledge graph infrastructure for the life sciences

We would like to present FAIR Research Data: Semantic Knowledge Graph Infrastructure for the Life Sciences (in short, FAIR.ReD), a project initiative that is currently being evaluated for funding. FAIR.ReD is a software environment for developing data management solutions according to the FAIR (Findable, Accessible, Interoperable, Reusable; Wilkinson et al. 2016) data principles. It utilizes what we call a Data Sea Storage, which employs the idea of Data Lakes to decouple data storage from data access but modifies it by storing data in a semantically structured format as either semantic graphs or semantic tables, instead of storing them in their native form. Storage follows a top-down approach, resulting in a standardized storage model, which allows sharing data across all FAIR.ReD Knowledge Graph Applications (KGAs) connected to the same Sea, with newly developed KGAs having automatically access to all contents in the Sea. In contrast access and export of data follows a bottom-up approach that allows the specification of additional data models to meet the varying domain-specific and programmatic needs for accessing structured data. The FAIR.ReD engine enables bidirectional data conversion between the two storage models and any additional data model, which will substantially reduce conversion workload for data-rich institutes (Fig. 1). Moreover, with the possibility to store data in semantic tables, FAIR.ReD provides high performance storage for incoming data streams such as sensory data. FAIR.ReD KGAs are modularly organized. Modules can be edited using the FAIR.ReD editor and combined to form coherent KGAs. The editor allows domain experts to develop their own modules and KGAs without any programming experience required, thus also allowing smaller projects and individual researchers to build their own FAIR data management solution. Contents from FAIR.ReD KGAs can be published under a Creative Commons license as documents, micropublications, or nanopublications, each receiving their own DOI. A publication-life-cycle is implemented in FAIR.ReD and allows updating published contents for corrections or additions without overwriting the originally published version. Together with the fact that data and metadata are semantically structured and machine-readable, all contents from FAIR.ReD KGAs will comply with the FAIR Guiding Principles. Due to all FAIR.Red KGAs providing access to semantic knowledge graphs in both a human-readable and a machine-readable version, FAIR.ReD seamlessly integrates the complex RDF (Resource Description Framework) world with a more intuitively comprehensible presentation of data in form of data entry forms, charts, and tables. Guided by use cases, the FAIR.ReD environment will be developed using semantic programming where the source code of an application is stored in its own ontology. The set of source code ontologies of a KGA and its modules provides the steering logic for running the KGA. With this clear separation of steering logic from interpretation logic, semantic programming follows the idea of separating main layers of an application, analog to the separation of interpretation logic and presentation logic. Each KGA and module is specified exactly in this way and their source code ontologies stored in the Data Sea. Thus, all data and metadata are semantically transparent and so is the data management application itself, which substantially improves their sustainability on all levels of data processing and storing.

A stream-sensitive distributed approach for configuring cascaded classifier topologies in real-time large-scale stream mining systems

Stream mining systems have received a great deal of attention in recent years. These systems process incoming data streams from different sources and extract high-level semantic features from them. They do this by passing data streams through an ensemble of classifiers. Owing to dynamic changes in characteristics of the data streams, these classifiers need to be configured dynamically to maximize the performance of the system. As a challenge, different data streams from different sources have different specifications from each other. This causes treating all the incoming data streams identically by a common topology configuration to be not appropriate for an optimal stream mining. Hence, an approach is required which allows each data stream to be processed by consideration of its own specifications. In this paper, by implementing a buffer for each source and using a time-sharing solution, we propose a distributed approach to solve the aforementioned problem for cascaded classifier topologies. We first formally define a utility metric which captures both the performance and the delay of a binary filtering classifier system. We then propose our solution for a base case and evolve it step by step until reaching the most general case for cascaded topologies. We finally test and compare our approach with the state-of-the-art solution on a text detection scenario from the incoming video streams to the system.

Lattice-Modeled Information Flow Control of Big Sensing Data Streams for Smart Health Application

Internet of Things (IoT) provides a promising opportunity to build powerful data analytics systems with real time event detection for smart health, and therefore wearable IoT has become a rising source of big data streams for smart health, for which security needs to be assured by detecting real-time event to avoid malicious activities, and meanwhile to control the information leakage of big sensing data streams. I refer to this as an information flow control problem. To address this problem, this paper proposes a static lattice model for information flow control over big sensing data streams. I initialize two static lattices, i.e., sensor lattice for wearable sensors and user lattice for users, and then static lattices aim to process the flow control model faster, because I am dealing with high volume and velocity of data streams. The experimental evaluation and results of the information flow model show that it can excellently handle the incoming big data streams with low latency and buffer requirement.

Histogram-based clustering of multiple data streams

This paper introduces a strategy for clustering online multiple data streams. We assume that several sources are used for recording, over time, data about some physical phenomena. Each source provides repeated measurements at a very high frequency so that it is not possible to store the whole amount of data into some easy-to-access media, but data are available only in batches. Our aim is to discover a partition of the sources (e.g. sensors) into homogeneous clusters, analysing the incoming streams of data. The proposed strategy is based on processing the incoming data batches independently, through an initial summarization of the data batches by histograms and, then, by means of a local clustering performed on the histograms which provides a further data summarization. To keep track of the data proximities among the data streams over time, we use local clustering outputs for updating a proximity matrix. The final partitioning of the streams is obtained by a clustering based on such proximity matrix. Through an application on real and simulated data, we show the effectiveness of our strategy in finding homogeneous groups of sources of data streams.

Summarizing and linking electronic health records

In recent years, several applications have emerged which require access to consolidated information that has to be computed and provided in near real-time. Traditional record linkage algorithms are unable to support such time-critical applications, as they perform the linkage offline and provide the result set only when the entire process has completed. To address this need, in this paper we propose the first summarization algorithms that operate in the blocking and matching steps of record linkage to speed up online linkage tasks. Our first method, called SkipBloom, efficiently summarizes the participating data sets, using their blocking keys, to allow for very fast comparisons among them. The second method, called BlockSketch, summarizes a block to achieve a constant number of comparisons for a submitted query record, during the matching phase. Our third method, SBlockSketch, operates on data streams, where the entire data set is unknown a-priori but, instead, there is a potentially unbounded stream of incoming data records. Finally, we introduce PBlockSketch, which adapts BlockSketch to privacy-preserving settings. Through extensive experimental evaluation, using real-world data sets, we show that our methods outperform the state-of-the-art algorithms for online record linkage in terms of the time needed, the memory used, and the recall and precision rates that are achieved during the linkage process. Following the evaluation of our approaches, we introduce SFEMRL, a novel framework that uses them to enable the linkage of electronic health records at large scale, while respecting patients’ privacy. Under this framework, patient records first undergo a data masking process that perturbs sensitive information in data fields of the records to protect it. Subsequently, they participate in a parallel and distributed ecosystem, whose goal is to persist these records in order to be queried efficiently and accurately. We demonstrate that the integration of our framework with Map/Reduce offers robust distributed solutions for performing on-demand large-scale privacy-preserving record linkage tasks in the health domain.