IBM InfoSphere Streams Research Articles

Performance analysis of real-time face detection system based on stream data mining frameworks

This work describes performance analysis results of two face detection systems based on Apache Storm and IBM InfoSphere Streams frameworks. Profiling was evaluated on image sequences of four different sizes: 100 x 100, 640 x 640, 1920 x 1080, and 4096 x 3112. Face detection was performed by OpenCV cascade classifier. Experiment was held under five CentOS nodes cluster. It was investigated that system based on Apache Storm was able to operate in real-time at 24 frames per second on used hardware configuration. Apache Storm was more scalable and demonstrated advantage in throughput over its counterpart. Experiment helped to reveal configuration parameters of frameworks that played a major role in face detection task on image sequences.

Real-Time Earthquake Intensity Estimation Using Streaming Data Analysis of Social and Physical Sensors

Earthquake intensity is one of the key components of the decision-making process for disaster response and emergency services. Accurate and rapid intensity calculations can help to reduce total loss and the number of casualties after an earthquake. Modern intensity assessment procedures handle a variety of information sources, which can be divided into two main categories. The first type of data is that derived from physical sensors, such as seismographs and accelerometers, while the second type consists of data obtained from social sensors, such as witness observations of the consequences of the earthquake itself. Estimation approaches using additional data sources or that combine sources from both data types tend to increase intensity uncertainty due to human factors and inadequate procedures for temporal and spatial estimation, resulting in precision errors in both time and space. Here we present a processing approach for the real-time analysis of streams of data from both source types. The physical sensor data is acquired from the U.S. Geological Survey (USGS) seismic network in California and the social sensor data is based on Twitter user observations. First, empirical relationships between tweet rate and observed Modified Mercalli Intensity (MMI) are developed using data from the M6.0 South Napa, CAF earthquake that occurred on August 24, 2014. Second, the streams of both data types are analyzed together in simulated real-time to produce one intensity map. The second implementation is based on IBM InfoSphere Streams, a cloud platform for real-time analytics of big data. To handle large processing workloads for data from various sources, it is deployed and run on a cloud-based cluster of virtual machines. We compare the quality and evolution of intensity maps from different data sources over 10-min time intervals immediately following the earthquake. Results from the joint analysis shows that it provides more complete coverage, with better accuracy and higher resolution over a larger area than either data source alone.

Real-Time Processing of Continuous Physiological Signals in a Neurocritical Care Unit on a Stream Data Analytics Platform.

Continuous high-volume and high-frequency brain signals such as intracranial pressure (ICP) and electroencephalographic (EEG) waveforms are commonly collected by bedside monitors in neurocritical care. While such signals often carry early signs of neurological deterioration, detecting these signs in real time with conventional data processing methods mainly designed for retrospective analysis has been extremely challenging. Such methods are not designed to handle the large volumes of waveform data produced by bedside monitors. In this pilot study, we address this challenge by building a prototype system using the IBM InfoSphere Streams platform, a scalable stream computing platform, to detect unstable ICP dynamics in real time. The system continuously receives electrocardiographic and ICP signals and analyzes ICP pulse morphology looking for deviations from a steady state. We also designed a Web interface to display in real time the result of this analysis in a Web browser. With this interface, physicians are able to ubiquitously check on the status of their patients and gain direct insight into and interpretation of the patient's state in real time. The prototype system has been successfully tested prospectively on live hospitalized patients.

A scalable architecture for real-time analysis of microblogging data

As events take place in the real world, e.g., sports games and marketing campaigns, people react and interact on online social networks (OSNs), especially microblog services such as Twitter, generating a large stream of data. Analyzing this data presents an opportunity for researchers and companies to better understand human behavior (both on the network and in real life) during the event's lifespan. Designing automated systems to conduct these analyses in fractions of minutes (or even seconds) is subjected to many challenges: the volume of data is large, the number of posts in future events cannot be predicted, and the system need to be always available and running smoothly to avoid information loss and delays on delivering the analytics results. In this paper, we present a scalable architecture for real-time analysis of microblogging data, with the ability to deal with large volumes of posts, by considering modular parallel workflows. This architecture, which has been implemented on the IBM InfoSphere Streams platform, was tested on a real-world use case to conduct sentiment analysis of Twitter posts during the games of the 2013 Fédération Internationale de Football Association (FIFA) Confederations Cup, and the system has successfully coped with the challenges of this task.

A design proto‐pattern for continuously evaluated forecasting in IBM® InfoSphere® Streams

SummaryDesign patterns relieve much of the difficulty of developing software solutions. As sources and volumes of generated data have increased in recent times, stream computing has emerged as a new paradigm for acquiring and harnessing the data's rich potential to provide insight in near real time. One prominent platform for processing streaming data is IBM® InfoSphere® Streams, but design patterns for the platform are relatively scarce. This article proposes a prototypical application design pattern for continuously evaluated forecasting on the IBM InfoSphere Streams platform. Towards demonstrating the prototypical pattern, the application was implemented in three real‐world scenarios: wind power, sentiment analysis, and stock market data. The purpose of this article is to specify the design proto‐pattern and describe its variants in each different implementation. We believe that the proto‐pattern will be applicable in IBM InfoSphere Streams to any data stream where forecasts over multiple forecast horizons are required. Copyright © 2015 John Wiley & Sons, Ltd.

A Pipelining Implementation for High Resolution Seismic Hazard Maps Production

Seismic hazard maps are a significant input into emergency hazard management that play an important role in saving human lives and reducing the economic effects after earthquakes. Despite the fact that a number of software tools have been developed (McGuire, 1976, 1978; Bender & Perkins, 1982, 1987; Robinson et al., 2005, 2006; Field et al., 2003), map resolution is generally low, potentially leading to uncertainty in calculations of ground motion level and underestimation of the seismic hazard in a region. In order to generate higher resolution maps, the biggest challenge is to handle the significantly increased data processing workload. In this study, a method for improving seismic hazard map resolution is presented that employs a pipelining implementation of the existing EqHaz program suite (Assatourians & Atkinson, 2013) based on IBM InfoSphere Streams–an advanced stream computing platform. Its architecture is specifically configured for continuous analysis of massive volumes of data at high speeds and low latency. Specifically, it treats processing workload as data streams. Processing procedures are implemented as operators that are connected to form processing pipelines. To handle large processing workload, these pipelines are flexible and scalable to be deployed and run in parallel on large-scale HPC clusters to meet application performance requirements. As a result, mean hazard calculations are possible for maps with resolution up to 2,500,000 points with near-real-time processing time of approximately 5-6 minutes.

Real-time analysis of parameters of multiple object detection systems

Analysis of two streaming frameworks: Apache Storm and IBM InfoSphere Streams was performed in solving a multiple object detection task. The analysis focused on two parameters: throughput and 95th percentile of image processing delay. Faces were chosen as the objects to be detected. Profiling was held under CentOS operating systems running on a five node cluster. Face detection was performed using an OpenCV cascade classifier. First architectures and the experiment description were covered. Final suggestions on the applicability of the two systems were made in the concluding section of the article. Apache Storm demonstrated a scalability advantage over IBM InfoSphere Streams in the experiment conducted. It was confirmed that the system based on Apache Storm was able to operate on FullHD video in real-time, achieving throughput of 24 images per second on the hardware used.

Fundamentals of stream processing: application design, systems, and analytics

Stream processing is a novel distributed computing paradigm that supports the gathering, processing, and analysis of high-volume, heterogeneous, continuous data streams, to extract insights and actionable results in real time. This comprehensive, hands-on guide combining the fundamental building blocks and emerging research in stream processing is ideal for application designers, system builders, analytic developers, as well as students and researchers in the field. This book introduces the key components of the stream computing paradigm, including the distributed system infrastructure, the programming model, design patterns, and streaming analytics. The explanation of the underlying theoretical principles, illustrative examples and implementations using the IBM InfoSphere Streams SPL language, and real-world case studies provide students and practitioners with a comprehensive understanding of such applications and the middleware that supports them.

SoLoMo analytics for telco Big Data monetization

The mobile Internet brought tremendous opportunities for businesses to capitalize on the vast amount of SoLoMo (social-location-mobile) data for delivering high-quality and personalized customer services. In this paper, we describe algorithms and technologies for discovering actionable customer insights using the combined power of social network, location pattern mining, and mobile usage analysis. We illustrate our implementation using Big Data platforms including IBM InfoSphere® BigInsights, IBM InfoSphere Streams, and IBM Netezza® Data Warehouse, while addressing various Big Data-related challenges, such as context generation of unstructured data and high-performance analytics for both data at rest and data in motion. The presented system combines location, social interactions, and user behavior data to find like-minded communities. The system leverages Big Data capabilities to attempt to scale to support the subscriber base of large telecoms in an efficient manner.

Comparison of system performance for streaming data analysis in image processing tasks by sliding window

Comparison of two streaming systems: Apache Storm and IBM InfoSphere Streams was performed in solving image sliding window filtering task. Analysis was focused on two parameters: throughput and memory consumption. Profiling was held under CentOS operating systems running on two virtual machines for each system. First, the most cheap in developing cost realizations were compared. After analysis optimization of memory consumption was performed. Final suggestions on applicability of two systems were made at the end of the article.

High-Performance Parallel and Stream Processing of X-ray Microdiffraction Data on Multicores

We present the design and implementation of a high-performance system for processing synchrotron X-ray microdiffraction (XRD) data in IBM InfoSphere Streams on multicore processors. We report on the parallel and stream processing techniques that we use to harvest the power of clusters of multicores to analyze hundreds of gigabytes of synchrotron XRD data in order to reveal the microtexture of polycrystalline materials. The timing to process one XRD image using one pipeline is about ten times faster than the best C program at present. With the support of InfoSphere Streams platform, our software is able to be scaled up to operate on clusters of multi-cores for processing multiple images concurrently. This system provides a high-performance processing kernel to achieve near real-time data analysis of image data from synchrotron experiments.

Data Stream Management System and Capital Market Surveillance

Importance of real-time data analysis has been felt since early '90s and thus processing of streaming data (from either sensor networks or telecom switches or web and other disparate systems) is the demand of the industries worldwide. Quicker detection of fraudulent activities in a financial system is the order of the day. Thus capital market surveillance, if can be performed by using the streaming input of various trading transactions, without being stored, that would be beneficial to the regulatory authorities and stock exchanges. In this paper, we describe how stream processing using a data stream management system (DSMS) can be used for the above task and how effective would be that in terms of performance and latency. We present results obtained from using a commercial event stream processing system (IBM InfoSphere Streams platform) for certain typical fraud detection scenarios.

A model‐based framework for building extensible, high performance stream processing middleware and programming language for IBM InfoSphere Streams

SUMMARYThis work presents an extensive case study on the model‐based design of a commercial‐grade stream processing middleware (IBM's InfoSphere Streams) its runtime and language (SPL) compiler. The model‐based underpinnings are pervasive throughout the whole environment, from describing inter‐process communication interfaces and objects to the design of the extensibility mechanism in the runtime and language. In addition to many software engineering advantages such as consistent, uniform, and self‐documented integration among the different parts of the system, we show intrinsic performance benefits to the platform derived from this design approach. First, we demonstrate how an incremental compilation strategy employed by the SPL compiler and rooted on the model description of the application, extracted by the compiler as part of the application building process, leads to better compile‐time performance. Second, we discuss how the model‐based code generation strategy employed by the SPL compiler also leads to increased runtime performance, by specializing the generated code to particular characteristics of the runtime environment. Finally, we show how the extensibility strategy used in the SPL language leads to automatic syntactic and semantic checks at compile time, while enabling behavioral reasoning and specific optimizations at runtime. Copyright © 2011 John Wiley & Sons, Ltd.

A Pipelining Implementation for Parsing X-ray Diffraction Source Data and Removing the Background Noise

Synchrotrons can be used to generate X-rays in order to probe materials at the atomic level. One approach is to use X-ray diffraction (XRD) to do this. The data from an XRD experiment consists of a sequence of digital image files which for a single scan could consist of hundreds or even thousands of digital images. Existing analysis software processes these images individually sequentially and is usually used after the experiment is completed. The results from an XRD detector can be thought of as a sequence of images, generated during the scan by the X-ray beam. If these images could be analyzed in near real-time, the results could be sent to the researcher running the experiment and used to improve the overall experimental process and results. In this paper, we report on a stream processing application to remove background from XRD images using a pipelining implementation. We describe our implementation techniques of using IBM Infosphere Streams for parsing XRD source data and removing the background. We present experimental results showing the super-linear speedup attained over a purely sequential version of the algorithm on a quad-core machine. These results demonstrate the potential of making good use of multi-cores for high-performance stream processing of XRD images.