Monitoring Queries Research Articles

Rule based complex event processing for an air quality monitoring system in smart city

Smart city development is accelerating, aiming for systematic urban living, but challenges like air quality and traffic management need addressing for sustainability in cities. The World Health Organization (WHO) links air pollution to severe respiratory diseases, emphasizing the need for real-time air quality monitoring and timely decisions based on varying pollutant concentrations (NH3, O3, SO2, NO2) across locations. This research work proposes a framework combining Siddhi Complex Event Processing (CEP) and SPARQL queries for air quality monitoring. Dataset collected from Central Pollution Control Board (CPCB) India, is preprocessed, passed through Apache Kafka, and transformed into a knowledge graph. The RDF data is integrated with a knowledge graph via Apache Jena to enhance CEP performance. Decision trees generate rules and enrich CEP engines with CPCB parameters to identify complex patterns. SPARQL queries on real-time RDF data categorize air quality conditions for all stakeholders. The model's effectiveness is validated by deploying RDF chunks to the CEP engine and evaluating performance through various query execution times. Our model efficiently processed numerous air pollutant events (200, 400, 600 etc.), filtering out about 80% and demonstrating robust event processing capabilities, which enhances air quality management and supports sustainable urban ecosystems and societies.

FTS Service Evolution and LHC Run-3 Operations

FTS Service Evolution and LHC Run-3 Operations

A generation of synthesized trajectories of free moving objects for testing a set of continuous spatio‐temporal queries

A generation of synthesized trajectories of free moving objects for testing a set of continuous spatio‐temporal queries

Efficient and error-bounded spatiotemporal quantile monitoring in edge computing environments

Underlying many types of data analytics, a spatiotemporal quantile monitoring (SQM) query continuously returns the quantiles of a dataset observed in a spatiotemporal range. In this paper, we study SQM in an Internet of Things (IoT) based edge computing environment, where concurrent SQM queries share the same infrastructure asynchronously. To minimize query latency while providing result accuracy guarantees, we design a processing framework that virtualizes edge-resident data sketches for quantile computing. In the framework, a coordinator edge node manages edge sketches and synchronizes edge sketch processing and query executions. The co-ordinator also controls the processed data fractions of edge sketches, which helps to achieve the optimal latency with error-bounded results for each single query. To support concurrent queries, we employ a grid to decompose queries into subqueries and process them efficiently using shared edge sketches. We also devise a relaxation algorithm to converge to optimal latencies for those subqueries whose result errors are still bounded. We evaluate our proposals using two high-speed streaming datasets in a simulated IoT setting with edge nodes. The results show that our proposals achieve efficient, scalable, and error-bounded SQM.

Newton: Intent-Driven Network Traffic Monitoring

Network monitoring systems are designed to fulfill operators’ intents and serve as essential tools to modern networks. As a result of rapidly increasing network bandwidth and scale nowadays, network monitors should satisfy on-demand network monitoring for continuously growing traffic volumes. However, existing monitoring systems either cannot satisfy flexible intents on demand or produce significant overheads. In this paper, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Newton</i> , an intent-driven traffic monitor that is able to specify operators’ intents with traffic monitoring queries and conduct dynamic and scalable network-wide queries deployment. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Newton</i> enables operators to customize and modify queries dynamically without interrupting the network workflow. Besides, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Newton</i> proposes systematic optimizations at device level and network-wide level to reduce resource consumption while deploying queries. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Newton</i> can combine the resources across switches to deploy complex queries with high resilience to dynamic network status. Evaluations prove that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Newton</i> is of high flexibility, scalability, and resource efficiency, which demonstrates <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Newton</i> is promising to be deployed in large-scale programmable networks.

Analysis of MP4 Videos in 5G Using SDN

Current video content providers require a huge demand for network resources to obtain the best quality of experience from the end-users perspective. This measure can be achieved if the network can anticipate itself to different existing issues (bandwidth problems, etc.). Due to traditional Internet Protocol networks cannot handle this problem due to the rigidity of its architecture, software defined networks is introduced as a potential answer for resolving the existing challenges in video content delivery decoupling the control plane and data plane. Utilizing the unified perspective of the network, it is possible to develop a monitoring tool obtaining the network metrics to be used afterwards. In this paper, we propose the application of different clustering algorithms for monitoring software defined networks to validate the number of statistics queries decrements using the data and error rates to improve the network traffic and to reduce the overload. The simulations demonstrate that the expectation-maximization algorithm highly reduces the monitoring queries maintaining the best data and error rate values in the video delivery.

Digital Intelligent Management Platform for High-Rise Building Construction Based on BIM Technology

In this study, the digital intelligent management platform of high-rise building construction based on BIM technology is used for real-time monitoring and management of construction progress and quality. In the data acquisition and processing layer, construction site data is obtained through RFID technology. After processing such as cleaning and integration, it is input to the BIM model layer to dynamically generate various real-time BIM models, and these real-time BIM model information is input to the application layer to query, monitor and correct the construction progress and quality. The results are presented by the display layer. The actual application results show that the real-time BIM models generated by the platform have clear details and can realize the query, monitoring and correction functions for the construction progress of high-rise buildings, and effectively correct the construction progress according to the construction progress monitoring query results to achieve the unification with the planned progress. It can effectively realize the visual measurement of the size of each component in construction and monitor the construction quality in real time.

Data Stream Processing for Packet-Level Analytics.

One of the most challenging tasks for network operators is implementing accurate per-packet monitoring, looking for signs of performance degradation, security threats, and so on. Upon critical event detection, corrective actions must be taken to keep the network running smoothly. Implementing this mechanism requires the analysis of packet streams in a real-time (or close to) fashion. In a softwarized network context, Stream Processing Systems (SPSs) can be adopted for this purpose. Recent solutions based on traditional SPSs, such as Storm and Flink, can support the definition of general complex queries, but they show poor performance at scale. To handle input data rates in the order of gigabits per seconds, programmable switch platforms are typically used, although they offer limited expressiveness. With the proposed approach, we intend to offer high performance and expressive power in a unified framework by solely relying on SPSs for multicores. Captured packets are translated into a proper tuple format, and network monitoring queries are applied to tuple streams. Packet analysis tasks are expressed as streaming pipelines, running on general-purpose programmable network devices, and a second stage of elaboration can process aggregated statistics from different devices. Experiments carried out with an example monitoring application show that the system is able to handle realistic traffic at a 10 Gb/s speed. The same application scales almost up to 20 Gb/s speed thanks to the simple optimizations of the underlying framework. Hence, the approach proves to be viable and calls for the investigation of more extensive optimizations to support more complex elaborations and higher data rates.

Video Monitoring Queries

Recent advances in video processing utilizing deep learning primitives achieved breakthroughs in fundamental problems in video analysis such as frame classification and object detection enabling an array of new applications. In this paper we study the problem of interactive declarative query processing on video streams. In particular we introduce a set of approximate filters to speed up queries that involve objects of specific type (e.g., cars, trucks, etc.) on video frames with associated spatial relationships among them (e.g., car left of truck). The resulting filters are able to assess quickly if the query predicates are true to proceed with further analysis of the frame or otherwise not consider the frame further avoiding costly object detection operations. We propose two classes of filters <inline-formula><tex-math notation="LaTeX">$IC$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$OD$</tex-math></inline-formula> , that adapt principles from deep image classification and object detection. The filters utilize extensible deep neural architectures and are easy to deploy and utilize. In addition, we propose statistical query processing techniques to process aggregate queries involving objects with spatial constraints on video streams and demonstrate experimentally the resulting increased accuracy on the resulting aggregate estimation. Finally, we introduce a framework based on extreme value theory to detect unexpected objects on video streams and experimentally demonstrate its utility. Combined these techniques constitute a robust set of video monitoring query processing techniques. We demonstrate that the application of the techniques proposed in conjunction with declarative queries on video streams can dramatically increase the frame processing rate and speed up query processing by at least two orders of magnitude. We present the results of a thorough experimental study utilizing benchmark video data sets at scale demonstrating the performance benefits and the practical relevance of our proposals.

Ananke

Data streaming enables online monitoring of large and continuous event streams in Cyber-Physical Systems (CPSs). In such scenarios, fine-grained backward provenance tools can connect streaming query results to the source data producing them, allowing analysts to study the dependency/causality of CPS events. While CPS monitoring commonly produces many events, backward provenance does not help prioritize event inspection since it does not specify if an event's provenance could still contribute to future results. To cover this gap, we introduce Ananke , a framework to extend any fine-grained backward provenance tool and deliver a live bipartite graph of fine-grained forward provenance. With Ananke , analysts can prioritize the analysis of provenance data based on whether such data is still potentially being processed by the monitoring queries. We prove our solution is correct, discuss multiple implementations, including one leveraging streaming APIs for parallel analysis, and show Ananke results in small overheads, close to those of existing tools for fine-grained backward provenance.

Pivot tracing

Monitoring and troubleshooting distributed systems are notoriously difficult; potential problems are complex, varied, and unpredictable. The monitoring and diagnosis tools commonly used today---logs, counters, and metrics---have two important limitations: what gets recorded is defined a priori, and the information is recorded in a component- or machine-centric way, making it extremely hard to correlate events that cross these boundaries. This paper presents Pivot Tracing, a monitoring framework for distributed systems that addresses both limitations by combining dynamic instrumentation with a novel relational operator: the happened-before join. Pivot Tracing gives users, at runtime, the ability to define arbitrary metrics at one point of the system, while being able to select, filter, and group by events meaningful at other parts of the system, even when crossing component or machine boundaries. Pivot Tracing does not correlate cross-component events using expensive global aggregations, nor does it perform offline analysis. Instead, Pivot Tracing directly correlates events as they happen by piggybacking metadata alongside requests as they execute. This gives Pivot Tracing low runtime overhead---less than 1% for many cross-component monitoring queries.

Bot detection by monitoring and grouping domain name server record response queries in DNS traffic

A Botnet is network of bots or infected systems, each running on a compromised host machine, controlled by command and control server. Botnet can be used for email spamming to launching DDoS attacks. Botnet attacks classified as topology based, protocol based, and architecture based. Designing a detection system for bots is becoming challenging as botnet attacks are upgrading the attacking methodology by hiding and changing identities (command and control server) periodically. This paper is proposing, bot detection methodology by monitoring domain name server record response (DNSRR) query traffic, which form a group activity in DNS traffic simultaneously sent by bot machines. The analysis is based on type of botnet attack, detection target, feature source, feature extraction, feature correlation, machine learning techniques. Few researchers proposed bot detection techniques based on DNS queries initiated by bots, but these can be easily avoided by changing bot program, architecture, protocol and encrypted network traffic. The proposed approach is versatile and robust than the existing detection approaches so that the presence of variety of bots can be detected by monitoring the group activities of DNSRR queries in DNS traffic. From the experiment and results, it is shown that proposed methodology able to detect bot efficiently while they are connected to controlling server or migrating to new server. The results are encouraging because of low false positive detection rate.

0322 Development of Complex Data Platform for the Stanford Technology Analytics and Genomics in Sleep (STAGES) Study

The Stanford Technology Analytics and Genomics in Sleep (STAGES) is a prospective cross-sectional, multi-site study charged with collecting and disseminating data to the scientific community. The following data elements will be obtained from 30,000 adult/adolescent sleep clinic patients: (1) in-lab polysomnography, (2) online questionnaire (Alliance Sleep Questionnaire), (3) blood sample (DNA, serum, plasma), (4) University of Pennsylvania’s computerized neurocognitive test battery (CNB), (5) 3D photo, (6) 2-weeks of actigraphy with sleep diary, and (7) clinical data from the electronic health record. The multifaceted nature of the project required a robust data platform employing creative techniques to receive, organize, and share an array of electronic data elements aggregated from multiple external systems. STAGES partnered with Prometheus Research, LLC to design and build a customized platform using their open source integrated research registry, RexStudy. The teams met regularly to develop a system that would meet the project needs. Requirements included: mechanisms to securely transfer data from multiple disparate data sources and match to correct subject, patient portal to enroll participants via electronic consent and accommodate electronic case report forms, extensive data validation and error reporting, granular access privileges for different user classes, sophisticated querying and analytic tools, and ability to export all data for complete portability. Each phase of the project, including creation of custom APIs, was heavily tested in a user acceptance testing (UAT) environment before being deployed to production. The STAGES data platform currently holds data from the 530 enrolled research participants. Fifty members of the STAGES team have accounts for entering, analyzing, or monitoring the 75 plus data tables, and so far, there are more than 20 queries for data monitoring and quality control. The STAGES data platform was successfully deployed and is being used to collect, monitor, and analyze data from multiple external sources. We are continuing to add functionality to enhance the system’s utility. Klarman Family Foundation

Clustering and Flow Conservation Monitoring Tool for Software Defined Networks.

Prediction systems present some challenges on two fronts: the relation between video quality and observed session features and on the other hand, dynamics changes on the video quality. Software Defined Networks (SDN) is a new concept of network architecture that provides the separation of control plane (controller) and data plane (switches) in network devices. Due to the existence of the southbound interface, it is possible to deploy monitoring tools to obtain the network status and retrieve a statistics collection. Therefore, achieving the most accurate statistics depends on a strategy of monitoring and information requests of network devices. In this paper, we propose an enhanced algorithm for requesting statistics to measure the traffic flow in SDN networks. Such an algorithm is based on grouping network switches in clusters focusing on their number of ports to apply different monitoring techniques. Such grouping occurs by avoiding monitoring queries in network switches with common characteristics and then, by omitting redundant information. In this way, the present proposal decreases the number of monitoring queries to switches, improving the network traffic and preventing the switching overload. We have tested our optimization in a video streaming simulation using different types of videos. The experiments and comparison with traditional monitoring techniques demonstrate the feasibility of our proposal maintaining similar values decreasing the number of queries to the switches.

The SSP-Tree: A Method for Distributed Processing of Range Monitoring Queries in Road Networks

This paper addresses the problem of processing range monitoring queries, each of which continuously retrieves moving objects that are currently located within a given query range. In particular, this paper focuses on processing range monitoring queries in the road network, where movements of the objects are constrained by a predefined set of paths. One of the most important challenges of processing range monitoring queries is how to minimize the wireless communication cost and the server computation cost, both of which are heavily dependent on the amount of location-update stream generated by moving objects. The traditional centralized methods for range monitoring queries assume that moving objects periodically send location-updates to the server. However, when the number of moving objects becomes increasingly large, such an assumption may no longer be acceptable because the amount of location-update stream becomes enormous. Recently, some distributed methods have been proposed, where moving objects utilize their available computational capabilities for sending location-updates to the server only when necessary. Unfortunately, the existing distributed methods only deal with the objects moving in Euclidean space, and thus they cannot be extended to processing range monitoring queries over the objects moving along the road network. In this paper, we propose the distributed method for processing range monitoring queries in the road network. To utilize the computational capabilities of moving objects, we introduce the concept of vicinity region. A vicinity region, assigned to each moving object o, makes o monitor whether or not it should be included in the results of nearby queries. The proposed method includes (i) a new spatial index structure, called the Segment-based Space Partitioning tree (SSP-tree) whose role is to efficiently search the appropriate vicinity regions for moving objects based on their heterogeneous computational capabilities and (ii) the details of the communication strategy between the server and moving objects, which significantly reduce the wireless communication cost as well as the server computation cost. Through simulations, we verify the effectiveness for processing range monitoring queries over a large number of moving objects (up to 100,000) in the road network (modeled as an undirected graph).

Subscription-based data aggregation techniques for top-k monitoring queries

With the increase of data generation in distributed fashions such as peer-to-peer systems and sensor networks, top-k query processing which is a way to aggregate only a small set of data that sufficiently satisfies many users' preferences, becomes a substantial issue. When data are periodically updated in each epoch e.g., weather information, without any techniques, a naive solution is to aggregate all data and their updates to ensure the completeness of final answers, however, it is too costly in terms of data transfer especially for data aggregator nodes (intermediate nodes). In this paper, we propose a top-k monitoring query processing method in 2-tier distributed systems based on a publish-subscribe scheme. A set of top-k subscriptions specifying summary scope of users' interests is informed to aggregators to limit the number of transferred data records for each epoch. In addition, instead of issuing subscriptions of all queries, our method identifies a small set of minimal subscriptions as well as utilizes some adaptive heuristic rules to efficiently maintain those subscriptions resulting in lower communication overhead. Our experiments through both synthetic and real datasets show that our technique is efficient and outperforms other comparative reactive methods.

Geohashed Spatial Index Method for a Location-Aware WBAN Data Monitoring System Based on NoSQL

The exceptional development of electronic device technology, the miniaturization of mobile devices, and the development of telecommunication technology has made it possible to monitor human biometric data anywhere and anytime by using different types of wearable or embedded sensors. In daily life, mobile devices can collect wireless body area network (WBAN) data, and the co-collected location data is also important for disease analysis. In order to efficiently analyze WBAN data, including location information and support medical analysis services, we propose a geohash-based spatial index method for a location-aware WBAN data monitoring system on the NoSQL database system, which uses an R-tree-based global tree to organize the real-time location data of a patient and a B-tree-based local tree to manage historical data. This type of spatial index method is a support cloud-based location-aware WBAN data monitoring system. In order to evaluate the proposed method, we built a system that can support a JavaScript Object Notation (JSON) and Binary JSON (BSON) document data on mobile gateway devices. The proposed spatial index method can efficiently process location-based queries for medical signal monitoring. In order to evaluate our index method, we simulated a small system on MongoDB with our proposed index method, which is a document-based NoSQL database system, and evaluated its performance.

Evaluation of Content-Matched Range Monitoring Queries over Moving Objects in Mobile Computing Environments.

A content-matched (CM) range monitoring query over moving objects continually retrieves the moving objects (i) whose non-spatial attribute values are matched to given non-spatial query values; and (ii) that are currently located within a given spatial query range. In this paper, we propose a new query indexing structure, called the group-aware query region tree (GQR-tree) for efficient evaluation of CM range monitoring queries. The primary role of the GQR-tree is to help the server leverage the computational capabilities of moving objects in order to improve the system performance in terms of the wireless communication cost and server workload. Through a series of comprehensive simulations, we verify the superiority of the GQR-tree method over the existing methods.

Sleepwalk: Scalable and Energy-Efficient Processing of Continuous Range Queries for Location-Aware Mobile Computing

Recently, monitoring queries are getting attention for various real-life applications such as safety, security, and personalization services. This work proposes a distributed sensing and monitoring technique (calledSleepwalk) for continuous range queries with energy- and computation-efficient optimizations. In our scheme, each mobile client (MC) is aware of its nearby monitoring queries by leveraging its processing power. The proposed Sleepwalk has three major contributions. First, withpiecewiselinear movement assumption and motion vectorv̅, it can locally preevaluate every possible query result in advancein bulkand sends them to the server at once. We also provide a timestamp-based invalidation technique for efficiently removing failed preevaluated results by computing the smallest valid timestamp. Second, an energy-conserving technique that repeatedly sleeps off MCs whenever possible is proposed by calculating thesafely sleepabletime. Third, we provide a set of localized query optimization techniques for MCs’ local query subset using plane-sweeping, which effectively minimize search space. Extensive experiments indicate that Sleepwalk technique remarkably outperforms existing state-of-the-art techniques in terms of server scalability, communication cost, and energy consumption of MCs.

Monitoring distributed streams using convex decompositions

Emerging large-scale monitoring applications rely on continuous tracking of complex data-analysis queries over collections of massive, physically-distributed data streams. Thus, in addition to the space- and time-efficiency requirements of conventional stream processing (at each remote monitor site), effective solutions also need to guarantee communication efficiency (over the underlying communication network). The complexity of the monitored query adds to the difficulty of the problem --- this is especially true for non-linear queries (e.g., joins), where no obvious solutions exist for distributing the monitored condition across sites. The recently proposed geometric method, based on the notion of covering spheres, offers a generic methodology for splitting an arbitrary (non-linear) global condition into a collection of local site constraints, and has been applied to massive distributed stream-monitoring tasks, achieving state-of-the-art performance. In this paper, we present a far more general geometric approach, based on the convex decomposition of an appropriate subset of the domain of the monitoring query, and formally prove that it is always guaranteed to perform at least as good as the covering spheres method. We analyze our approach and demonstrate its effectiveness for the important case of sketch-based approximate tracking for norm, range-aggregate, and join-aggregate queries , which have numerous applications in streaming data analysis. Experimental results on real-life data streams verify the superiority of our approach in practical settings, showing that it substantially outperforms the covering spheres method.