Streaming Engine Research Articles

Using Polars to Improve String Similarity Performance in Python

IntroductionString similarity is central to textual record linkage, and is often calculated with Levenshtein distance or Jaro-Winkler distance. In polars-strsim, we leverage the Polars DataFrame interface to surpass all existing Python libraries in computing these distances. Polars-strsim is especially useful for larger-than-memory datasets. Objectives and ApproachPolars is a library built to analyse and manipulate tabular data. It is written in Rust, but can be called from Python for ease of use. Polars-strsim implements the Levenshtein and Jaro-Winkler algorithms as a Polars extension. We compare performance against nine other Python libraries on three million pairs of first names. ResultsWhen computing Levenshtein/Jaro-Winkler distance for this benchmark, polars-strsim is respectively 5x/4x faster than the current fastest Python alternative and 47x/35x faster than the median. The Polars streaming engine can be seamlessly used to manipulate larger than memory datasets. One practical application of this in record linkage is to find all pairs of records in a dataset where the Levenshtein distance computed across one (or more) fields is below a certain threshold (e.g., during blocking). With polars-strsim, this calculation can be performed in Python with CPU parallelism even if the input and/or output doesn’t fit in memory. Conclusions/ImplicationsThis work introduces a powerful library for record linkage practitioners. Polars-strsim provides the convenience of Python, the performance of a strongly/statically typed compiled language, and the full power of the Polars streaming engine when computing string similarity, and can be updated to implement additional similarity algorithms.

Conservation Challenges Imposed by Evolutionary History and Habitat Suitability Shifts of Endangered Freshwater Mussels under a Global Climate Change Scenario

Climate change and associated shifts in temperature and precipitation patterns have become an increasing concern as drivers of ongoing biodiversity loss. The Mediterranean region is particularly vulnerable, being both a biodiversity hotspot and a region very prone to desertification. Freshwater mussels are amongst the most threatened invertebrate taxa worldwide. Unio tumidiformis is an endemic and endangered species restricted to the southern Iberian Peninsula, living in temporary Mediterranean-type streams. Freshwater mussels need a fish host for successful larval transformation, meaning U. tumidiformis must belong to the genus Squalius. The main objective of this study was to evaluate the vulnerability of U. tumidiformis to climate change, by studying its population genetics and evolutionary history, its current and future habitat suitability, and that of its hosts. Genetic population structure and diversity were assessed using Single-Nucleotide Polymorphisms through Genotyping by Sequencing and used to infer species evolutionary history. The species potential distribution was modeled using an ensemble forecasting approach, and future shifts in habitat suitability were assessed with the projected climate data layers from Worldclim. Most populations showed extreme genetic differentiation (Fst up to 0.745), even from close neighboring ones. Upper Guadiana populations were more diverse and less differentiated. We hypothesize that U. tumidiformis originated in Upper Guadiana and followed the same colonization routes as their hosts with numerous founder effects and bottlenecks. Our results also predicted a reduction of 99% of climatically suitable areas for U. tumidiformis in the Iberian Peninsula until 2040. For the fish hosts, a maximum 42% reduction in suitable areas was estimated throughout the century, with remaining adequate habitats in the north. Our results suggest that difficult conservation options are necessary, prioritizing the preservation of populations, translocations to the northern area of its historical range and stream engineering to increase resilience to droughts.

SMSE: A serverless platform for multimedia cloud systems

SummaryAlong with the rise of domain‐specific computing (ASICs hardware) and domain‐specific programming languages, we envision that the next step is the emergence of domain‐specific cloud platforms. Considering multimedia streaming as one of the most trendy applications in the IT industry, the goal of this study is to develop serverless multimedia streaming engine (SMSE), the first domain‐specific serverless platform for multimedia streaming. SMSE democratizes multimedia service development via enabling content providers (or even end‐users) to rapidly develop their desired functionalities on their multimedia contents. Upon developing SMSE, the next goal of this study is to deal with its efficiency challenges and develop a function container provisioning method that can efficiently utilize cloud resources and improve the users' quality of service. In particular, we develop a dynamic method that provisions durable or ephemeral containers depending on the spatiotemporal and data‐dependency characteristics of the functions. Evaluating the prototype implementation of SMSE under real‐world settings demonstrates its capability to reduce both the containerization overhead, and the makespan time of serving multimedia processing functions (by up to 30%) in compare to the function provision methods that are being used in the general‐purpose serverless cloud systems.

Developing STREAM (Science, Technology, Religiosity, Engineering, Art, and Mathematics) Based Tiktok Video on Buffer Solution Lesson

Developing STREAM (Science, Technology, Religiosity, Engineering, Art, and Mathematics) Based Tiktok Video on Buffer Solution Lesson

Retracted: A Novel Intelligent Hybrid Optimized Analytics and Streaming Engine for Medical Big Data.

[This retracts the article DOI: 10.1155/2022/7120983.].

Inovasi Pembelajaran Science, Technology, Religion, Engineering, Art, and Mathematics Pada Mata Pelajaran Pendidikan Agama Islam

The development and progress of the times always demands change and innovation, including in the education and learning sector of Islamic Religious Education. The purpose of this study is to discuss theoretically and practical findings about integrated learning, namely innovation learning STREAM (Science, Technology, Religion, Engineering, Art, and Mathematics) in Islamic Religious Education subjects. This type of research is qualitative with analytical descriptive approach, uncovering findings and literature study. The findings show that Islamic Religious Education learning at SMAN 5 Bandung in class XII on the Basic Competence of Marriage in Islam, has been carried out using the STREAM approach in four meetings, and collaborates with the subjects of Cultural Arts, Crafts and Entrepreneurship, Sundanese Language, Civics, Biology, Economics-Accounting, and Mathematics. The learning model used is Project Based Learning (PjBL), Role Playing, Time Line of the River of Life and Presentation-Discussion. The products produced are in the form of power point teaching materials, photos and videos of marriage practice simulations with walimah (reception), as well as graphic info on time line planning for life targets until the estimate returns to Allah SWT.

Visual Language Translation Analysis and Scenic Beauty Estimation of Mountain Stream Facilities

The aesthetic design of mountain stream facilities has not received much attention in Taiwan. In recent years, in addition to safety concerns, the focus has increasingly been on landscape assessment or the sense of integration with the environment. This research is the first attempt to use visual language translation in qualitative research analysis for the landscape assessment of mountain stream facilities. This method is different from a traditional qualitative narrative analysis. It also addresses the shortcomings of previous quantitative analysis methods, in which the topic discussions are too limited. First, mountain stream engineering projects are selected as the research objects. This study uses questionnaire analysis and on-site surveys to summarize the elements, representations, and perceptions of the mountain stream facilities of the subjects concerned to examine their preferences for the visual system. Furthermore, we also employ the scenic beauty estimation (SBE) method for a comparison between the qualitative and quantitative analyses. This study proposes a new method using visual language translation and SBE that combines the features of qualitative research and quantitative analysis. However, the potential limitations include an inability to have a large sample number and the biases caused by the cultural, regional, or personal characteristics of the subjects.

Dr. Gadgal in the Midst of the Industrial Evolution in India

The first Industrial Revolution began in England in about 1750-1760 that lasted to sometime between 1820 and 1840. It is one of the most distinguished turning points in human history. During this period human and animal labor technology transform into machinery, such as the stream engine, the spinning jenny, and coke something pudding and rolling processes for making iron, etc. Industrial Revolution is renewed for global economic growth, increase in production and consumption of common people. The system of transportation communication through canals, road and rails had improved. Also, banking and other financial systems improved to run the industries and business firms smoothly. Child and infant mortality rate decreased and fertility rate increased. As a result, population growth had dramatically changed. On the other hand, women and child unhygienic condition. Factory workers have to work sixteen hours in the day merely to save the family from starvation. Industrial Revolution created a wide gap between the rich and the poor. An attempt has taken here to describe the various effects of Industrial Revolution.

Subscribing to big data at scale

Today, data is being actively generated by a variety of devices, services, and applications. Such data is important not only for the information that it contains, but also for its relationships to other data and to interested users. Most existing Big Data systems focus on passively answering queries from users, rather than actively collecting data, processing it, and serving it to users. To satisfy both passive and active requests at scale, application developers need either to heavily customize an existing passive Big Data system or to glue one together with systems like Streaming Engines and Pub-sub services. Either choice requires significant effort and incurs additional overhead. In this paper, we present the BAD (Big Active Data) system as an end-to-end, out-of-the-box solution for this challenge. It is designed to preserve the merits of passive Big Data systems and introduces new features for actively serving Big Data to users at scale. We show the design and implementation of the BAD system, demonstrate how BAD facilitates providing both passive and active data services, investigate the BAD system’s performance at scale, and illustrate the complexities that would result from instead providing BAD-like services with a “glued” system.

A Novel Intelligent Hybrid Optimized Analytics and Streaming Engine for Medical Big Data.

Medical data processing is exponentially increasing day by day due to the frequent demand for many applications. Healthcare data is one such field, which is dynamically growing day by day. In today's scenario, an enormous amount of sensing devices and data collection units have been employed to generate and collect medical data all over the world. These healthcare devices will result in big real-time data streams. Hence, healthcare-based big data analytics and monitoring have gained hawk-eye importance but needs improvisation. Recently, machine and deep learning algorithms have gained importance to analyze huge amounts of medical data, extract the information, and even predict the future insights of diseases and also cope with the huge volume of data. But applying the learning models to handle big/medical data streams remains to be a challenge among the researchers. This paper proposes the novel deep learning electronic record search engine algorithm (ERSEA) along with firefly optimized long short-term memory (LSTM) model for better data analytics and monitoring. The experimentations have been carried out using Apache Spark using the different medical respiratory data. Finally, the proposed framework results are contrasted with existing models. It shows the accuracy, sensitivity, and specificity like 94%, 93.5%, and 94% for less than 5 GB dataset, and also, more than 5 GB it provides 94%, 92%, and 93% to prove the extraordinary performance of the proposed framework.

Harnessing the Potential of Function-Reuse in Multimedia Cloud Systems

Cloud-based computing systems can get oversubscribed due to the budget constraints of their users or limitations in certain resource types. The oversubscription can, in turn, degrade the users perceived Quality of Service (QoS). The approach we investigate to mitigate both the oversubscription and the incurred cost is based on smart reusing of the computation needed to process the service requests (i.e., tasks). We propose a reusing paradigm for the tasks that are waiting for execution. This paradigm can be particularly impactful in serverless platforms where multiple users can request similar services simultaneously. Our motivation is a multimedia streaming engine that processes the media segments in an on-demand manner. We propose a mechanism to identify various types of “mergeable” tasks and aggregate them to improve the QoS and mitigate the incurred cost. We develop novel approaches to determine when and how to perform task aggregation such that the QoS of other tasks is not affected. Evaluation results show that the proposed mechanism can improve the QoS by significantly reducing the percentage of tasks missing their deadlines and reduce the overall time (and subsequently the incurred cost) of utilizing cloud services by more than 9 percent.

Visual Harmony of Engineering Structures in a Mountain Stream

This study uses the cognitive factor of “visual harmony” to assess the visual quality of stream engineering in a mountainous region. Images of engineering structures such as revetments and submerged dams in the mountain streams of Taiwan were collected. Three image groups with different structures invaded by vegetation were used for a questionnaire survey, which yielded 154 valid samples. We used statistical analysis to develop a model of visual harmony H with respect to the percentage of visible greenery GR, that is, the perceived curve of vegetation change. A comparison of our data with the literature determined the upper and lower bound curves of the relationship between H and GR. We found that the physical elements of “softscape” and “hardscape”—namely, percentage of visible water WR, visible structure IR, and visible natural material on the structure NR—affected this relationship. Results show that H is equivalent to visual preference P, and both can be improved by better green visibility (increasing GR and GR < 50%), avoiding low water visibility (WR < 10%), or increasing the amount of visible natural material (NR > 0.9). High visibility of the structures (IR > 0.3) may decrease H and P. We ultimately propose a visual harmony or preference model concerning a combined physical indicator that comprises GR, WR, IR and NR. Results of this study could be helpful to improve or access the aesthetics of stream engineering design.

StreamQL: a query language for processing streaming time series

Real-time data analysis applications increasingly rely on complex streaming computations over time-series data. We propose StreamQL, a language that facilitates the high-level specification of complex analyses over streaming time series. StreamQL is designed as an algebra of stream transformations and provides a collection of combinators for composing them. It integrates three language-based approaches for data stream processing: relational queries, dataflow composition, and temporal formalisms. The relational constructs are useful for specifying simple transformations, aggregations, and the partitioning of data into key-based groups or windows. The dataflow abstractions enable the modular description of a computation as a pipeline of stages or, more generally, as a directed graph of independent tasks. Finally, temporal constructs can be used to specify complex temporal patterns and time-varying computations. These constructs can be composed freely to describe complex streaming computations. We provide a formal denotational semantics for StreamQL using a class of monotone functions over streams. We have implemented StreamQL as a lightweight Java library, which we use to experimentally evaluate our approach. The experiments show that the throughput of our implementation is competitive compared to state-of-the-art streaming engines such as RxJava and Reactor.

BAD to the bone: Big Active Data at its core

Virtually all of today's Big Data systems are passive in nature, responding to queries posted by their users. Instead, we are working to shift Big Data platforms from passive to active. In our view, a Big Active Data (BAD) system should continuously and reliably capture Big Data while enabling timely and automatic delivery of relevant information to a large pool of interested users, as well as supporting retrospective analyses of historical information. While various scalable streaming query engines have been created, their active behavior is limited to a (relatively) small window of the incoming data. To this end we have created a BAD platform that combines ideas and capabilities from both Big Data and Active Data (e.g., Publish/Subscribe, Streaming Engines). It supports complex subscriptions that consider not only newly arrived items but also their relationships to past, stored data. Further, it can provide actionable notifications by enriching the subscription results with other useful data. Our platform extends an existing open-source Big Data Management System, Apache AsterixDB, with an active toolkit. The toolkit contains features to rapidly ingest semistructured data, share execution pipelines among users, manage scaled user data subscriptions, and actively monitor the state of the data to produce individualized information for each user. This paper describes the features and design of our current BAD data platform and demonstrates its ability to scale without sacrificing query capabilities or result individualization.

Cultivating Disciplinary Expectations for Engineering Education Research in Canada

Engineering education research in Canada is, in many ways, still establishing itself as a discipline, yet much is already being asked of it. The diversity of career pathways for engineering graduates, along with the increased complexity of global engineering challenges and the ramifications of technological opportunities, is raising important questions about current engineering education foci and practices. This paper brings together issues surfaced through the process of compiling a Special Theme on engineering education research in Canada, positioning the collection of papers in terms of the challenges and opportunities faced by this maturing Canadian research community. We seek to reconcile how this research can incorporate the theoretical and methodological depth valued in the social sciences while honoring the focus on effective instructional practices that has become an important strand of inquiry in engineering education. Challenges to the growth of engineering education research, such as funding and graduate student inclusion, are also discussed. In addition, we explore the emergence of teaching stream engineering faculty and problematize the role of research and scholarship in their career trajectories. Finally, we call for more strategic thinking to guide the evolution of engineering education research in Canada. We hope that, collectively, this editorial and the other four papers in this Special Theme will help to stimulate discussion about how we define and recognize excellence in the domain of engineering education research in Canada.

Modeling Access Control on Streaming Data in Apache Storm

Abstract Streaming analytics on real-time data is very important in the IOT world since analytics, decisions and operations have to be done in real-time for practical applicability. In such scenarios, one key consideration is the security of the real-time stream. Stream security as in other network security can be modeled using the CIA (Confidentiality/Integrity/Availability) model [1]. However, most practical implementations just take care of the first two aspects viz Confidentiality and Integrity using standard techniques such as encryption and signatures. In this paper, we introduce an access control mechanism on the stream that annotates the stream with additional security metadata. This metadata can be used to allow/deny access to elements in the stream and also protect the privacy of the data. We demonstrate a work in progress implementation of the access control mechanism on the popular streaming engine Apache Storm [2] and demonstrate how access control can be modeled and implemented on streaming data.

Intrusion Detection based on Graph oriented Big Data Analytics

Abstract Intrusion detection has been the subject of numerous studies in industry and academia. Still, Cybersecurity analysts always want greater precision and global threat analysis to secure their systems in cyberspace. To improve the intrusion detection system, the visualization of the security events in the form of graphs and diagrams is significant to improve the accuracy of alerts. In this paper, we propose an approach of an IDS based on cloud computing, big data technique and, using a machine learning graph algorithm which can detect in real-time different attacks as early as possible. We use the MAWILab intrusion detection dataset. We choose Microsoft Azure as a unified cloud environment to load our dataset on Azure blob storage. We implement the k2 algorithm, which is a graphical machine learning algorithm to classify attacks. Our system showed a great performance due to the graphical machine learning algorithm and Apache Spark structured streaming engine.

Benchmarking big data systems: A survey

With the enormous growth on the availability and usage of Big Data storage and processing systems, it has become essential to assess the various performance aspects of these systems so that we can carefully understand their strong and weak aspects. In practice, currently, when an individual/enterprise aims to develop a Big Data storage and processing solution for harnessing the knowledge inside their data, they will get challenged by the availability of several frameworks from which they need to select. This is a challenging task which needs to directed by with good knowledge about various perspectives of such systems. Additionally, the choice normally vary from one scenario to another according to the essential needs of the application. In practice, there is no single benchmark study which can cover the different types of big data processing requirements, systems, application scenarios and metrics. Several benchmarks and benchmarking studies have been developed where each study focuses on some representative type of frameworks and only consider some aspects to cover. In this article, we provide a comprehensive survey and analysis of the state-of-the-art of benchmarking the different types of big data systems (e.g., NoSQL databases, Big SQL engines, Big Streaming engines, Big Graph Processing engines, Big Machine/Deep Learning engines). Additionally, we highlight some of the significant open challenges and missing requirements of current benchmarks of big data systems with suggestions of directions for future extensions and improvements.

StreamAligner: a streaming based sequence aligner on Apache Spark

Next-Generation Sequencing technologies are generating a huge amount of genetic data that need to be mapped and analyzed. Single machine sequence alignment tools are becoming incapable or inefficient in keeping track of the same. Therefore, distributed computing platforms based on MapReduce paradigm, which uses thousands of commodity machines to process and analyze huge datasets, are emerging as the best solution for growing genomics data. A lot of MapReduce-based sequence alignment tools like CloudBurst, CloudAligner, Halvade, and SparkBWA are proposed by various researchers in recent few years. These sequence aligners are very fast and efficient. These sequence aligners are capable of aligning billions of reads (stored as fasta or fastq files) on reference genome in few minutes. In the current era of fastly growing technology, analyzing huge genome data fast is not enough. We need to analyze data in real time to automate alignment process. Therefore, we propose a MapReduce-based sequence alignment tool StreamAligner which is implemented on Spark streaming engine. StreamAligner can align stream of reads on reference genome in real time. Therefore, it can be used to automate sequencing and alignment process. It uses suffix array index for read alignment which is generated using distributed index generation algorithm. Due to distributed index generation algorithm, index generation time is very less. It needs to upload index only once when StreamAligner is launched. After that index stays in Spark memory and can be used for an unlimited times without reloading. Whereas, current state-of-the-art sequence aligner either generate (hash index based) or load (sorted index based) index for every task. Hence, StreamAligner reduces time to generate or load index for every task. A working and tested implementation of streamAligner is available on GitHub for download and use. We tested the effectiveness, efficiency, and scalability of our aligner for various standard and real-life datasets.

REAL-TIME EARTHQUAKE MONITORING WITH SPATIO-TEMPORAL FIELDS

Abstract. With live streaming sensors and sensor networks, increasingly large numbers of individual sensors are deployed in physical space. Sensor data streams are a fundamentally novel mechanism to deliver observations to information systems. They enable us to represent spatio-temporal continuous phenomena such as radiation accidents, toxic plumes, or earthquakes almost as instantaneously as they happen in the real world. Sensor data streams discretely sample an earthquake, while the earthquake is continuous over space and time. Programmers attempting to integrate many streams to analyze earthquake activity and scope need to write code to integrate potentially very large sets of asynchronously sampled, concurrent streams in tedious application code. In previous work, we proposed the field stream data model (Liang et al., 2016) for data stream engines. Abstracting the stream of an individual sensor as a temporal field, the field represents the Earth’s movement at the sensor position as continuous. This simplifies analysis across many sensors significantly. In this paper, we undertake a feasibility study of using the field stream model and the open source Data Stream Engine (DSE) Apache Spark(Apache Spark, 2017) to implement a real-time earthquake event detection with a subset of the 250 GPS sensor data streams of the Southern California Integrated GPS Network (SCIGN). The field-based real-time stream queries compute maximum displacement values over the latest query window of each stream, and related spatially neighboring streams to identify earthquake events and their extent. Further, we correlated the detected events with an USGS earthquake event feed. The query results are visualized in real-time.