High-volume Data Processing Research Articles

Advanced Authentication and Authorization Mechanisms in Apache Kafka: Enhancing Security for High-Volume Data Processing Environments

Apache Kafkahas become a cornerstone technology for high-volume, real-time data processing in many organizations. As its adoption grows, so does the need for robust security measures to protect sensitive data and ensure compliance with regulatory requirements. This paper explores advanced authentication and authorization mechanisms in Apache Kafka, focusing on their application in high-volume data processing environments. We analyze current security features, identify challenges, and propose enhancements to strengthen Kafka's security posture. Through case studies and practical examples, we demonstrate the implementation and effectiveness of these advanced security measures in real-world scenarios.

Si-CMOS compatible epsilon-near-zero metamaterial for two-color ultrafast all-optical switching

Driven by the escalating demands of advanced technologies, developing integration strategies has kept pace with the realization of ultrafast components during the past two decades. Ultrafast all-optical switches enabled by artificial materials are considered at the forefront of the next generation of photonic integration for communications and high-volume data processing. Encouraged by these advancements, applications, and interest have increased toward all-optical switches based on epsilon-near-zero (ENZ) materials. However, some all-optical switches lack CMOS compatibility, require high energy activation, and are limited in switching speed and working wavelength. Here, we propose and demonstrate a multilayered ENZ metamaterial utilizing Si-compatible titanium nitride and indium-tin-oxide materials with two effective working wavelengths in the visible and near-infrared spectrum. This device enables switching time down to a few hundred femtoseconds utilizing minimal energy at the corresponding ENZ regions induced by intraband pumping. Our approach can enhance the adaptability of designing ENZ metamaterials for new hybrid integrated photonic components for low-power ultrafast all-optical terahertz modulation.

From fundamentals to frontiers: a review of memristor mechanisms, modeling and emerging applications

The escalating demand for artificial intelligence (AI), the internet of things (IoTs), and energy-efficient high-volume data processing has brought the need for innovative solutions to the forefront.

Self-Supervised Learning for Demand Forecasting Using Sparse and Noisy Social Media Sentiment Data

This research work focuses on the foundational principles of SLO monitoring, architectural considerations for high-volume data processing systems, and advanced techniques for implementing and scaling SLO monitoring solutions. The research includes areas like metric selection, instrumentation techniques, data collection strategies, statistical analysis, and emerging trends in the field. It is a synthesis of current literature and industry practices that presents an organized guide for organizations that want to implement robust SLO monitoring in their data processing infrastructure.

Modeling of Supervised Machine Learning using Mechanism of Quantum Computing.

Mechanism of quantum computing helps to propose several task of machine learning in quantum technology. Quantum computing is enriched with quantum mechanics such as superposition and entanglement for making new standard of computation which will be far different than classical computer. Qubit is sole of quantum technology and help to use quantum mechanism for several tasks. Tasks which are non-computable by classical machine can be solved by quantum technology and these tasks are classically hard to compute and categorised as complex computations. Machine learning on classical models is very well set but it has more computational requirements based on complex and high-volume data processing. Supervised machine learning modelling using quantum computing deals with feature selection, parameter encoding and parameterized circuit formation. This paper highlights on integration of quantum computation and machine learning which will make sense on quantum machine learning modeling. Modelling of quantum parameterized circuit, Quantum feature set design and implementation for sample data is discussed. Supervised machine learning using quantum mechanism such as superposition and entanglement are articulated. Quantum machine learning helps to enhance the various classical machine learning methods for better analysis and prediction using complex measurement.

A Study on the Efficacy of Deep Reinforcement Learning for Intrusion Detection

The world has witnessed a fast-paced digital transformation in the past decade, giving rise to all-connected environments. While the increasingly widespread availability of networks has benefited many aspects of our lives, providing the necessary infrastructure for smart autonomous systems, it has also created a large cyber attack surface. This has made real-time network intrusion detection a significant component of any computerized system. With the advances in computer hardware architectures with fast, high-volume data processing capabilities and the developments in the field of artificial intelligence, deep learning has emerged as a significant aid for achieving accurate intrusion detection, especially for zero-day attacks. In this paper, we propose a deep reinforcement learning-based approach for network intrusion detection and demonstrate its efficacy using two publicly available intrusion detection datasets, namely NSL-KDD and UNSW-NB15. The experiment results suggest that deep reinforcement learning has significant potential to provide effective intrusion detection in the increasingly complex networks of the future.

The Need for Advanced Intelligence in NFV Management and Orchestration

With the constant demand for connectivity at an all-time high, Network Service Providers (NSPs) are required to optimize their networks to cope with rising capital and operational expenditures required to meet the growing connectivity demand. A solution to this challenge was presented through Network Function Virtualization (NFV). As network complexity increases and futuristic networks take shape, NSPs are required to incorporate an increasing amount of operational efficiency into their NFV-enabled networks. One such technique is Machine Learning (ML), which has been applied to various entities in NFV-enabled networks, most notably in the NFV Orchestrator. While traditional ML provides tremendous operational efficiencies, including realtime and high-volume data processing, challenges such as privacy, security, scalability, transferability, and concept drift hinder its widespread implementation. Through the adoption of Advanced Intelligence techniques such as Reinforcement Learning and Federated Learning, NSPs can leverage the benefits of traditional ML while simultaneously addressing the major challenges traditionally associated with it. This work presents the benefits of adopting these advanced techniques, provides a list of potential use cases and research topics, and proposes a bottom-up micro-functionality approach to applying these methods of Advanced Intelligence to NFV Management and Orchestration.

Implementing Effective SLO Monitoring in High-Volume Data Processing Systems

In the development of high-volume data processing systems, effective monitoring of Service Level Objective (SLO) turns out to be a crucial topic. The needs and importance of critical operations require large-scale data processing. Large-scale data processing necessitates maintaining the performance, reliability, and efficiency of such organizations. This research work focuses on the foundational principles of SLO monitoring, architectural considerations for high-volume data processing systems, and advanced techniques for implementing and scaling SLO monitoring solutions. The research includes areas like metric selection, instrumentation techniques, data collection strategies, statistical analysis, and emerging trends in the field. It is a synthesis of current literature and industry practices that presents an organized guide for organizations that want to implement robust SLO monitoring in their data processing infrastructure.

Application of the deep learning for the prediction of rainfall in Southern Taiwan

Precipitation is useful information for assessing vital water resources, agriculture, ecosystems and hydrology. Data-driven model predictions using deep learning algorithms are promising for these purposes. Echo state network (ESN) and Deep Echo state network (DeepESN), referred to as Reservoir Computing (RC), are effective and speedy algorithms to process a large amount of data. In this study, we used the ESN and the DeepESN algorithms to analyze the meteorological hourly data from 2002 to 2014 at the Tainan Observatory in the southern Taiwan. The results show that the correlation coefficient by using the DeepESN was better than that by using the ESN and commercial neuronal network algorithms (Back-propagation network (BPN) and support vector regression (SVR), MATLAB, The MathWorks co.), and the accuracy of predicted rainfall by using the DeepESN can be significantly improved compared with those by using ESN, the BPN and the SVR. In sum, the DeepESN is a trustworthy and good method to predict rainfall; it could be applied to global climate forecasts which need high-volume data processing.

Application of comparative strainer clustering as a novel method of high volume of data clustering to optimal power flow problem

An increase of the volume of data and uncertain parameters, such as renewable generations and load consumption in power systems, causes new challenges in the optimal power flow (OPF) problem. Clustering is one of the fast and accurate tools, and it is able to deal with them by classifying the unpredictable data into categories based on their similarity. Clustering approaches can be directly applied to the data. However, it requires more time, storage, and processing of high volume data. In this paper, a new clustering method called acomparative strainer (CS) is proposed to cover the aforementioned drawbacks. Then, the efficiency of the proposed clustering method of the OPF problem was evaluated based on technical and economic indices, such as voltage magnitude as well as angle and locational marginal price (LMP). The proposed clustering approach is employed on the IEEE57-bus test system to handle the uncertainty of the active and reactive loads. Then, the optimal clusters were determined. Finally, the results were compared with other clustering algorithms, such as raw data clustering, the famous method of dimension reduction, and principal component analysis (PCA) clustering, in order to reveal that the proposed method is more accurate and faster than others and leads to precise outcomes when it is utilized for the OPF problem.

Internet of Things, big data and the economics of networked vehicles

The evolution of special purpose standards for networked vehicles towards IP based communication networks and the high-volume data processing requirements call for the introduction of a new concept – the big data virtual network. The virtual network operator, which may be integrated with the operators of networked vehicle applications, combines sensor and camera devices with big data, either using large data centers (cloud computing) or local processing (fog computing), and complementary QoS (Quality of Service) differentiated bandwidth capacities from the all-IP network. The growing importance of camera-based sensors as well as geo-awareness requirements create a need for more pronounced QoS differentiated bandwidth as well as big data processing efforts. In particular, combining QoS differentiated all-IP bandwidth capacities with local data processing within the edge cloud constitutes a new task for big data virtual network service providers, enabling deterministic guarantees for ultra-low latencies required for networked fully automated (driverless) vehicles.

EVALUATION OF OPENMP OPTIMIZATION IN HETEROGENEOUS COMPUTING MODE BY CODE OFFLOADING ON INTEL XEON PHI CO-PROCESSOR

As the computing needs are increasing, the utilization of compute powers of multi-processors and co-processors together is an active area of research. This paradigm is known as Heterogeneous computing. With the increasing data sizes and complexity of algorithms, and dead lock reached in processor clock frequency due to power constraints, multi core and many core CPUs and GPUs have been used for parallel computing. This has become an inevitable approach for high volume data processing such as image processing. In the process of evaluation the authors had earlier tested on the Intel Quad Core i7 (8 threads) processor and dual Intel Xeon 12 core (48 threads) CPUs by optimizing an image processing code on K-Means clustering in multispectral feature space using remote sensing data[1]. The maximum speedup 5x is achieved on Intel i7 core CPU and speedup of 13x is achieved on Intel Xeon CPU by invoking dynamic scheduling when number of threads deployed are large. In continuation of the earlier studies, the present study analyses the Intel Xeon phi coprocessor 7120P(device) HPC accelerator performance with processor base frequency of 1.24 GHz along with OpenMP Parallel computing model. It is observed that the offloading will not give best result with small data size. To get the full benefits of offloading on Intel Xeon phi coprocessor, computation offloading with OpenMP utilizing both processor and coprocessor gains accelerations and increases the performance if communication overhead is less than the computation times which is highly application dependent.

Métodos Kjeldahl, Elementar e o Potencial da Espectroscopia do Infravermelho Próximo para Determinação de Nitrogênio em Solos da Bacia do Acre

The Acre basin comprises a distinct geological environment with different environments in relation to Amazon biome, which results in soils with mineralogical and chemical properties different of other soils of this biome. For these environment there are no evaluations regarding the accuracy and precision of analysis methods for nitrogen: Kjeldahl, dry combustion and NIR spectroscopy. The objective of the research was to assess and compare two conventional methods and near infrared spectroscopy to determine the nitrogen content in the Acre basin soil. To evaluate these three methods, were used 176 representative soil samples of Acre basin. To evaluate the applicability of NIR was adopted as reference the levels determined by the elementary method, the NIR spectra are used in the range 1100-2500 nm and 2.0 nm resolution by diffuse reflectance. The Kjeldahl and dry combustion methods showed no differences in determining the nitrogen content in the soil, and both methods showed coefficient of variation of less than 5%, indicating the same accuracy of the two methods for determining the nitrogen contents in the Acre Basin soil. Adopting the elemental analysis to calibrate and validate the estimates of nitrogen in the soil by near infrared spectroscopy, there was obtained the standard error of calibration 0.28 g kg -1 and coefficient of determination of 84%; and the standard validation error model was 0.35 g kg -1 with the coefficient of determination of 77%. This has been interpreted as NIR spectroscopy has potential to be used in the quantification of the nitrogen content in the soil in research that require high-volume data processing. Keywords : soil testing; elemental analysis; Amazon; green chemistry.

Performance Measurement of Complex Event Platforms

The aim of this paper is to find and compare existing solutions of complex event processing platforms (CEP). CEP platforms generally serve for processing and/or predicting of high frequency data. We intend to use CEP platform for processing of complex time series and integrate a solution for newly proposed method of decision making. The decision making process will be described by formal grammar. As there are lots of CEP solutions we will take the following characteristics under consideration - the processing in real time, possibility of processing of high volume data from multiple sources, platform independence, platform allowing integration with user solution and open license. At first we will talk about existing CEP tools and their specific way of use in praxis. Then we will mention the design of method for formalization of business rules used for decision making. Afterwards, we focus on two platforms which seem to be the best fit for integration of our solution and we will list the main pros and cons of each approach. Next part is devoted to benchmark platforms for CEP. Final part is devoted to experimental measurements of platform with integrated method for decision support.

Method based on bioinspired sample improves autofocusing performances

In order to solve the issue between fast autofocusing speed and high volume data processing, we propose a bioinspired sampling method based on a retina-like structure. We develop retina-like models and analyze the division of sampling structure. The optimal retina-like sample is obtained by analyzing two key parameters (sectors and radius of blind area) of the retina-like structure through experiments. Under the typical autofocus functions, including Vollath–4, Laplacian, Tenengrad, spatial frequency, and sum-modified-Laplacian (SML), we carry out comparative experiments of computation time based on the retina-like sample and a traditional uniform sample. The results show that the retina-like sample is suitable for those autofocus functions. Based on the autofocus function of SML, the average time of uniform sample decreases from 3.5 to 2.1 s for the retina-like sample.

A 6.4-Gb/s Near-Ground Single-Ended Transceiver for Dual-Rank DIMM Memory Interface Systems

The emergence of cloud computing has driven the demand for high-density, low-latency and high-speed memory interfaces. For such applications the use of multiple dual-inline memory modules (DIMMs) with multiple ranks enables time-efficient processing of high-volume data. However, the deterioration of the channel frequency response due to the presence of DIMM connectors and multiple ranks makes it challenging to perform low-power read and write (R/W) operations at high-speed. Recent works have demonstrated the use of near-ground signaling (NGS) for low-power operation and signal-integrity enhancement with the aid of transmit supply regulation. In contrast to their differential nature, this paper introduces a single-ended NGS transceiver that achieves 6.4Gb/s R/W operations with the aid of low-power equalization and in-situ reference-voltage calibration over a 3.5 total FR4 PCB routing with more than 25mm of package traces in a dual-rank DIMM memory interface system.

PoGO: Prediction of Gene Ontology terms for fungal proteins

BackgroundAutomated protein function prediction methods are the only practical approach for assigning functions to genes obtained from model organisms. Many of the previously reported function annotation methods are of limited utility for fungal protein annotation. They are often trained only to one species, are not available for high-volume data processing, or require the use of data derived by experiments such as microarray analysis. To meet the increasing need for high throughput, automated annotation of fungal genomes, we have developed a tool for annotating fungal protein sequences with terms from the Gene Ontology.ResultsWe describe a classifier called PoGO (Prediction of Gene Ontology terms) that uses statistical pattern recognition methods to assign Gene Ontology (GO) terms to proteins from filamentous fungi. PoGO is organized as a meta-classifier in which each evidence source (sequence similarity, protein domains, protein structure and biochemical properties) is used to train independent base-level classifiers. The outputs of the base classifiers are used to train a meta-classifier, which provides the final assignment of GO terms. An independent classifier is trained for each GO term, making the system amenable to updating, without having to re-train the whole system. The resulting system is robust. It provides better accuracy and can assign GO terms to a higher percentage of unannotated protein sequences than other methods that we tested.ConclusionsOur annotation system overcomes many of the shortcomings that we found in other methods. We also provide a web server where users can submit protein sequences to be annotated.

The 8 requirements of real-time stream processing

Applications that require real-time processing of high-volume data steams are pushing the limits of traditional data processing infrastructures. These stream-based applications include market feed processing and electronic trading on Wall Street, network and infrastructure monitoring, fraud detection, and command and control in military environments. Furthermore, as the "sea change" caused by cheap micro-sensor technology takes hold, we expect to see everything of material significance on the planet get "sensor-tagged" and report its state or location in real time. This sensorization of the real world will lead to a "green field" of novel monitoring and control applications with high-volume and low-latency processing requirements.Recently, several technologies have emerged---including off-the-shelf stream processing engines---specifically to address the challenges of processing high-volume, real-time data without requiring the use of custom code. At the same time, some existing software technologies, such as main memory DBMSs and rule engines, are also being "repurposed" by marketing departments to address these applications.In this paper, we outline eight requirements that a system software should meet to excel at a variety of real-time stream processing applications. Our goal is to provide high-level guidance to information technologists so that they will know what to look for when evaluation alternative stream processing solutions. As such, this paper serves a purpose comparable to the requirements papers in relational DBMSs and on-line analytical processing. We also briefly review alternative system software technologies in the context of our requirements.The paper attempts to be vendor neutral, so no specific commercial products are mentioned.

A plug and play approach to data acquisition

Backplane buses are becoming a legacy for high-rate, high-volume data processing applications. Higher efficiency at lower cost is offered by the PCI bus technology, compared to crate-embedded processors. Becoming part of the plug and play domain of the host's operating system, no additional data transfer protocols are needed. We have combined PCI technology with high-density field-programmable gate array (FPGA) logic and common mezzanine standards on a flexible PCI card. First applications cover readout controllers for legacy bus protocols, high-speed link I/O and fast analog input data conversion. An FPGA with embedded PCI master/target core serves as a programmable interface between the PCI bus, mezzanine cards, and a local SDRAM. Adapter mezzanine cards, implemented according to the IEEE P1386 or similar common standards, are used for voltage level conversion, trigger interfacing or preprocessing. The application-dependent controller functions as well as SDRAM and PCI interfacing are handled by FPGA logic. A Linux driver was developed to achieve high bandwidth via CPU-initiated transfers. Control software for Windows and an interface for LabView target control and monitoring applications via graphical interfaces. First experience and applications are reported.

An Experimental Study to Analyze and Optimize Hartree-Fock Application's I/O with Passion

Many scientific applications tend to perform high-volume data storage, data retrieval, and data processing, all of which demand high performance from the I/O subsystem. The focus and contribution of this work is to study the I/O behavior of the Hartree-Fock (HF) method using PAS SION. HF's I/O phases can contribute up to 62.34% of the total execution time. The authors reduce the execution time and I/O time up to 54% and 6%, respectively, of that of the original case through PASSION and its optimiza tions. Additionally, the authors categorize the factors that affect the I/O performance of HF into key application- related parameters and key system-related parameters. Based on extensive empirical results and within the ex perimental space presented in this paper, the authors orderthe parameters according to the their impact on HF's I/O performance as follows: efficient interface, prefetching, buffering, number of I/O nodes, striping factor, and striping unit. The authors conclude that application-related factors have a more significant effect on HF's I/O performance than the system-related factors within the experimental space presented in this paper.