Big Data Processing Research Articles

Efficient Real-Time Information Interaction and Discrimination: Exploration and Application of IT System Algorithms Based on Big Data Processing Technology

Efficient Real-Time Information Interaction and Discrimination: Exploration and Application of IT System Algorithms Based on Big Data Processing Technology

An Adaptation of the Input Doubling Method for Solving Classification Tasks in Case of Small Data Processing

In the era of big data processing, numerous techniques prove valuable for analyzing large-scale datasets. However, the efficient processing of small data is equally crucial, particularly in the context of intelligent mobile systems. This importance stems from resource constraints, real-time requirements, privacy concerns, and the emergence of edge computing paradigms. In this paper, we adapt an existing input doubling method to address classification tasks when facing limited training datasets. The proposed solution is implemented by modifying the procedure for aggregating predicted values, based on the principles of majority voting. This approach differs significantly from the existing method, which relies on averaging prediction results. A flowchart of the adapted method is presented, detailing the procedures for training and application based on the use of a plurality voting scheme. Modeling is conducted on a short, imbalanced dataset for solving express diagnostic tasks in the field of transplantology. The results demonstrate the high effectiveness of employing the adapted input doubling method for classification tasks in transplantology, where acquiring a substantial training dataset for analysis is challenging.

Integration of building information modeling and big data processing technologies

Integration of building information modeling and big data processing technologies

The Construction of Higher Vocational English Language Bank Based on Neurolinguistic Knowledge

Abstract With the advent of artificial intelligence and big data technologies, traditional approaches to teaching English in higher vocational education are increasingly inadequate for contemporary pedagogical demands. This study involved the collection and analysis of an extensive corpus of English learning materials facilitated by the use of neural network models to process this data. Consequently, we established an English language library tailored to the authentic language use and learning needs of higher vocational students. A substantial array of textual materials pertinent to higher vocational English instruction, including textbooks, online articles, and practical dialogues, was amassed using web crawler technology. These texts underwent natural language processing that included tokenization and the elimination of stop words. To develop a language model, we deployed a Transformer-based neural network for deep learning analysis of the processed data. Our findings indicate that the constructed language bank excels over traditional higher vocational English teaching resources in terms of vocabulary breadth, grammatical diversity, and practical utility. The language bank demonstrates notable benefits in analyzing vocabulary usage biases and enhancing students’ pragmatic language skills. This research substantiates the feasibility and effectiveness of constructing a language bank for higher vocational English utilizing neurolinguistic methodologies. The integration of advanced language modeling and big data processing technologies can significantly elevate the caliber of educational resources and instructional efficacy

In-Storage Computing for Hadoop MapReduce Framework: Challenges and Possibilities

Solid State Drives (SSDs) were initially developed as faster storage devices intended to replace conventional magnetic Hard Disk Drives (HDDs). However, high computational capabilities enable SSDs to be computing nodes, not just faster storage devices. Such capability is generally called ”In-Storage Computing (ISC)”. Today’s Hadoop MapReduce framework has become a de facto standard for big data processing. This paper explores In-Storage Computing challenges and opportunities for the Hadoop MapReduce framework. For this, we integrate a Hadoop MapReduce system with ISC SSD devices that implement the Hadoop Mapper inside real SSD firmware. This offloads Map tasks from the host MapReduce system to the ISC SSDs. We additionally optimize the host Hadoop system to make the best use of our proposed ISC Hadoop system. Experimental results demonstrate our ISC Hadoop MapReduce system achieves a remarkable performance gain (2.3 faster) as well as significant energy savings (11.5 lower) compared to a typical Hadoop MapReduce system. Further, the experiment suggests such ISC augmented systems can provide a very promising computing model in terms of a system scalability.

Developing a Data Pipeline Solution for Big Data Processing

Developing a Data Pipeline Solution for Big Data Processing

Optimization and Upgrading of Big Data Processing Techniques in High Performance Computing Environments

Optimization and Upgrading of Big Data Processing Techniques in High Performance Computing Environments

Developing a data pipeline solution for big data processing

Developing a data pipeline solution for big data processing

Design of Accounting Information Big Data Analysis Platform Based on Cloud Computing

The advent of the era of big data provides unprecedented opportunities for the transformation and innovation of corporate accounting, but also poses severe challenges to its development. Taking Company M as an example, this paper analyzes the accounting transformation under the current big data environment, and raises many new issues, such as the security of accounting information and the authenticity of information, and the transformation of accounting information data processing mode. At the same time, enterprises are also facing many opportunities, such as: the impact on accounting confirmation, enterprise financial forecasting, decision-making and cost control, and the management of enterprise accounting electronic documents in the cloud computing environment and big data environment. This paper intends to study the accounting information system under the environment of cloud computing and big data, aiming to improve the security and reliability of enterprise accounting information. This article constructed an accounting information big data analysis platform. Through experimental testing, it was found that before using the platform, the cost of hardware investment and system maintenance in 2019 was 19,803 yuan, while the cost after using the platform was 13,256 yuan. Studies have shown that using big data processing technology to extract useful information from huge financial and accounting databases can help enhance the core competitiveness of enterprises.

Optimizing Healthcare Big Data Processing with Containerized PySpark and Parallel Computing: A Study on ETL Pipeline Efficiency

Optimizing Healthcare Big Data Processing with Containerized PySpark and Parallel Computing: A Study on ETL Pipeline Efficiency

Renewable Energy Integration and Carbon Neutrality Green Technology Innovation based on Intelligent Microgrid Control

This paper focuses on the topic of renewable energy integration and carbon neutrality green technology innovation based on intelligent microgrid and focuses on the design and optimization of intelligent microgrid control to improve the utilization efficiency of renewable energy and achieve the goal of carbon neutrality. In this paper, the hardware structure of the microgrid is designed, and the solar, wind and biomass power generation systems are integrated, and the battery energy storage and supercapacitor energy storage systems are optimized. Subsequently, a complete software control system was developed, and based on the application of big data processing and machine learning technology, the intelligent management of the microgrid was realized. Then, based on the research of intelligent control strategy and optimization algorithm, the overall response speed and operation efficiency of the system were improved, and a simulation platform was built to achieve effective simulation experiments, and then the efficiency and stability of the system were verified.

Research on Management and Optimization of Big Data Computing Engine Based on Cloud Native Technology IT Architecture

In the wave of digital transformation, we are in a data-driven era. With the explosive growth of data volume, traditional big data processing platforms are facing more and more challenges, including insufficient computational efficiency and resource utilization. At the same time, cloud native technology has emerged as a key factor in filling this gap due to its flexibility, automated management, and cross platform compatibility. Although cloud native technology has broad prospects, its application in the field of big data is still in its early stages, and there are still many problems that need to be solved. Among them, the unity of cluster management, the collaboration between operation and development, and the efficiency of technology architecture transformation are the main challenges currently faced. To address these issues, we will conduct in-depth research around cloud native technology, big data computing engine technology, and containerized services, exploring and implementing a big data computing engine management and optimization solution based on cloud native technology. By proposing and implementing cloud native deployment methods, containerized construction methods, and a one-stop big data application management and development platform, the aim is to provide new ideas and methods for solving many problems in traditional big data computing engine management, and to contribute to the development and innovation of the big data field.

Retracted: Secure Big Data Processing in Multihoming Networks with AI‐Enabled IoT

Retracted: Secure Big Data Processing in Multihoming Networks with AI‐Enabled IoT

The influence of current results in a event­oriented data collection system

The article discusses synchronous and asynchronous procedures in the implementation of the microservices management algorithm in an adaptive system for processing large data flows when collecting information on the main event-oriented approach in the implementation of the architecture of a software system that processes information in real-time. This approach is important when processing large volumes of data from heterogeneous information sources, especially when the task is to minimize the total processing time of large data streams and reduce the total number of calls to data sources. The proposed approach makes it possible, based on its adaptability, to manage the selection of the composition and the number of calls of microservices to the sources, by the events that occur during the collection of information. Thus, it is possible to determine the choice of sources based on the assessment of the effectiveness of obtaining relevant information from them. This is especially important when processing large data flows from heterogeneous information sources when the task is to minimize the total time of collection and processing of large data flows. In turn, this poses the task of minimizing the number of requests to information sources to obtain a sufficient number of data units that are relevant to the search query. The creation of effective big data processing systems requires constant development of approaches to the architecture of building software applications. The event-oriented microservice architecture of the system makes it possible to adapt the operation of the system to the loads on individual microservices and the efficiency of their work by forming and responding to relevant events based on the analysis of relevant events that occur during the collection and initial processing of the received data. Depending on the specific task, it is possible to use both a synchronous and an asynchronous microservice management algorithm. The article provides an analysis of the effectiveness of obtaining relevant data depending on the degree of consideration in the evaluation of results and the formation of events, both the depth of the previous history of the results of requests and the size of their quantitative impact. The use of event-oriented microservice architecture can be especially effective when developing various information and analytical systems that need, according to user requests, to access various sources of information, analyze their data for relevance according to the request, and process large data streams.

Clustering of bacteria by the triplet composition of 5S RNA

The relationship between the structure of nucleotide sequences, the function they encode, and the taxonomy of the host is an important subject of study for molecular biologists and bioinformaticians, as well as specialists in processing big data arrays. There are various approaches in this direction; one of them is connected with the study of the relationship between the structure of biological macromolecules (specifically, bacterial ribosomal RNA molecules) and the taxonomy of their carriers. For these organisms, 16S RNA sequences are a classic subject of study. However, of no less interest is the study of the relationship be-tween structure and taxonomy for other sequences. In the framework of this work, we studied the relationship between structure and taxonomy using bacterial 5S RNA sequences as an example.

Статистический анализ данных цифрового сервиса электронных и аудиокниг ЛитРес через разработку программного обеспечения

Processing of statistical data of an industry through the use of modern approaches and methods of working with data can solve many problems for forecasting and understanding of the current situation, and is a particularly relevant toolkit for processing big data in the economy. The book industry is of great importance in the economic and cultural component of any country’s development. The book market contributes to the economy by creating jobs and stimulating related industries such as design, advertising, marketing, and is also part of the cultural significance in the form of knowledge preservation and transmission. The article aims to analyze data of the digital service of e-books and audiobooks LitRes. The results of the analysis will be useful for people whose interests are closely related to reading; experts who pay special attention to the digital book market; big data analysts. The research methods employed were statistical analysis and graphical methods. In this study, software was developed to collect and realize the solution of a number of problems through statistical analysis of data from the LitRes website. In the course of the study, more than 10 analytical tasks were solved in the field of genre analytics, calculating the cost of books by genre, making a rating of the most popular books according to reader’s opinion, as well as the most expensive books on the portal. A separate block of economic literature analytics is given. According to the results of analytics of economic literature, it is worth noting that the LitRes service has a wide range of scientific and educational literature, which is represented by various monographs and textbooks for bachelor, master, and graduate students. The most popular book in the study sample is Vladimir Andersons book The Storm in the American Market, the rating of which is 5.0, which demonstrates the reader’s positive response. The compiled rating TOP-3 Most Appreciated Books in the Economic Field of Knowledge can be recommended for reading to people whose interests are closely related to the field of economics, business and finance. The data obtained with the help of the developed software can help to collect statistics, analyze them, and give answers to the set tasks; on their basis, one can also build models to predict the demand for books of certain genres. Certainly, the range of data analysis tasks for the LitRes site can be extended, which can form the basis for further research.

Exploring the Layout Optimization of Commercial Service Space Affecting Energy Consumption in Communities Using Machine Learning

The current design of commercial service spaces in many communities faces issues like incoherence, irrational resource allocation, and low utilization rates. These challenges contribute to increased energy consumption in communities, hindering the overall sustainable development of cities. As a representative community space in the urban environment, the commercial space within the campus requires continuous energy input. Its energy-efficient layout aligns with the principles of sustainable development. This paper uses the university campus as a case study to examine energy-efficient commercial space layout and community practices for environmental protection. Various factors influence the layout of inter-community commercial spaces, and the parameters for measuring the layout structure are diverse, considering the large sample size. Employing machine learning and big data processing to quantify development indicators across various industries and optimize their structure, resource allocation, and energy use has emerged as a viable tool for sustainable urban planning practices. This research seeks to utilize machine learning and data-driven optimization techniques to formulate a comprehensive framework for the sustainable allocation and design of business service spaces within communities. Firstly, we conduct a comprehensive investigation, which includes data collected by applying questionnaire surveys and field research, to assess and model the factors influencing the spatial layout of commercial services on university campuses. Secondly, the AEL machine learning model is constructed by combining the analytic hierarchy process to determine subjective weights, the entropy weight method to calculate objective weights, and the Lagrange algorithm to determine comprehensive weights. Thirdly, we assess and improve the layout of commercial service spaces. Then, by training and testing the Neural Network Model, we apply cases to ensure the accuracy of the machine learning calculation results. Qualitative analysis elucidates the varying factors influencing the sustainable layout of different commercial spaces. Quantitative analysis indicates that, within university campuses, the distance between commercial service spaces and residence halls is a crucial factor in fostering a more sustainable layout. Other significant factors include their location along major student routes and proximity to teaching areas. This study makes contributions not only to the specific field of optimizing commercial service space in communities but also to the broader discourse on sustainable urban development. It advances our understanding of the complex dynamics involved in crafting urban environments that are both efficient and environmentally friendly. Beyond theoretical considerations, the study provides practical solutions and recommendations applicable to implementing tangible improvements in resource allocation. These contributions aim to foster urban environments that are not only environmentally conscious but also economically viable.

Leveraging AI and Machine Learning for Cyber Threat Analysis

This paper explores how AI and Machine Learning is becoming a key participant in cybersecurity's emergence and constantly shifting nature. It disillusioned traditional forms of detection, such as signature-based detection and rule-based systems, with the idea that they are suitable for dealing with new complex threats like zero-day vulnerabilities, APTs, and social engineering attacks. AI and ML provide enhanced features for accurate time operation, real-time threat identification tools, anomaly identification tools, behavioral tools, and predictive analysis, and therefore, have become helpful in the modern context and central to the cybersecurity architecture of many organizations. The paper fleshes out the strengths of AI in intrusion detection, dynamic malware analysis, and phishing detection, among others. The following AI methods are covered in detail: Supervised and unsupervised learning algorithms are used in processing datasets, and autoencoders are used. Deep learning models are suitable for processing big data in real-time. The issues of AI incorporating false positive outcomes, adversarial AI, and data quality problems are also addressed, along with ethical and privacy considerations. The roles of automated incident reporting and self-learning security systems, as forecasted for AI's contributions in managing cyber threats, are designated as crucial. Finally, the paper concludes by stressing the need for organizations to adopt AI solutions for the preemptive approach in the cybersecurity domain.

Comparative analysis of MapReduce and Apache Tez Performance in Multinode clusters with data compression

This article conducts a thorough comparative analysis of Apache Tez and MapReduce in the context of big data processing. It focuses on key performance metrics, scalability, and ease of use. The analysis begins with an overview of the architectural distinctions between the two frameworks, emphasizing their fundamental design principles. A detailed performance evaluation follows, considering factors such as execution time, resource utilization, and throughput across diverse workloads. The study explores scalability by examining how Apache Tez and MapReduce respond to increasing data volumes and computational demands. Cluster size effects, resource allocation strategies, and adaptability to dynamic workloads are scrutinized. Additionally, the article evaluates the frameworks' ease of use for developers and administrators, incorporating aspects like programming model simplicity, debugging capabilities, and system configurability. User experiences are gathered through surveys and practical use cases. The conclusions drawn from this analysis offer valuable insights for organizations and practitioners seeking suitable distributed computing frameworks. By addressing both performance and user experience, the article aims to provide a comprehensive perspective on the strengths and weaknesses of Apache Tez and MapReduce, assisting decision-makers in making informed choices for their big data processing requirements.

The Application of Artificial Intelligence to The Bayesian Model Algorithm for Combining Genome Data

The combination of bioinformatics and artificial intelligence (AI) has made significant progress in the current phase. AI technology, especially deep learning, has been widely used in biology, resulting in many innovations. Currently, AI plays a key role in genomics, proteomics, and drug discovery. Deep learning models are used to predict protein structures, discover potential drug compounds, interpret genomic sequences, analyze medical images, and make personalized medical recommendations. In addition, AI can also help accelerate the processing and interpretation of biological big data, helping biologists to understand complex problems in the life sciences more deeply. Compared to traditional genetic data analysis methods, AI combined with bioinformatics methods are often faster and more accurate, and are capable of processing large-scale, high-dimensional biological data, opening up unprecedented opportunities for life science research. In this paper, the whole genome data and biomedical imaging data are used from the perspective of Bayesian hypothesis testing. Genome-wide association analysis, led by large-scale multiple tests, is a very popular tool for identifying genetic variation points in new complex diseases. In genome-wide association analysis, tens of thousands of SNPS need to be tested simultaneously to find out some SNPS related to traits. These tests are related due to factors such as linkage imbalances in the genetic process, and the test questions are set against the background of high-dimensional data (p >=n).