Hadoop Big Data Research Articles

Intelligent Light Emitting Diode Flashing Supplementary Lighting Control Design and Its Big Data Illumination Analysis

Considering the inherent advantages of the Light Emitting Diode (LED) in the field of illumination, this work designs an intelligent supplementary lighting system using LED as the light source. Combining microcontroller and electronic circuit theory, the circuit is built with the microcontroller PIC16F873 as the core control chip. The system utilizes an external 220 V AC-20 V DC conversion power supply, hence operating on a 20 V DC power source. The system consists of four hardware parts: the onboard power supply uses TI-produced TPS54331 as the control chip to achieve voltage conversion; the external signals (flashing and burst flashing signals) are isolated from the microcontroller through an optocoupler circuit; the PWM pulses output from the microcontroller’s RC1/CCP2 pins drive the corresponding switching tubes to achieve the flashing function; the flashing synchronization signal is output externally after optocoupler isolation, and its synchronous output with the flashing signal is achieved through an optocoupler after LED conduction; the circuit is established using TI-produced differential bus transceiver SN65LBC184D to convert the external 485 differential signal to the level signal required by the microcontroller. In the experiment, after completing the hardware design, connecting the LED panel, and debugging the test program, it is found that the designed lighting system has a good supplementary lighting effect. According to the PWM output waveform, the flashing effect meets the design expectations. The Hadoop big data computing platform is introduced. Simulation testing reveals that under no backlight conditions, the system achieves an illumination intensity of around 20 klx at a distance of about 10 meters. With backlight conditions, the system maintains an illumination intensity of around 1.5 klx at a distance of about 10 meters. Further calculations are performed to analyze the variation in foot traffic within the test area’s illumination over 24 hours. The total illumination intensity during different time intervals is compiled, confirming that the system can autonomously adjust the illumination intensity of the area based on changes in foot traffic.

Hadoop-Based Political and Ideological Big Data Platform Architecture and Pattern Mining in Higher Education

There is a growing opportunity for self-improvement within the realm of higher education, represented by the combination of big data technologies and higher education. The Education Big Data Processing Framework (EBDPF) is the name of the platform that is the subject of this article, which focuses on the development of an advanced big data processing platform. In order to process huge datasets in an effective manner, the platform makes use of Hadoop's big data storage architecture, in addition to Hive, Flume, and Sqoop for data collection and synchronization, respectively. In this study, the implementation of standard data mining techniques is investigated through the use of MapReduce programming. More specifically, the study investigates the execution efficiency and scalability of these algorithms within the Hadoop platform. The data clustering task in data mining is chosen as a representative problem in this study so that the effectiveness of the EBDPF may be evaluated accordingly. An implementation of the clustering job that is based on MapReduce is created and tested on the Hadoop platform. Extensive experiments are conducted with a range of cluster sizes and data sizes. The findings provide more evidence that Hadoop distributed systems are more efficient and successful than traditional methods when it comes to managing data mining applications. The extended performance analysis of computing power not only provides evidence that the platform is currently effective, but it also provides a signal that it has a significant amount of potential that has not yet been utilized. At the convergence of big data technology and higher education, the EBDPF emerges as a strong and promising framework, demonstrating considerable steps in accelerating data processing within educational contexts. In conclusion, the EBDPF is a framework that demonstrates significant progress.

Application of Big Data Technology in the Study of “UGS” Collaborative Training Mechanism for Musicology Majors in Colleges and Universities in the Context of Professional Accreditation

Abstract The traditional education model gradually reveals the corresponding problems and deficiencies in the face of the update of the concept of education frontier and the change of talent cultivation mode. In this paper, we first sorted out the relevant features of the big data and data mining process, mined out the specific needs and realistic development dilemmas of musicology professional certification, and explored the appropriate mechanisms and strategies of UGS collaborative education and training. Using big data Hadoop technology as the infrastructure, a platform for music discipline services is designed to support interoperability and sharing of music education resources among universities, governments, and primary and secondary schools. Specifically, semantic similarity calculation is performed with music resource domain ontology to improve the intelligent retrieval ability, and the recommendation method of user dynamic preference is introduced to provide more accurate music teaching resources for platform users. The effect of cultivating musicology majors in colleges and universities is analyzed in terms of emotional attitudes, knowledge, and skills through the collaborative cultivation mechanism. The accuracy of the semantic similarity intelligent search method based on domain ontology is above 50%, and the mean value of students’ satisfaction with the intelligently recommended music learning resources is 4.3 points. 83.42% of the teaching carried out based on the subject service platform indicated that it met their learning needs, and the overall performance of the students who carried out the platform teaching was 16 points higher than that of the traditional classes. By using big data technology, the UGS collaborative cultivation mechanism can be utilized as a tool to enhance the quality of talent cultivation for musicology majors.

Application of Big Data Technology Combined with Clustering Algorithm in Manufacturing Production Analysis System

Production data analysis is crucial for production planning in the manufacturing industry, and accurate and a comprehensive and accurate analysis can improve production planning. To analyze the production of manufacturing industry, the research proposes the big data technology research method of the set clustering algorithm. In the process of this research method, the K-means clustering algorithm is first used to build the production measurement data system, and the Apache Hadoop Big data framework is applied. Then it introduces the Apriori algorithm for data association mining, and finally uses the genetic algorithm for production scheduling in view of big data analysis. The experiment showcases that the research method achieved a support level of 0.002 with 729 association rules when testing the number of association rules. When conducting data throughput testing, the research method achieved a data throughput of 7789 threads/s when the number of threads reached 8 in the Sleep scenario. In the analysis error testing, the error rate of the research method in the retained data fluctuates around 3.1%. When testing the number of processing operations in the process, the maximum error in the analysis results of the processing operations in the research method is 2. The results indicate that the research method possesses exceptional computational performance, carrying out manufacturing production analysis effectively and efficiently. The manufacturing production analysis system designed by the research institute offers valuable reference solutions for the informationization and intelligent development of the manufacturing industry.

Design and Implementation of Vehicle Scheduling Optimization for Smart Logistics Platform Powered by Hadoop Big Data

E-commerce has become the mainstream consumption mode for people, and with the continuous increase of online shopping business volume, controlling logistics costs has become an urgent problem to be solved. In order to solve the challenges in vehicle scheduling and achieve a scientific and reasonable vehicle scheduling scheme, this research is based on Hadoop Big data platform, introduces the concept of time axis, and builds a vehicle scheduling model based on Hadoop intelligent platform. Based on quantum genetic algorithm and combined with MapReduce model, a quantum improved genetic algorithm is constructed to solve vehicle scheduling optimization problems. The results show that the traditional quantum genetic algorithm converges after 20 iterations, achieving an optimal value of 575 and taking 360 seconds. The improved quantum genetic algorithm converged after 10 iterations and achieved an optimal value of 675, taking 200 seconds. Compared with quantum genetic algorithm, the improved quantum genetic algorithm reduces the time spent by 44.4%. Selecting customer data from a certain logistics company for testing, the improved algorithm shortens the delivery time and achieves the design of the optimal path scheduling plan. This study optimized transportation routes and resource scheduling, reduced transportation costs, and played an important role in optimizing vehicle scheduling in the logistics industry.

Emergency Response Plan for Spontaneous Combustion Based on Case-Based Reasoning

In order to avoid missing the best opportunity for emergency rescue in the event of spontaneous combustion and to prevent accidents from developing further, this paper proposed a method to generate an emergency response plan for spontaneous combustion based on case-based reasoning (CBR). Firstly, we adopted Hadoop big data retrieval technology to establish a case library for spontaneous combustion; then, our team applied CBR technology and introduced the differential determination symbol to calculate case similarity; furthermore, we quickly generated an emergency response plan for spontaneous combustion with the help of the Python program; and finally, we used a case to verify the effectiveness of the method. Overall, the results showed that the emergency response plan generated using this proposed method was consistent with the actual situation of the accident case and, compared with other relevant representative algorithms, the results in this paper were more accurate. In practice, this method may be helpful in providing support for generating emergency response plans for spontaneous combustion.

Machine learning and uLBP histograms for posture recognition of dependent people via Big Data Hadoop and Spark platform

For dependent population, falls accident are a serious health issue, particularly in a situation of pandemic saturation of health structures. It is, therefore, highly desirable to quarantine patients at home, in order to avoid the spread of contagious diseases. A dedicated surveillance system at home may become an urgent need in order to improve the patients’ living autonomy and significantly reduce assistance costs while preserving their privacy and intimacy. The domestic fall accident is regarded as an abrupt pose transition. Accordingly, normal human postures have to be recognized first. To this end, we proposed a novel big data scalable method for posture recognition using uniform local binary pattern (uLBP) histograms for pattern extraction. Instead of saving the pixels of the entire image, only the patterns were kept for the identification of human postures. By doing so, we tried to preserve people’s intimacy, which is very important in ehealth. To our knowledge, our work is the first to use this approach in a big data platform context for fall event detection while using Random Forest instead of complex deep learning methods. Application results of our conduct are very interesting in comparison to complex architectures such as convolutional deep neural networks (CNN) and feedforward deep neural networks (DFFNN).

Research on K-means Clustering Algorithm Based on MapReduce Distributed Programming Framework

As a classical clustering algorithm, K-means algorithm has a profound research background. In the of big data era, K-means algorithms will play a greater advantage, being able to quickly divide similar data into the same cluster. Combining K-means algorithm with MapReduce distributed computing framework and running on Hadoop big data platform can significantly improve the clustering effect. Based on MapReduce framework structure, this paper studies K-means model, including K-means principle, distance calculation, content validity index and external validity index. On this basis, the K-means clustering flow based on MapReduce big data programming framework is proposed, and the execution process of the algorithm flow is described in detail, which provides a guide for the algorithm implementation.

ПРЕИМУЩЕСТВА И НЕДОСТАТКИ СОВРЕМЕННЫХ ТЕХНОЛОГИЙ АНАЛИЗА БОЛЬШИХ ДАННЫХ С ТОЧКИ ЗРЕНИЯ ВЛИЯНИЯ НА БИЗНЕС-СТРАТЕГИЮ ОРГАНИЗАЦИИ (НА ПРИМЕРЕ HADOOP)

The purpose of the article is to analyze the key advantages and limitations of the innovative Hadoop big data analysis technology (HDFS) from the standpoint of the impact on the business strategy of an enterprise. The areas of application of this technology are identified, and an assessment of their capabilities is given. The analysis of key tasks for which Hadoop technology is used is carried out. An assessment of possible problems associated with the introduction of this technology for storing big data is given.

AN OVERVIEW OF HADOOP JOB SCHEDULING ALGORITHMS FOR BIG DATA

Rapid advancements in Big data systems have occurred over the last several decades. The significant element for attaining high performance is "Job Scheduling" in Big data systems which requires more utmost attention to resolve some challenges of scheduling. To obtain higher performance when processing the big data, proper scheduling is required. Apache Hadoop is most commonly used to manage immense data volumes in an efficient way and also proficient in handling the issues associated with job scheduling. To improve performance of big data systems, we significantly analyzed various Hadoop job scheduling algorithms. To get an overall idea about the scheduling algorithm, this paper presents a rigorous background. This paper made an overview on the fundamental architecture of Hadoop Big data framework, job scheduling and its issues, then reviewed and compared the most important and fundamental Hadoop job scheduling algorithms. In addition, this paper includes a review of other improved algorithms. The primary objective is to present an overview of various scheduling algorithms to improve performance when analyzing big data. This study will also provide appropriate direction in terms of job scheduling algorithm to the researcher according to which characteristics are most significant.

ТЕХНОЛОГИЯ ОБРАБОТКИ БОЛЬШИХ ДАННЫХ (BIG DATA)

The description of big data, big data processing technologies (big data), Hadoop big data storage systems are considered

Performance optimization of computing task scheduling based on the Hadoop big data platform

AbstractHadoop, a distributed computing framework that can efficiently process large-scale datasets, has been used by an increasing number of organizations as the basic computing framework to build cloud computing platforms. Improving its execution efficiency is a hot research direction in the industry, and the scheduling problem is a key factor affecting the execution efficiency of Hadoop. It is very important to identify its shortcomings and improve them. This paper examines and analyses the optimization of computing task scheduling performance based on the Hadoop big data platform. This paper first analyses Hadoop big data processing. Hadoop has high scalability. Computing nodes can be added at any time, and they can participate in cluster work through simple configuration. The paper discusses the improvement in the Hadoop resource scheduling algorithm. The task scheduling algorithm in the Hadoop-based data task localization proposed in this paper is compared with the default algorithm used in the Hadoop task scheduling algorithm. The former shows better local data in all four jobs, there are more data localization tasks, and the expected goal is achieved. The effectiveness of the algorithm is verified, and the performance is improved by 30%.

Power system abnormal pattern detection for new energy big data

Abstract The energy crisis is a problem that countries all over the world pay more and more attention to, and a series of ecological problems caused by it have become increasingly prominent. It is difficult for traditional fossil fuels to maintain a healthy and coordinated sustainable development of society and economy. The establishment of a sustainable energy system has become the development trend of various countries to solve energy problems. Electric energy is a secondary energy that all primary energy can be converted into, and an irreplaceable consumable for all industrial technologies and people’s lives. Electric power data has the characteristics of large rate span, numerous data sources, complicated interaction methods, and various types of data. The existence of abnormal data in the power system will greatly reduce the accuracy of the system state estimation and the state estimation convergence rate. This paper introduces the power grid industrial control system, combines the data flow of power big data, and analyzes the abnormal information detection process in detail. It takes the data stream acquired by the acquisition unit PMU of the wide area measurement system as the research object. The rapid development of the Hadoop big data platform provides important technical support for the research of power grid big data. Based on the Hadoop platform, the clustering algorithm is used to complete the anomaly detection of real-time data. The LOF algorithm has poor performance when dealing with a large amount of high-dimensional data, and has high time and space complexity. In order to make up for the shortcomings of the LOF algorithm, this paper uses the K-means clustering algorithm to propose an improved algorithm K-LOF of the density-based local abnormal factor detection algorithm LOF, and optimizes the neighborhood query process. It is verified by experiments that the K-LOF algorithm can effectively reduce the time complexity of the anomaly detection algorithm and improve the detection accuracy by 2–4.2%.

Research on Enterprise Financial Accounting Information Security Model Based on Big Data

Powered by the rapid development of the information age, big data technology has had a great impact on China’s socialist economic development. Big data technology is integrated into all fields of nowadays society, especially in the accounting industry of enterprises. Accounting computerization quickly replaces the traditional manual bookkeeping, realizes the timeliness and accuracy of accounting information data processing, and improves the work efficiency and quality of accounting staff to a great extent, but at the same time, the security of financial accounting information is also very prominent. To tackle this problem, this paper proposes a hybrid encryption algorithm based on double chaotic system and improved AES encryption algorithm. The improved AES algorithm uses affine transformation pairs (A7 and 6F) to generate new S-boxes. The double four-dimensional hyperchaotic system is transformed from two three-dimensional chaotic systems, and then, the transformed hyperchaotic system is used to generate chaotic sequences, and a block encryption scheme is designed. On Hadoop big data platform, double hyperchaotic encryption scheme and improved AES algorithm are combined. Simulation test results show that this method can safely transmit enterprise financial accounting information data packets. With the increase of computing cluster nodes, its encryption transmission efficiency continues to improve. This scheme not only solves the problem of enterprise financial accounting data security encryption but also realizes the parallel transmission of encrypted data in the information age, forming a double guarantee of data transmission security and efficiency.

An approach for brain tumour detection based on dual-tree complex Gabor wavelet transform and neural network using Hadoop big data analysis

An approach for brain tumour detection based on dual-tree complex Gabor wavelet transform and neural network using Hadoop big data analysis

Research on Situational Awareness Technology of Industrial Control Network Based on Big Data

With the rapid advancement of national strategies such as "Internet+" and "Made in China 2025", industrial control systems have been widely used in various industries such as energy, municipalities, transportation, water conservancy and aerospace, etc. The security of industrial control networks is always affecting the lifeline of the national economy. Therefore, the security of industrial control networks has not been given enough attention, resulting in frequent industrial control network security incidents, which makes us realize the importance of creating an industrial control network situational awareness system that integrates assessment and prediction. In this paper, we study the situational awareness technology of industrial control network based on big data, and integrate situational extraction as the premise, situational assessment as the core, and situational prediction as the goal to comprehensively sense the situational awareness of industrial control network system. Firstly, we adopt two ways of industrial control network data collection in the situational extraction, one is the use of traffic mirroring bypass access to sensor-aware terminals, without affecting the original production services on the premise of network traffic data collection. The second is the use of WireShark tools to achieve industrial control network traffic packet collection and statistics of traffic packets per second, analysis of the current network state, build Hadoop big data platform to achieve offline data pre-processing and feature extraction, and build Flink and TensorFlow model for graph neural network model training and complete prediction. Secondly, we use a combination of hierarchical analysis and correlation analysis to evaluate the situation, and use the evaluation graph to show the current network security state. Thirdly, the situation prediction is done by training the graph neural network model offline. We use the Flink real-time computation engine to read the data of the industrial control network into the graph neural network model in real time, which is used to enhance the feature representation of each node and to predict the probability of anomaly occurrence of the industrial control network in the future period. Finally, the graph neural network model of this paper is compared with traditional neural networks and machine learning models, and the accuracy and false alarm rate indexes are used to demonstrate the high accuracy and robustness of this model.

Automated hierarchical classification of scanned documents using convolutional neural network and regular expression

<p>This research proposed automated hierarchical classification of scanned documents with characteristics content that have unstructured text and special patterns (specific and short strings) using convolutional neural network (CNN) and regular expression method (REM). The research data using digital correspondence documents with format PDF images from pusat data teknologi dan informasi (technology and information data center). The document hierarchy covers type of letter, type of manuscript letter, origin of letter and subject of letter. The research method consists of preprocessing, classification, and storage to database. Preprocessing covers extraction using Tesseract optical character recognition (OCR) and formation of word document vector with Word2Vec. Hierarchical classification uses CNN to classify 5 types of letters and regular expression to classify 4 types of manuscript letter, 15 origins of letter and 25 subjects of letter. The classified documents are stored in the Hive database in Hadoop big data architecture. The amount of data used is 5200 documents, consisting of 4000 for training, 1000 for testing and 200 for classification prediction documents. The trial result of 200 new documents is 188 documents correctly classified and 12 documents incorrectly classified. The accuracy of automated hierarchical classification is 94%. Next, the search of classified scanned documents based on content can be developed.</p>

An Auxiliary Decision-Making System for Electric Power Intelligent Customer Service Based on Hadoop

Aiming at the problems of low security, high occupancy rate, and long response time in the current power intelligent customer service assistant decision-making system, a power intelligent customer service assistant decision-making system based on the Hadoop big data framework is designed. By analyzing the Hadoop big data framework, according to the characteristics and core elements of the HDFS distributed file system, the MapReduce programming model, and the data mining algorithm, the basic process of power intelligent customer service assistance decision-making is established. We analyze the overall and functional requirements of the system, design the overall architecture and application architecture of the system, design the E-R diagram and table structure of the database according to the database design principle, and realize the design of power intelligent customer service auxiliary decision-making system based on Hadoop big data framework. The test results show that the proposed method has high system security and low system occupancy and can effectively shorten the system response time. The systems run more flawlessly as compared to the existing methods and give impressing results with lesser CPU utilization. The response time was recorded to be about 12.2 seconds for 1000 power intelligent customer servers, which is much lower than that of the competitors.

BigFiRSt: A Software Program Using Big Data Technique for Mining Simple Sequence Repeats From Large-Scale Sequencing Data.

BackgroundSimple Sequence Repeats (SSRs) are short tandem repeats of nucleotide sequences. It has been shown that SSRs are associated with human diseases and are of medical relevance. Accordingly, a variety of computational methods have been proposed to mine SSRs from genomes. Conventional methods rely on a high-quality complete genome to identify SSRs. However, the sequenced genome often misses several highly repetitive regions. Moreover, many non-model species have no entire genomes. With the recent advances of next-generation sequencing (NGS) techniques, large-scale sequence reads for any species can be rapidly generated using NGS. In this context, a number of methods have been proposed to identify thousands of SSR loci within large amounts of reads for non-model species. While the most commonly used NGS platforms (e.g., Illumina platform) on the market generally provide short paired-end reads, merging overlapping paired-end reads has become a common way prior to the identification of SSR loci. This has posed a big data analysis challenge for traditional stand-alone tools to merge short read pairs and identify SSRs from large-scale data.ResultsIn this study, we present a new Hadoop-based software program, termed BigFiRSt, to address this problem using cutting-edge big data technology. BigFiRSt consists of two major modules, BigFLASH and BigPERF, implemented based on two state-of-the-art stand-alone tools, FLASH and PERF, respectively. BigFLASH and BigPERF address the problem of merging short read pairs and mining SSRs in the big data manner, respectively. Comprehensive benchmarking experiments show that BigFiRSt can dramatically reduce the execution times of fast read pairs merging and SSRs mining from very large-scale DNA sequence data.ConclusionsThe excellent performance of BigFiRSt mainly resorts to the Big Data Hadoop technology to merge read pairs and mine SSRs in parallel and distributed computing on clusters. We anticipate BigFiRSt will be a valuable tool in the coming biological Big Data era.

Research on Organization Design of College Chinese Teaching under Big Data Environment.

The pivotal and most important technology point of the application of big data in modern network teaching is the establishment of teaching platform. Based on Hadoop big data technology, this paper establishes a big data platform designed by Chinese teaching institutions in colleges and universities. The teaching system not only realizes the diversified development of network teaching but also realizes the discovery of teaching platform users' needs and the push of resources. In addition, it is found that a small number of people do not have a high understanding of the teaching mode of big data through extensive studies, which also shows that traditional teaching is still lagging behind in big data education. However, on the whole, the Chinese teaching model in universities based on the big data environment deserves further promotion and application in universities.