Hadoop Big Data Platform Research Articles

Design and Implementation of Vehicle Scheduling Optimization for Smart Logistics Platform Powered by Hadoop Big Data

E-commerce has become the mainstream consumption mode for people, and with the continuous increase of online shopping business volume, controlling logistics costs has become an urgent problem to be solved. In order to solve the challenges in vehicle scheduling and achieve a scientific and reasonable vehicle scheduling scheme, this research is based on Hadoop Big data platform, introduces the concept of time axis, and builds a vehicle scheduling model based on Hadoop intelligent platform. Based on quantum genetic algorithm and combined with MapReduce model, a quantum improved genetic algorithm is constructed to solve vehicle scheduling optimization problems. The results show that the traditional quantum genetic algorithm converges after 20 iterations, achieving an optimal value of 575 and taking 360 seconds. The improved quantum genetic algorithm converged after 10 iterations and achieved an optimal value of 675, taking 200 seconds. Compared with quantum genetic algorithm, the improved quantum genetic algorithm reduces the time spent by 44.4%. Selecting customer data from a certain logistics company for testing, the improved algorithm shortens the delivery time and achieves the design of the optimal path scheduling plan. This study optimized transportation routes and resource scheduling, reduced transportation costs, and played an important role in optimizing vehicle scheduling in the logistics industry.

Research on K-means Clustering Algorithm Based on MapReduce Distributed Programming Framework

As a classical clustering algorithm, K-means algorithm has a profound research background. In the of big data era, K-means algorithms will play a greater advantage, being able to quickly divide similar data into the same cluster. Combining K-means algorithm with MapReduce distributed computing framework and running on Hadoop big data platform can significantly improve the clustering effect. Based on MapReduce framework structure, this paper studies K-means model, including K-means principle, distance calculation, content validity index and external validity index. On this basis, the K-means clustering flow based on MapReduce big data programming framework is proposed, and the execution process of the algorithm flow is described in detail, which provides a guide for the algorithm implementation.

Performance optimization of computing task scheduling based on the Hadoop big data platform

AbstractHadoop, a distributed computing framework that can efficiently process large-scale datasets, has been used by an increasing number of organizations as the basic computing framework to build cloud computing platforms. Improving its execution efficiency is a hot research direction in the industry, and the scheduling problem is a key factor affecting the execution efficiency of Hadoop. It is very important to identify its shortcomings and improve them. This paper examines and analyses the optimization of computing task scheduling performance based on the Hadoop big data platform. This paper first analyses Hadoop big data processing. Hadoop has high scalability. Computing nodes can be added at any time, and they can participate in cluster work through simple configuration. The paper discusses the improvement in the Hadoop resource scheduling algorithm. The task scheduling algorithm in the Hadoop-based data task localization proposed in this paper is compared with the default algorithm used in the Hadoop task scheduling algorithm. The former shows better local data in all four jobs, there are more data localization tasks, and the expected goal is achieved. The effectiveness of the algorithm is verified, and the performance is improved by 30%.

Power system abnormal pattern detection for new energy big data

Abstract The energy crisis is a problem that countries all over the world pay more and more attention to, and a series of ecological problems caused by it have become increasingly prominent. It is difficult for traditional fossil fuels to maintain a healthy and coordinated sustainable development of society and economy. The establishment of a sustainable energy system has become the development trend of various countries to solve energy problems. Electric energy is a secondary energy that all primary energy can be converted into, and an irreplaceable consumable for all industrial technologies and people’s lives. Electric power data has the characteristics of large rate span, numerous data sources, complicated interaction methods, and various types of data. The existence of abnormal data in the power system will greatly reduce the accuracy of the system state estimation and the state estimation convergence rate. This paper introduces the power grid industrial control system, combines the data flow of power big data, and analyzes the abnormal information detection process in detail. It takes the data stream acquired by the acquisition unit PMU of the wide area measurement system as the research object. The rapid development of the Hadoop big data platform provides important technical support for the research of power grid big data. Based on the Hadoop platform, the clustering algorithm is used to complete the anomaly detection of real-time data. The LOF algorithm has poor performance when dealing with a large amount of high-dimensional data, and has high time and space complexity. In order to make up for the shortcomings of the LOF algorithm, this paper uses the K-means clustering algorithm to propose an improved algorithm K-LOF of the density-based local abnormal factor detection algorithm LOF, and optimizes the neighborhood query process. It is verified by experiments that the K-LOF algorithm can effectively reduce the time complexity of the anomaly detection algorithm and improve the detection accuracy by 2–4.2%.

Research on Enterprise Financial Accounting Information Security Model Based on Big Data

Powered by the rapid development of the information age, big data technology has had a great impact on China’s socialist economic development. Big data technology is integrated into all fields of nowadays society, especially in the accounting industry of enterprises. Accounting computerization quickly replaces the traditional manual bookkeeping, realizes the timeliness and accuracy of accounting information data processing, and improves the work efficiency and quality of accounting staff to a great extent, but at the same time, the security of financial accounting information is also very prominent. To tackle this problem, this paper proposes a hybrid encryption algorithm based on double chaotic system and improved AES encryption algorithm. The improved AES algorithm uses affine transformation pairs (A7 and 6F) to generate new S-boxes. The double four-dimensional hyperchaotic system is transformed from two three-dimensional chaotic systems, and then, the transformed hyperchaotic system is used to generate chaotic sequences, and a block encryption scheme is designed. On Hadoop big data platform, double hyperchaotic encryption scheme and improved AES algorithm are combined. Simulation test results show that this method can safely transmit enterprise financial accounting information data packets. With the increase of computing cluster nodes, its encryption transmission efficiency continues to improve. This scheme not only solves the problem of enterprise financial accounting data security encryption but also realizes the parallel transmission of encrypted data in the information age, forming a double guarantee of data transmission security and efficiency.

Research on Situational Awareness Technology of Industrial Control Network Based on Big Data

With the rapid advancement of national strategies such as "Internet+" and "Made in China 2025", industrial control systems have been widely used in various industries such as energy, municipalities, transportation, water conservancy and aerospace, etc. The security of industrial control networks is always affecting the lifeline of the national economy. Therefore, the security of industrial control networks has not been given enough attention, resulting in frequent industrial control network security incidents, which makes us realize the importance of creating an industrial control network situational awareness system that integrates assessment and prediction. In this paper, we study the situational awareness technology of industrial control network based on big data, and integrate situational extraction as the premise, situational assessment as the core, and situational prediction as the goal to comprehensively sense the situational awareness of industrial control network system. Firstly, we adopt two ways of industrial control network data collection in the situational extraction, one is the use of traffic mirroring bypass access to sensor-aware terminals, without affecting the original production services on the premise of network traffic data collection. The second is the use of WireShark tools to achieve industrial control network traffic packet collection and statistics of traffic packets per second, analysis of the current network state, build Hadoop big data platform to achieve offline data pre-processing and feature extraction, and build Flink and TensorFlow model for graph neural network model training and complete prediction. Secondly, we use a combination of hierarchical analysis and correlation analysis to evaluate the situation, and use the evaluation graph to show the current network security state. Thirdly, the situation prediction is done by training the graph neural network model offline. We use the Flink real-time computation engine to read the data of the industrial control network into the graph neural network model in real time, which is used to enhance the feature representation of each node and to predict the probability of anomaly occurrence of the industrial control network in the future period. Finally, the graph neural network model of this paper is compared with traditional neural networks and machine learning models, and the accuracy and false alarm rate indexes are used to demonstrate the high accuracy and robustness of this model.

Application of Travel Service Recommendation Algorithm Based on Cloud Computing

In the context of big data, traditional collaborative filtering recommendation algorithms cannot provide users with accurate recommendation services, making the sparsity of user data become an important factor affecting the accuracy of recommendation in the complex social network environment. This paper mainly studies the design and application of travel service recommendation algorithm based on cloud computing. In this paper, the data is processed based on MapReduce parallel computing framework to improve the performance and speed of the algorithm. The Linux-equipped cluster is deployed under the Hadoop framework. The cluster feasibility test is carried out to realize the recommendation function of the algorithm, and the function of the algorithm is realized according to the actual problems. On the Hadoop big data platform, the traditional algorithm was improved and the hadoop-based collaborative filtering recommendation algorithm was implemented to further improve the recommendation rate and accuracy of the traditional recommendation filtering algorithm.

RETRACTED ARTICLE: Research on intelligent medical big data system based on Hadoop and blockchain

In order to improve the intelligence of the medical system, this paper designs and implements a secure medical big data ecosystem on top of the Hadoop big data platform. It is designed against the background of the increasingly serious trend of the current security medical big data ecosystem. In order to improve the efficiency of traditional medical rehabilitation activities and enable patients to maximize their understanding of their treatment status, this paper designs a personalized health information system that allows patient users to understand their treatment and rehabilitation status anytime and anywhere, and all medical health data distributed in different independent medical institutions to ensure that these data are stored independently. As a distributed accounting technology for multi-party maintenance and backup information security, blockchain is a good breakthrough point for innovation in medical data sharing. In this paper, the system realizes the personal health data centre on the Hadoop big data platform, and the original distributed data are stored and analyzed centrally through the data synchronization module and the independent data acquisition system. Utilizing the advantages of the Hadoop big data platform, the personalized health information system for stroke has designed to provide personalized health management services for patients and facilitate the management of patients by medical staff.

Application of machine learning algorithms in MBR simulation under big data platform

Abstract Membrane bioreactors (MBRs) are a sewage treatment process that combines membrane separation with bioreactor technology. It has great advantages in sewage treatment. Membrane fouling hinders MBR process development, however. Studies have shown that the degree of membrane fouling can be judged using the membrane flux rate. In this study, principal component analysis was used to extract the main factors affecting membrane fouling, then the random forest algorithm on the Hadoop big data platform was used to establish an MBR membrane flux prediction model, which was tested. In order to verify the model's effectiveness, BP neural network and SVM support vector machine models were established using the same experimental data. The experimental results from the different models were compared, and the results showed that the random forest algorithm gave the best MBR membrane flux predictions.

A Novel Big Data Index Architecture for Programming Environment

With the rapid development of computer information technology and the upgrading of programming software, the types and number of project codes are growing rapidly, showing typical characteristics of large data such as massive, instantaneous, diverse and variable. The distributed column storage database HBase based on the Hadoop big data platform, has the characteristics of high reliability, high performance, column-oriented, and scalability. It also has good scalability, can store more than ten billions of data, and is suitable for large-scale data reading and writing, which takes advantages in processing large-scale unstructured and semi-structured software data on the programming site. So, this paper studies the big data index architecture in the programming field. In view of the low efficiency of HBase non-primary key attribute query, the inverted index is a natural platform for cloud-based big data storage and query in this large development site. Based on this advantage, this paper designs and optimizes the secondary index architecture based on the HBase for programming field big data inverted index.

The Design and Implementation of a Cleaning System Prototype

As we all know, data is one of the most valuable assets, however, raw data is often problematic, not conducive to the training of algorithm models. To cope with this, we can process the dirty data with cleaning systems [1] to obtain standard clean data for data statistics, data mininig and other use. Instead of manually modifying data, writing SQLs or other cumbersome methods which are popular present ways to clean data, the article proposes an approach by making use of the Hadoop big data platform to support massive data and support the cleaning of multiple heterogeneous data sources. Moreover, our system prototype supports custom rules and algorithms, can export results to a specified database, greatly simplifying the workload of data cleaning personnel. Based on the system design and theoretical verification presented in this paper, the author implemented a big data cleaning tool based on big data platform. The typical data cleaning process shows that the data cleaning can be achieved and user operations can be simplified on the basis of the theory proposed in this paper.

Dynamic data auditing scheme for big data storage

When users store data in big data platforms, the integrity of outsourced data is a major concern for data owners due to the lack of direct control over the data. However, the existing remote data auditing schemes for big data platforms are only applicable to static data. In order to verify the integrity of dynamic data in a Hadoop big data platform, we presents a dynamic auditing scheme meeting the special requirement of Hadoop. Concretely, a new data structure, namely Data Block Index Table, is designed to support dynamic data operations on HDFS (Hadoop distributed file system), including appending, inserting, deleting, and modifying. Then combined with the MapReduce framework, a dynamic auditing algorithm is designed to audit the data on HDFS concurrently. Analysis shows that the proposed scheme is secure enough to resist forge attack, replace attack and replay attack on big data platform. It is also efficient in both computation and communication.

HDFS Optimization Strategy Based On Hierarchical Storage of Hot and Cold Data

HDFS has been the core storage service of Hadoop big data platform for many years, with the continuous development of deep learning, artificial intelligence and other technologies, the amount of data continues to grow, the initial batch processing offline scenario did not fit the user’s needs, the data’s access heat appears big difference. Therefore, the HDFS storage system needs to be optimized according to the degree of heat and cold of the data. The heterogeneous storage and erasure-correction code technology of HDFS can improve the storage efficiency to some extent, but users need to specify the behavior of the storage for that particular data. Therefore, it will cause a waste of cluster resources. SSM systems can follow a predetermined set of rules, according to the degree of heat and cold, it uses different storage strategies to optimize the data, so as to improve the efficiency of the whole storage system.

Development and Application of Big Data Platform for “Bohai granary”

In order to effectively solve series of problems which affect grain production in implementation of the “Bohai granary” project, such as the wide varieties of factors (including soil fertility, soil salinity, pH, underground water level, water salinity, crop varieties and management etc.), the huge spatial and temporal variability, the difficulty of real-time information collection, the complexity of analysis and processing etc., a “Bohai granary” big data platform is developed. With the invention of “Shennong IOAT” series of sensing equipment, the acquisition of multi-site real-time synchronization field information is achieved. And by combining a variety of data collection methods, the collection of factors comprehensively affecting the grain production has been realized. Based on the Hadoop big data platform, it becomes a reality to store, analyze and apply the collected data efficiently. Through the big data platform, the precise management of grain production, the automatic warning of soil moisture content, the disaster and typical insect damage in crop growth, and the automatic control of typical link production are all achieved.

Regression Approach for Optimal Purchase of Hosts Cluster in Fixed Fund for Hadoop Big Data Platform

Regression Approach for Optimal Purchase of Hosts Cluster in Fixed Fund for Hadoop Big Data Platform

The Cooperative Study Between the Hadoop Big Data Platform and the Traditional Data Warehouse

The Cooperative Study Between the Hadoop Big Data Platform and the Traditional Data Warehouse

A platform for big data analytics on distributed scale-out storage system

Big data analytics is the process of examining large amounts of data of a variety of types to uncover hidden patterns, unknown correlations and other useful information. Hadoop-based platform emerges to deal with big data. In Hadoop NameNode is used to store metadata in a single system's memory, which is a performance bottleneck for scale-out. Gluster file system has no performance bottlenecks related to metadata. To achieve massive performance, scalability and fault tolerance for big data analytics, a big data platform is proposed. The proposed big data platform consists of big data storage and big data processing. The Hadoop big data platform and the proposed big data platform are implemented on commodity Linux virtual machines clusters and performance evaluations are conducted. According to the evaluation analysis, the proposed big data platform provides better scalability, fault tolerance, and faster query response time than the Hadoop platform.