MapReduce Parallel Research Articles

The efficiency assessment of using hybrid neural networks for the detection of forged audio data in socially oriented systems

The subject of the research is the problem of detecting falsified data, in particular in audio format, in socially oriented systems. The goal of the work is to develop an effective model based on recurrent and convolutional neural networks for determining the fact of forgery of sound data, using MapReduce technology for parallelization. The article addresses the following tasks: determining the features of audio in socially-oriented systems, conducting an analysis of algorithms for processing audio information both in the form of text and in the form of a signal, forming a list of target architectures of neural networks and revealing the features of their implementation, conducting an experimental test of effectiveness selected approaches. The following methods used are – analytical and inductive method for determining the target set of neural network architectures; expert assessment for the formation of the most influential efficiency factors; experimental, multi-criteria evaluation and statistical methods of data augmentation to determine the most effective model. The following results were obtained: an audio data reprocessing algorithm was developed for the possibility of using recurrent and convolutional networks. Several approaches to data classification using augmentation based on vector autoregression and MapReduce parallelization technology have been implemented. It was determined that the most effective model for the multi-criteria selection problem is a combination of a bidirectional recurrent neural network with support for short- and long-term memory with several convolutional networks. The advantages of using MapReduce technology to optimize training time and data processing are shown, and a set of open questions for further research and applied implementation is defined. Conclusions: he application of an analytical and inductive approach followed by experimental verification made it possible to develop an effective (with an accuracy of more than 96%) a mechanism for detecting fabricated data both in the form of a signal and in text form. The obtained result makes it possible to assert the feasibility of implementing the proposed approach, and, accordingly, makes it possible to reduce the influence of such information in socially oriented systems, especially during crisis events.

Log specification and intelligent analysis method based on oil and gas pipeline SCADA system

Abstract As the control center of the natural gas long-distance pipeline network, the SCADA system shoulders the important tasks of data collection and monitoring of the whole long-distance pipeline, gas transmission management, production scheduling, operation and maintenance coordination in production, and plays a very important role in the whole oil and gas pipeline. In this paper, firstly, the SCADA system for localized long-distance pipelines is explained in detail, including its basic structure and special solutions to problems. Secondly, the AdaBoost algorithm, which combines the MapReduce parallel computing framework, is introduced to collect and process data from the operation logs of the SCADA system and normalize the logs. Finally, to test the interaction between the improved AdaBoost algorithm and the SCADA system, a system test was conducted. The results show that the average latency of scheduling the logs of the SCADA system by AdaBoost algorithm with MapReduce parallel computing framework is only 39.82ms, the average processing speed of the log normalization file data under the multi-threaded mode of the system reaches 86.51GB/s, and the effective accuracy of the fault diagnosis is as high as 90.36%. This shows that the oil and gas pipeline SCADA system interacting with data visualization technology can process operation logs more quickly and can carry out real-time intelligent supervision of the working status and operation parameters of the whole auxiliary system, promoting the intelligent development of the oil and gas pipeline SCADA system.

Coupled coordination analysis of green finance on economic growth based on big data

Abstract Exploring the coupled and coordinated relationship between green finance and economic growth is to achieve the high-quality dual development of green and economy. This paper studies the rough set and information entropy attribute approximate calculation based on MapReduce parallel computing framework under big data technology. The basic principles of rough set and information entropy are described, and the MapReduce algorithm is used to calculate information entropy, mutual information, and attribute kernel, which shows that MapReduce calculation is always on the key-value pairs. For the correlation analysis of green finance and economic growth, the coupled coordination model of green finance and economic growth is constructed using the MapReduce algorithm, and the coupled coordination analysis of green finance and economic growth is carried out for this model. From the comprehensive evaluation index, the comprehensive evaluation index of green finance and economic growth increased by 6308.44% and 2242.21%, respectively. The coupling degree of green finance and economic growth increased by 164.66% and 835.84% from the coupling degree and coordination degree, respectively. Thus, the coupling relationship between green finance and economic growth based on big data is relatively low, but the coordination degree is high, indicating that the two influence each other and interact.

An Optimized IoT-enabled Big Data Analytics Architecture for Edge-Cloud Computing.

The awareness of edge computing is attaining eminence and is largely acknowledged with the rise of Internet of Things (IoT). Edge-enabled solutions offer efficient computing and control at the network edge to resolve the scalability and latency-related concerns. Though, it comes to be challenging for edge computing to tackle diverse applications of IoT as they produce massive heterogeneous data. The IoT-enabled frameworks for Big Data analytics face numerous challenges in their existing structural design, for instance, the high volume of data storage and processing, data heterogeneity, and processing time among others. Moreover, the existing proposals lack effective parallel data loading and robust mechanisms for handling communication overhead. To address these challenges, we propose an optimized IoT-enabled big data analytics architecture for edge-cloud computing using machine learning. In the proposed scheme, an edge intelligence module is introduced to process and store the big data efficiently at the edges of the network with the integration of cloud technology. The proposed scheme is composed of two layers: IoT-edge and Cloud-processing. The data injection and storage is carried out with an optimized MapReduce parallel algorithm. Optimized Yet Another Resource Negotiator (YARN) is used for efficiently managing the cluster. The proposed data design is experimentally simulated with an authentic dataset using Apache Spark. The comparative analysis is decorated with existing proposals and traditional mechanisms. The results justify the efficiency of our proposed work.

A Distributed Framework for Predictive Analytics Using Big Data and MapReduce Parallel Programming

With the advancement of Internet technologies and the rapid increase of World Wide Web applications, there has been tremendous growth in the volume of digital data. This takes the digital world into a new era of big data. Various existing data processing technologies are not consistent and scalable in handling the complexity as well as the large‐size datasets. Recently, there are many distributed data processing, and programming models have been proposed and implemented to handle big data applications. The open‐source‐implemented MapReduce programming model in Apache Hadoop is the foremost model for data exhaustive and also computational‐intensive applications due to its inherent characteristics of scalability, fault tolerance, and simplicity. In this research article, a new approach for the prediction of target labels in big data applications is developed using a multiple linear regression algorithm and MapReduce programming model, named as MR‐MLR. This approach promises optimum values for MAE, RMSE, and determination coefficient (R2) and thus shows its effectiveness in predictions in big data applications.

Retraction Note: Implementation of MapReduce parallel computing framework based on multi-data fusion sensors and GPU cluster

Retraction Note: Implementation of MapReduce parallel computing framework based on multi-data fusion sensors and GPU cluster

Tunnel Security Management Based on Association Rule Mining under Hadoop Platform

The problem of how to use large amounts of historical data for tunnel safety management has a greater practical application value. The association rule method in data mining technology can provide effective decision support for tunnel safety prevention by mining historical data. To address the problem of large data volume and sparse data items in tunnel safety management, an association rule method—Apriori algorithm—based on the Hadoop platform is proposed to improve the efficiency and accuracy of data mining in cloud environment. First, the parallel MapReduce implementation steps are analyzed on the basis of the distributed Hadoop framework. Then, the existing single-user data validation algorithm is improved by applying a multiuser parallel validation algorithm to Apriori in order to reduce the number of validations. Next, the traditional association rule Apriori algorithm is MapReduce optimized to generate a smaller set of useless candidate items. At the same time, Boolean ranking is used to optimize the way transactional data are stored in the database, reducing the number of redundant subsets and the number of times the database is connected, and shortening the task processing time. The experimental results show that the proposed method is able to mine the relationships between tunnel safety hazards and provide effective decision support for tunnel safety prevention. At the same time, the proposed method more efficiently operates than other association rule methods.

Intelligent Analysis of Logistics Information Based on Dynamic Network Data

In order to realize the intellectualization of logistics information analysis, this paper proposes an intelligent analysis method of logistics information based on dynamic network data cloud mining. This paper selects the data of a shipping logistics platform to realize the intelligent analysis experiment of logistics information based on cloud clustering mining. The purpose of the experiment is to find out the advantages of logistics information intelligent analysis based on cloud mining by comparing the performance differences between cloud clustering mining and traditional clustering mining in logistics information intelligent analysis. This paper builds an experimental environment based on Hadoop and MapReduce parallelization based on K-means algorithm. Taking the obtained logistics data as the analysis object, preprocess it and get the results based on cloud clustering mining. The experimental results show that the parallel mining analysis method is 179.2% slower than the traditional mining analysis method in dataset data1, 60.4% slower in dataset data2, and 2.8% faster in dataset data1. The intelligent analysis method of logistics information based on cloud clustering mining has good scalability and speedup ratio. Conclusion. Applying cloud mining to logistics information analysis and realizing the intelligent analysis of logistics information has great advantages, and can well meet the content and efficiency needs of logistics information analysis stakeholders.

System Evaluation and Management of College Students’ Physical Exercise Behavior Stages Integrating Bayesian Association Rules Data Mining Algorithm

The form of association rules is simple, and it is efficient and convenient to apply. However, because association rules cannot express the connection between different rules, in some more complex application fields, when it is necessary to comprehensively consider the impact of multiple factors on the results, the application of rules is more difficult. In the process of reasoning about the node state, the influence of various factors (parent nodes) can be comprehensively considered. This paper proposes a Bayesian network-based association rule representation method. After mining the association rules from the data, through structural learning and conditional probability table learning, the original rules are finally used as Bayesian nets. This effectively expands the application of association rules. The experimental results show that after using MapReduce parallelization, the improved algorithm can not only process larger-scale data sets, but also save a lot of running time. The correlation between physical exercise behavior evaluation management, dimensional exercise motivation, exercise method, and college students’ school adaptability reached a significance of 0.001, and the correlation between dimension exercise time and college students’ school adaptability reached a significance of 0.05. The correlation between physical exercise behavior evaluation management and school adaptability in terms of interpersonal relationship adaptation, learning adaptation, campus adaptation, career adaptation, emotional adaptation, self-adjustment, and satisfaction reached significant levels.

Adaptive Integration Algorithm of Sports Event Network Marketing Data Based on Big Data

To address the issues of low-data integration accuracy and efficiency, as well as a lack of data integration impact, an adaptive data integration algorithm for sports event network marketing data based on big data is presented. The fundamental theory of tensor is researched by examining the notion and features of big data and by using the associated technologies of the big data framework. Collect network-marketing data from a variety of sporting events and feed it to a big data platform. Combined with MapReduce parallelization mode, tensor represents the online marketing data of sports events according to the structured, semistructured, and unstructured characteristics of different big data. Integrate each tensor model based on semitensor product, build a unified data adaptive integration tensor model, and realize the adaptive integration of sports event network marketing data. The experimental results show that the proposed algorithm has a good effect on the adaptive integration of sports event network marketing data and can effectively improve the accuracy and efficiency of data adaptive integration.

Internet Rumor Audience Response Prediction Algorithm Based on Machine Learning in Big Data Environment

Rumors are an important factor affecting social stability in some special times. Therefore, the dissemination and prevention and control mechanisms of rumors have always been issues of concern to the academic community and have long been highly valued and widely discussed by experts and scholars. However, in combination with the Internet as a new type of media, although people have begun to pay attention to online rumors, research on it is still relatively fragmented, especially in the cross-domain research specific to the social influence of online rumors, and there is no clear indication of online rumors. The specific definition also did not analyze in detail the internal connection between its influence and group behavior. Therefore, this article will combine actual cases to explore and analyze the spread and influence process of online rumors and show its social influence, hoping to enrich the research of online rumors. Nowadays, the Internet has become the most important carrier to reflect the public grievances. Internet users have expressed their opinions on hot issues such as enterprises, people’s livelihood, and government management, which has formed a powerful public opinion pressure, which has far exceeded the traditional media. The hidden dangers of security cannot be ignored. Therefore, how to monitor network public opinion from a large amount of network data is a difficult problem that needs to be solved urgently. Firstly, this consists of four modules: information collection, web page preprocessing, public opinion analysis, and public information report. Secondly, text clustering, the core technology of network public opinion, is optimized, and single-pass algorithm based on double threshold is proposed. Then the dual-threshold single-pass algorithm is optimized based on the MapReduce parallel computing model, and finally a network public opinion collection technology is formed under the background of big data. Simulation results can greatly improve the performance of text clustering and can effectively optimize the design using the parallel computing model based on MapReduce. The average miss rate after optimization is 0.7569 times, the average false alarm rate is 0.5556 times, and C det is 0.5714 times. It proves that the collection technology based on machine learning under the background of big data is effective and has good performance.

An Optimization Model for Landscape Planning and Environmental Design of Smart Cities Based on Big Data Analysis

This paper adopts “digital landscape design logic,” analyzes and researches smart city and digital landscape design, and builds a digital city based on design logic, design basis, environment analysis, and results in a presentation based on the dilemma of landscape garden planning and design at this stage and the development trend of the smart garden and digital landscape design. The optimization model of the landscape and environment design is constructed based on design logic, design basis, environment analysis, and result presentation. First, on the Hadoop distributed computing platform based on the MapReduce parallel processing framework, we implement the massive small file processing methods (Hadoop Archives, CombineFileInputFormat, and Sequence Files) to compensate for the inherent defects of Hadoop and experimentally compare the memory consumption and execution efficiency of the three methods to propose a choice. The memory consumption and execution efficiency of the three methods are experimentally compared to propose a selection strategy. Finally, based on the MR-PFP algorithm, we parallelize the frequent itemset in-cab trajectory big data to generate interesting strong association rules. The experimental results show that the MR-PFP algorithm has better speedup ratio performance and higher mining efficiency than the parallel frequent pattern (PFP) growth algorithm. The research and analysis focused on the digital implementation of the standalone environmental analysis, using Rhino software and Grasshopper visual programming language to build parametric logic, establish parametric analysis models, and conduct a comprehensive analysis of the current environment. The study explores the use of digital landscape design methods and technologies in the landscape design process. Using Rhino + Grasshopper parametric and visualization programming software, we built parametric analysis models around elevation, slope, slope direction, water catchment, and viewable area and used mapping and overlay techniques to quantify the urban space. Finally, the purpose of collecting, monitoring, analyzing, simulating, creating, and reproducing landscape information is achieved.

A Framework for Big Earth Observation Data Using Horizontal Scaling Strategy

The rapid growth of earth observation (EO) data poses a challenge to the way of data management. An efficient framework based on big data technology can bring new solutions. Some excellent frameworks have been proposed, which provide efficient organization and management of EO data. However, they are not optimized for data distribution in the storage environment. In this paper, an optimized EO data management strategy is proposed. Different horizontal scaling strategies are designed to explore the optimal scheme of EO data distribution. The MapReduce parallel computing model was used to test the performance of data retrieval in the experiment. The results show that the proposed strategy contributes to the efficient organization and arrangement of data. Remote sensing (RS) data blocks can be evenly distributed to different shards according to the time characteristics and hash characteristics of the strategy, and the logical index of the data reduces the time consumed by the routing process. This distributed management mode that achieves load balancing provides a framework foundation for parallel computing. Therefore, the framework with an efficient strategy can improve the performance of data management.

Block Storage Optimization and Parallel Data Processing and Analysis of Product Big Data Based on the Hadoop Platform

The traditional distributed database storage architecture has the problems of low efficiency and storage capacity in managing data resources of seafood products. We reviewed various storage and retrieval technologies for the big data resources. A block storage layout optimization method based on the Hadoop platform and a parallel data processing and analysis method based on the MapReduce model are proposed. A multireplica consistent hashing algorithm based on data correlation and spatial and temporal properties is used in the parallel data processing and analysis method. The data distribution strategy and block size adjustment are studied based on the Hadoop platform. A multidata source parallel join query algorithm and a multi-channel data fusion feature extraction algorithm based on data-optimized storage are designed for the big data resources of seafood products according to the MapReduce parallel frame work. Practical verification shows that the storage optimization and data-retrieval methods provide supports for constructing a big data resource-management platform for seafood products and realize efficient organization and management of the big data resources of seafood products. The execution time of multidata source parallel retrieval is only 32% of the time of the standard Hadoop scheme, and the execution time of the multichannel data fusion feature extraction algorithm is only 35% of the time of the standard Hadoop scheme.

Efficient parallel derivation of short distinguishing sequences for nondeterministic finite state machines using MapReduce

Distinguishing sequences are widely used in finite state machine-based conformance testing to solve the state identification problem. In this paper, we address the scalability issue encountered while deriving distinguishing sequences from complete observable nondeterministic finite state machines by introducing a massively parallel MapReduce version of the well-known Exact Algorithm. To the best of our knowledge, this is the first study to tackle this task using the MapReduce approach. First, we give a concise overview of the well-known Exact Algorithm for deriving distinguishing sequences from nondeterministic finite state machines. Second, we propose a parallel algorithm for this problem using the MapReduce approach and analyze its communication cost using Afrati et al. model. Furthermore, we conduct a variety of intensive and comparative experiments on a wide range of finite state machine classes to demonstrate that our proposed solution is efficient and scalable.

RETRACTED ARTICLE: Implementation of MapReduce parallel computing framework based on multi-data fusion sensors and GPU cluster

Nowadays, with the rapid growth of data volume, massive data has become one of the factors that plague the development of enterprises. How to effectively process data and reduce the concurrency pressure of data access has become the driving force for the continuous development of big data solutions. This article mainly studies the MapReduce parallel computing framework based on multiple data fusion sensors and GPU clusters. This experimental environment uses a Hadoop fully distributed cluster environment, and the entire programming of the single-source shortest path algorithm based on MapReduce is implemented in Java language. 8 ordinary physical machines are used to build a fully distributed cluster, and the configuration environment of each node is basically the same. The MapReduce framework divides the request job into several mapping tasks and assigns them to different computing nodes. After the mapping process, a certain intermediate file that is consistent with the final file format is generated. At this time, the system will generate several reduction tasks and distribute these files to different cluster nodes for execution. This experiment will verify the changes in the running time of the PSON algorithm when the size of the test data set gradually increases while keeping the hardware level and software configuration of the Hadoop platform unchanged. When the number of computing nodes increases from 2 to 4, the running time is significantly reduced. When the number of computing nodes continues to increase, the reduction in running time will become less and less significant. The results show that NESTOR can complete the basic workflow of MapReduce, and simplifies the process of user development of GPU positive tree order, which has a significant speedup for applications with large amounts of calculations.

Application of Travel Service Recommendation Algorithm Based on Cloud Computing

In the context of big data, traditional collaborative filtering recommendation algorithms cannot provide users with accurate recommendation services, making the sparsity of user data become an important factor affecting the accuracy of recommendation in the complex social network environment. This paper mainly studies the design and application of travel service recommendation algorithm based on cloud computing. In this paper, the data is processed based on MapReduce parallel computing framework to improve the performance and speed of the algorithm. The Linux-equipped cluster is deployed under the Hadoop framework. The cluster feasibility test is carried out to realize the recommendation function of the algorithm, and the function of the algorithm is realized according to the actual problems. On the Hadoop big data platform, the traditional algorithm was improved and the hadoop-based collaborative filtering recommendation algorithm was implemented to further improve the recommendation rate and accuracy of the traditional recommendation filtering algorithm.

Deep mining of communication information association based on discrete Fourier transform

In order to better analyze communication information and improve the efficiency of communication information association mining, a discrete Fourier algorithm is proposed to extract association rules from information flow data with different time granularity. With the help of the MapReduce parallel programming model, a deep mining algorithm on the association of communication information is designed. Firstly, the data is linearly normalized, and the algorithm model of data depth mining is proposed according to the discrete Fourier transform. The data source uses the message middleware Kafka, and the intermediate results are stored in the memory database redis through the serialization technology. By using the structural information and comparison algorithm of the database, K pairs with the largest Pearson correlation coefficient are searched, association rules are mined, meaningless items are filtered out, construct an association deep mining model to realize rapid mining of association rules related to tasks. Experimental results and analysis show that the algorithm is efficient and stable in discovering association rules from high-speed information data flow, and supports different time granularity.

Research on C4.5 algorithm improvement strategy based on MapReduce

Abstract By analyzing the shortcomings of C4.5 decision tree algorithm, this paper combines MapReduce parallel model in Hadoop platform with C4.5 decision tree algorithm, so that C4.5 algorithm can be executed in parallel, so as to improve the efficiency of the algorithm. The improved C4.5 algorithm implementation process is given and the process is analyzed. Finally, the C4.5 algorithm improved experimental environment configuration is explained.

Telecom fraud detection with big data analytics

The rapid development in telecom has also led to an increase in fraud activities, which causes both revenue and reputation losses. For this reason, this paper proposes a new telecom fraud detection model based on behaviour deviations of users expressed through time-varying signatures. In line with the similarity of these deviations to known frauds, a suspect list has been created and reported to fraud experts for the final decision. The proposed model was developed with the MapReduce parallel programming paradigm, which provides simplicity and flexibility for large-scale applications. Finally, the model was applied on call detail records of a telecom company. The obtained results have shown that the proposed approach detects the telecom frauds with 86% success and is suitable for application into a fraud management system for real-world implementation.