Big Data Environment Research Articles

Big Data Security Storage Based on Hybrid Large-scale Database

With the development and progress of network technology, hidden dangers of network security have also become a threat to network storage systems. This research mainly discusses the secure storage of big data based on hybrid large-scale databases. The user sends an authentication request to the SSP, and the SSP verifies the user’s time stamp ts to ensure that it is not a replayed message, and checks whether the user ID is legal. The virtualization layer logically divides the hardware devices at the facility layer through virtualization technology, provides unified services to the outside world, and supports the operation of the platform layer; the platform layer builds a file system and database to provide an operating environment for the operation layer; the operation layer provides users with a big data environment resource access behavior and platform traffic are reasonably allocated and managed, platform business deployment, etc.; the application layer provides users with big data computing services. During the file download process, the big data storage module is relatively fast (300-600k/s), while the speed of Seccloud and invisible cloud is relatively slow (200-400k/s). This research helps to improve the data access efficiency of the big data secure storage module.

Research on the Application of Computer Software Technology in Big Data

This article mainly briefly introduces the related concepts of big data and computer software technology, and expounds the main technology of computer software technology in big data. The author discusses the effective application of computer software technology in big data through the current development of computer software technology. This article puts forward the considerations for the application of computer software technology in big data to clarify the important role of computer software technology in big data. The author strengthened the research on computer software technology. Relevant personnel need to continuously innovate and adapt to the big data environment, so as to improve the application level of computer software technology and accelerate the construction of modern society informatization.

An Iterative Algorithm to Derive Priority From Large-Scale Sparse Pairwise Comparison Matrix

Pairwise comparison matrix (PCM) is an important tool to rank items by deriving priorities and has been used in various applications. Though large-scale sparse PCMs appear frequently in today’s big data environment, it is hard for existing prioritization methods to handle large-scale sparse PCMs efficiently due to the curse of dimensionality. The goal of this article is to propose a new algorithm, bipartite graph iterative method (BGIM), to derive priorities from large-scale sparse PCMs. We first extended graph representations of PCMs to bipartite graphs. A transition matrix was induced by resource allocation on the bipartite graph. Finally, an iterative algorithm was designed to calculate priorities. The theoretical properties of the BGIM were analyzed to show its ability to derive priorities from large-scale sparse PCMs. Two experiments were conducted to validate the proposed approach. The numerical examples indicated that the BGIM can deal with traditional decision problems and derive reliable priorities with minimum Euclidean distance (ED) and minimum violation (MV) among the tested methods. The simulation examples suggested that the BGIM can not only derive reliable priorities from large-scale sparse PCMs but also require the least computation time compared with eight prioritization approaches. To demonstrate its applicability to real-world large-scale problems, we applied the BGIM to rank movies using MovieLens dataset with more than 100 000 ratings for 9125 movies. The results showed that the BGIM was the fastest approach and obtained the best ranking among the average ratings and the five prioritization methods.

Inheritage and Evolution of Dragon Boat Culture in Jiangxi Province Under Big Data Environment

Inheritage and Evolution of Dragon Boat Culture in Jiangxi Province Under Big Data Environment

Progress on material characterization methods under big data environment

Material characterization is a non-substitutable and important part of high-throughput computational material science under big data environment. With the development of machine learning, digitization and characterization of materials have become more and more important in the field of material science and engineering. Here, we demonstrate the development of material characterization in recent years and conduct a comprehensive analysis of the material characteristics. For high-throughput computing materials, the characteristics that directly affect the thermoelectric, ferromagnetic, superconducting, and other physical properties of materials mainly include three aspects: element composition, molecular structure, and energy band distribution of the constituent material molecules. Feature extraction and feature selection from the three aspects of material data can effectively improve the quality of the special machine learning model for high-throughput computing materials, quickly predict physical properties, and save computing resources. In addition, the characterization of material data can strengthen the understanding of the correlation of related feature attributes of materials, laying a foundation for further in-depth research on computational material science. We investigate the importance of material characterization under big data environment and analyze the relationship between it and material properties from the four sections.

PFIMD: a parallel MapReduce-based algorithm for frequent itemset mining

Frequent itemset mining (FIM) is a significant data mining technique which is widely adopted in numerous applications for exploring frequent items. With the rapid growth and expansion of datasets, FIM has become an interesting topic for many researchers, which has triggered many innovations of numerous FIM algorithms in the big data environment. This study aims to design an optimization parallel frequent itemset mining algorithm based on MapReduce, named as $${\text{PFIMD}}$$ algorithm, to deal with the problem of time and space complexity during processing and computing item sets, as well as the failure to adequately balance the load among parallel tasks in the existing parallel FIM algorithms. First, a structure called $${\text{DiffNodeset}}$$ is adopted for avoiding the increase of $$N{-}list$$ cardinality in the $${\text{MRPrePost}}$$ algorithm effectively. Then, a 2-way comparison strategy is designed to speed up the $${\text{DiffNodeset}}$$ generation of 2-itemsets and reduce the time complexity of the algorithm. Finally, the steps of the improved algorithm are parallelized using the cloud computing platform Hadoop and the programming model MapReduce. Moreover, to achieve a uniform grouping of each item in $$F{-}list$$ , a load balancing strategy based on dynamic grouping is proposed, which solves the problem of uneven load of each node in the cluster. The experimental results show that the modified algorithm not only overcomes the shortcoming of $${\text{MRPrePost}}$$ in the big data environment, but also greatly reduces the time and space complexity. Finally, the specific applications of $${\text{PFIMD}}$$ algorithm in several multimedia data sets are listed to illustrate its universality.

Groundwater quality evaluation model based on multi-scale fuzzy comprehensive evaluation and big data analysis method

Abstract The reasonable use of water resources has become an important issue for the sustainable development of humanity in the future. Many researches focus on groundwater quality inspection, but not groundwater quality assessment. This paper aims to study groundwater quality evaluation models based on multi-scale fuzzy comprehensive evaluation and big data analysis methods. We combine coarse-grained multi-scale fuzzy entropy and fuzzy comprehensive evaluation method to establish a groundwater quality evaluation model based on big data environment. The evaluation of groundwater samples from 327 test points in Huangpu District, Xuhui District, Hongkou District, and Putuo District of Shanghai was conducted. The results show that the overall condition of Shanghai groundwater is better, and more than 94% of samples qualified as drinking water sources. The method presented in this paper not only guarantees that the coarse-grained data on all scales are consistent with the length of the original data, but also avoids the phenomenon of data loss, which greatly improves the accuracy of subsequent algorithms.

Processes meet Big Data: Scaling process discovery algorithms in Big Data environment

Process mining is one of business process management techniques which is used to extract values from process execution logs. Process discovery algorithms, like alpha and heuristic miners, are used to automatically discover/rebuild business process models from event logs. However, the performance of these techniques is limited when dealing with Big Data. To cope with this issue, we propose a distributed implementation, based on Spark framework, of the alpha and heuristic algorithms to support efficient scalable process discovery for big process data. The approach consists of distributing the CPU intensive phases, such as the construction of the causality matrix related to these algorithms. Experimental results show that the proposed algorithms speed-up and scale-up well with regard to the variation of both data size and the number of nodes in the cluster.

A parallel approach for high utility-based frequent pattern mining in a big data environment

A parallel approach for high utility-based frequent pattern mining in a big data environment

Can Smart City Data be Used to Create New Official Statistics?

Abstract In this article we evaluate the viability of using big data produced by smart city systems for creating new official statistics. We assess sixteen sources of urban transportation and environmental big data that are published as open data or were made available to the project for Dublin, Ireland. These data were systematically explored through a process of data checking and wrangling, building tools to display and analyse the data, and evaluating them with respect to 16 measures of their suitability: access, sustainability and reliability, transparency and interpretability, privacy, fidelity, cleanliness, completeness, spatial granularity, temporal granularity, spatial coverage, coherence, metadata availability, changes over time, standardisation, methodological transparency, and relevance. We assessed how the data could be used to produce key performance indicators and potential new official statistics. Our analysis reveals that, at present, a limited set of smart city data is suitable for creating new official statistics, though others could potentially be made suitable with changes to data management. If these new official statistics are to be realised then National Statistical Institutions need to work closely with those organisations generating the data to try and implement a robust set of procedures and standards that will produce consistent, long-term data sets.

Retraction: Research on Physical Education in Colleges and Universities Based on Multimedia Big Data Environment (J. Phys.: Conf. Ser. 1802 032008)

This article has been retracted by IOP Publishing following an allegation that raises concerns this article may have been created, manipulated, and/or sold by a commercial entity. In addition, IOP Publishing has seen no evidence that reliable peer review was conducted on this article, despite the clear standards expected of and communicated to conference organisers. The authors of the article have been given opportunity to present evidence that they were the original and genuine creators of the work, however at the time of publication of this notice, IOP Publishing has not received any response. IOP Publishing has analysed the article and agrees there are enough indicators to cause serious doubts over the legitimacy of the work and agree this article should be retracted. The authors are encouraged to contact IOP Publishing Limited if they have any comments on this retraction. Retraction published: 9 September 2022

Research on Physical Education in Colleges and Universities Based on Multimedia Big Data Environment

With the advancement of modern society, information exchange is convenient and fast, and the increase in data volume makes data more convincing. Therefore, in the context of big data, the evaluation of college physical education quality is more scientific and objective. In the context of the development of the era of big data, college physical education teaching models mainly involve three micro-classes, MOOCs, and flipped courses. Some colleges and universities use the Internet to innovate physical education teaching methods or introduce innovative teaching software to effectively improve college physical education. The individualized level of class teaching makes it more in line with the physical learning needs of students in the era of big data. Based on this, starting from the big data era and the development of college physical education, combined with specific teaching cases, several college physical education teaching models are introduced, in order to provide help for college physical education teachers to carry out class teaching.

Privacy Detection and Protection for Intelligent Transportation Shared Travel Service

There is a large amount of privacy data in the big data environment. This paper studies the privacy detection and protection for intelligent transportation shared travel service. On the basis of traditional security policy, through the operation of anonymization of private data, the purpose of keeping private data secret under the premise of protecting data characteristics is realized. The anonymized private data can be used by data engineers to upgrade system functions and improve system user experience. In this paper, through the use of Hadoop project HBase tools, complete the data cleaning of privacy data, data desensitization operation. This paper designs and implements a privacy protection scheme for intelligent shared transportation system in big data environment. This paper uses K anonymous protection technology, data encryption technology to achieve the protection of privacy data. In this paper, HBase technology is used to desensitize the privacy data of the server. The experimental results show that the privacy protection scheme proposed in this paper can meet the security requirements of privacy data in transmission, storage and application.

LOF 이상치 알고리즘을 활용한 국내 초미세먼지 발생 특성 탐지 분석

Air quality management is significant to guarantee public health and ecosystem. For the effective control and management of atmospheric environment through different forms of approaches, analyzing of monitoring data and emission inventory should be firstly considered. Air monitoring stations and their real-time data play important roles to understand pollution circumstance around certain region. However, many of atmospheric researchers frequently face difficulties to overview large volumes of monitoring data, because it requires a lot of time and efforts. Local outlier factor (LOF) broadly applied in different research fields is useful data analysis technique to discover particular values have different pattern or/and characteristic in a data group. The authors in this paper has attempted to apply the LOF algorithm to one-year of air quality monitoring data covers the whole Korean territory for suggesting easy way to preliminarily identify the high-level of PM episodes as their occurrence causes i.e. yellow dust, non-yellow dust and domestic wildfire events. As a result, it was effective to figure out particular LOF ranges as the high-level of PM2.5 episodes, because LOF values show comparatively high numbers when concentrations of PM2.5 are rapidly increased or decreased due to LOF algorithm characteristic. The attempt of this research has found the LOF algorithm would be useful to apply as preliminary methodology for understanding which certain episode has been occurred in tremendous data group. Therefore, it would bring further consideration for effectively analyzing the environmental big data in order to establish the air quality management strategy to environmental scientists and policy makers.

Fault detection of energy‐aware grid systems in big data environment

SummaryGrid technology based on parallel theory is used to study a large amount of data processing. This research mainly discusses the fault detection of grid dynamic system in a big data environment. In order to verify the communication efficiency between different security nodes of the ZSS system under different operating system conditions, this study will use four physical computers and VM Ware software to establish a simulated communication environment, using GlobusToolkit (GT)'s existing deployment mechanism to realize the overall grid dynamic deployment architecture. Using Web service technology remotely calls the monitoring module interface of resource nodes in the domain and uses standard protocols to send monitoring data. At the same time, in order for the management node to save the monitoring data in the local standardized form of XML, the administrator can use the saved historical data to process system failures, analyze, and predict system performance. In order to solve the problem of information explosion, a fault detector is used to precisely control the cost of message detection and the state distribution of the detection result and the time of message delay. Under special circumstances, it takes 60 min to execute a 30‐min job, and the fault detection point consumes an average of 54.33 min. The research results show that the fault‐tolerant strategy can improve the performance of resource consumption in the fault‐tolerant tracking system. Adopting the method of dynamically adjustable detection interval can effectively reduce the overhead of the grid dynamic system. The detection method used in this study can meet the application requirements of the grid.

A novel method of material demand forecasting for power supply chains in industrial applications

AbstractBased on research on big data, data mining and other relevant technical theories, a power material demand analysis system is designed and implemented based on big data technology. The main aim of the study is to forecast material demand and provide data support for decision‐makers. The system includes a data centre subsystem and an application subsystem. At the same time, two kinds of collaborative transmission process models of supply chain information are established, and simulation analysis is carried out on the two models by the Monte Carlo method to verify the effect of collaborative transmission of information flow in supply chains within big data environments. The major contribution of the work is the design of a supply chain model with the help of big data. The impacts of the Internet of Things with empirical studies and limited models are the focuses of the study. It can be seen from the simulation results that there will be a minimum to minimise the cost under the two supply chain information‐transfer process models. The manufacturing cost of the big data platform is about lower than that of the traditional supply chain, and increased delay costs accordingly lead to increased costs for manufacturers in both supply chains.

DDF Library: Enabling functional programming in a task-based model

In recent years, the areas of High-Performance Computing (HPC) and massive data processing (also know as Big Data) have been in a convergence course, since they tend to be deployed on similar hardware. HPC systems have historically performed well in regular, matrix-based computations; on the other hand, Big Data problems have often excelled in fine-grained, data parallel workloads. While HPC programming is mostly task-based, like COMPSs, popular Big Data environments, like Spark, adopt the functional programming paradigm. A careful analysis shows that there are pros and cons to both approaches, and integrating them may yield interesting results. With that reasoning in mind, we have developed DDF, an API and library for COMPSs that allows developers to use Big Data techniques while using that HPC environment. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in run time. It brings some of the qualities of Big Data programming, making it easier for application domain experts to write Data Analysis jobs. In this article we discuss the API and evaluate the impact of the techniques used in its implementation that allow a more efficient COMPSs execution. In addition, we present a performance comparison with Spark in several application patterns. The results show that each technique significantly impacts the performance, allowing COMPSs to outperform Spark in many use cases.

Security Threats and Defensive Approaches in Machine Learning System Under Big Data Environment

Under big data environment, machine learning has been rapidly developed and widely used. It has been successfully applied in computer vision, natural language processing, computer security and other application fields. However, there are many security problems in machine learning under big data environment. For example, attackers can add “poisoned” sample to the data source, and big data process system will process these “poisoned” sample and use machine learning methods to train model, which will directly lead to wrong prediction results. In this paper, machine learning system and machine learning pipeline are proposed. The security problems that maybe occur in each stage of machine learning system under big data processing pipeline are analyzed comprehensively. We use four different attack methods to compare the attack experimental results.The security problems are classified comprehensively, and the defense approaches to each security problem are analyzed. Drone-deploy MapEngine is selected as a case study, we analyze the security threats and defense approaches in the Drone-Cloud machine learning application envirolment. At last,the future development drections of security issues and challenages in the machine learning system are proposed.

Identifying the key factors of subsidiary supervision and management using an innovative hybrid architecture in a big data environment

In a highly intertwined and connected business environment, globalized layout planning can be an effective way for enterprises to expand their market. Nevertheless, conflicts and contradictions always exist between parent and subsidiary enterprises; if they are in different countries, these conflicts can become especially problematic. Internal control systems for subsidiary supervision and management seem to be particularly important when aiming to align subsidiaries’ decisions with parent enterprises’ strategic intentions, and such systems undoubtedly involve numerous criteria/dimensions. An effective tool is urgently needed to clarify the relevant issues and discern the cause-and-effect relationships among them in these conflicts. Traditional statistical approaches cannot fully explain these situations due to the complexity and invisibility of the criteria/dimensions; thus, the fuzzy rough set theory (FRST), with its superior data exploration ability and impreciseness tolerance, can be considered to adequately address the complexities. Motivated by efficient integrated systems, aggregating multiple dissimilar systems’ outputs and converting them into a consensus result can be useful for realizing outstanding performances. Based on this concept, we insert selected criteria/dimensions via FRST into DEMATEL to identify and analyze the dependency and feedback relations among variables of parent/subsidiary gaps and conflicts. The results present the improvement priorities based on their magnitude of impact, in the following order: organizational control structure, business and financial information system management, major financial management, business strategy management, construction of a management system, and integrated audit management. Managers can consider the potential implications herein when formulating future targeted policies to improve subsidiary supervision and strengthen overall corporate governance.

A decision tree classifier for credit assessment problems in big data environments

Financial institutions have long sought to reduce the risk of consumer loans by improving their credit assessment methods. As new information and network technologies enable massive data collections from many different sources, credit assessment has become a challenge in the big data environment. Complicated processing is required to deal with vast, messy data sources and ever-changing loan regulations. This study proposes a decision tree credit assessment approach (DTCAA) to solve the credit assessment problem in a big data environment. Decision tree models offer good interpretability and easily understood rules, with competitive performance capabilities. In addition, DTCAA features various data consolidation methods to eliminate some of the noise in raw data and facilitate the construction of decision tree. By using a large volume data set from one of the biggest car collateral loan companies in Taiwan, this study verifies the efficiency and validity of DTCAA. The results indicate that DTCAA is competitive in various situations and across multiple factors, in support of the applicability of DTCAA to credit assessment practices.