Big Data Environment Research Articles

Tension in the data environment: How organisations can meet the challenge

Big Data is becoming ubiquitous - widely applied across organisations, industry sectors and society. However, the opportunities and risks it presents are not yet fully understood. In this paper we identify and explore the tensions that Big Data can create at multiple levels, focusing on the need for organisations to meet the challenges that can arise. We draw on insights from twelve papers published in the Special Issue of Technological Forecasting & Social Change entitled “Tension in the Data Environment: Can Organisations Meet the Challenge?” in order to build a ‘Multi-Layer Tensions Model’ that highlights key pressures and challenges in the BD environment. We find evidence of tensions of three types, which we summarise as “Organisational Learning”, “Organisational Leadership” and “Societal” tensions. We contribute, first, by identifying and developing a nuanced understanding of the tensions faced in the Big Data environment; and second, by elaborating on the capabilities that can be developed and the actions taken to maximise the benefits of Big Data. We end with a “Learning, Leading, Linking” framework, which points to implications for practice and a future research agenda.

A Proposed Framework for Secure Data Storage in a Big Data Environment Based on Blockchain and Mobile Agent

The sum of Big Data generated from different sources is increasing significantly with each passing day to extent that it is becoming challenging for traditional storage methods to store this massive amount of data. For this reason, most organizations have resolved to use third-party cloud storage to store data. Cloud storage has advanced in recent times, but it still faces numerous challenges with regard to security and privacy. This paper discusses Big Data security and privacy challenges and the minimum requirements that must be provided by future solutions. The main objective of this paper is to propose a new technical framework to control and manage Big Data security and privacy risks. A design science research methodology is used to carry out this project. The proposed framework takes advantage of Blockchain technology to provide secure storage of Big Data by managing its metadata and policies and eliminating external parties to maintain data security and privacy. Additionally, it uses mobile agent technology to take advantage of the benefits related to system performance in general. We present a prototype implementation for our proposed framework using the Ethereum Blockchain in a real data storage scenario. The empirical results and framework evaluation show that our proposed framework provides an effective solution for secure data storage in a Big Data environment.

Construction of Knowledge Service Model of Guizhou Supply Chain Enterprises Based on Big Data

In the era of big data, "knowledge" scope is expanded. To realize the optimization of supply chain collaborative innovation in the era of big data, the platform of the collection, analysis, mining and application of massive data resources is needed. By analyzing the sources of big data of collaborative innovation of supply chain, a basic framework of knowledge innovation platform of Guizhou supply chain enterprises under the environment of big data is proposed. The effect of big data technology on supply chain logistics mode is analyzed, and the current situation of logistics industry modernization in Guizhou province is discussed. The mathematical model of big data processing is designed, and an example is simulated to validate the advantage of the proposed method.

Grid-based spatiotemporal modeling of ambient ozone to assess human exposure using environmental big data

Ozone (O3) pollution in China is increasing. It is the primary pollutant in summer and ambient O3 can lead to serious health problems for the public. Therefore, characterizing the spatiotemporal distribution of O3 is required for better environmental management and human exposure assessment. Statistical models, especially those based on machine learning, can be more convenient to use than chemical transport models and have shown improved accuracy. However, the quality of data affects model precision especially at fine spatiotemporal scales. Web-based environmental data can improve the spatial and temporal resolution of modeling data. This study applied high spatiotemporal resolution source information and emission inventories based on point of interest and real-time traffic data in a fine scale grid network to predict the O3 concentrations and assess the human exposure within Chengdu, China. The results showed that the web-based environmental data could be combined with statistical models such as random forest in air quality modeling. The model precision was high, especially at finer spatiotemporal scales. The R2 of the hourly and daily maximum 8-h mean concentrations of O3 models built in this study were 0.83 and 0.91 for sample-based cross-validation, and 0.79 and 0.90 for site-based cross-validation, respectively. Meteorological variables had the greatest impact on O3 concentrations especially sea-level pressure, temperature, vapor pressure, and humidity. People within the research area had a relatively high exposure level to pollutants over a longer time scale in the summer and spring. Findings from this work provide a good reference for related research on modeling air quality, and human health risk assessment using environmental big data.

Distributed authentication framework for Hadoop based bigdata environment

Distributed authentication framework for Hadoop based bigdata environment

Assessments of Air Pollution Control Effectiveness Based on a Sharp Regression Discontinuity Design —Evidence From China’s Environmental Big Data

The Assessment andAppraisal Method for Ecological Construction Targets (the Method) was promulgated in 2016, which provided a concrete instruction for China’s air pollution control and established an explicit standard for reducing air pollutant concentration. This study implements a sharp regression discontinuity (RD) design and makes an assessment on air quality control effectiveness of the Method based on the high-volume big data acquired from 173 cities in China. The results show that the Method has significantly improved air pollution control on the overall air quality index (AQI) and reducing concentrations of PM2.5, PM10, SO2, NO2, and CO across the country in the observation periods. However, no reduction effect was observed for O3. The robustness tests support the conclusion as well. Besides, the heterogeneity analysis illustrates that the policy had a significant short-term treatment effect in East, South, Central, North, Northwest, Southwest, and Northeast China. However, the Method’s effect is found to decline over time either nationwide or regionally according to the persistence analysis. Therefore, this article puts forward several suggestions regarding the formulation of long-term regulations for air pollution control, the transformation of the growth model for sustainable development, and optimization of the incentive system for improved pollution control and prevention.

Digital Coal Geological Map

From 1974 to 1994, the Korea Institute of Geoscience and Mineral Resources (KIGAM) systematically prepared and published relatively precise coal cell geology maps of 1:10,000 or 1:25,000 scale for major coal fields across the country. Such a coal cell geology map includes information about the coal seams as well as the geology of the coal field area, so it can be used as an important basic data for coal development. In this paper, the current state of the coal geology map, which was digitized into a spatial DB using GIS, was introduced. This digital coal geological map can be downloaded free of charge from the Geo-Big Data Open Platform (data.kigam.re.kr) and the Environmental Big Data Platform (www.bigdata-environment.kr).

A comparative study to recognize fake ratings in recommendation system using classification techniques

The recommendation system is affected with attacks when the users are given liberty to rate the items based on their impression about the product or service. Some malicious user or other competitors’ try to inject fake rating to degrade the item’s graces that are mostly adored by several users. Attacks in the rating matrix are not executed just by a single profile. A group of users profile is injected into rating matrix to decrease the performance. It is highly complex to extract the fake ratings from the mixture of genuine profile as it resides the same pattern. Identifying the attacked profile and the target item of the fake rating is a challenging task in the big data environment. This paper proposes a unique method to identify the attacks in collaborating filtering method. The process of extracting fake rating is carried out in two phases. During the initial phase, doubtful user profile is identified from the rating matrix. In the following phase, the target item is analysed using push attack count to reduce the false positive rates from the doubtful user profile. The proposed model is evaluated with detection rate and false positive rates by considering the filler size and attacks size. The experiment was conducted with 6%, 8% and 10% filler sizes and with different attack sizes that ranges from 0%–100%. Various classification techniques such as decision tree, logistic regression, SVM and random forest methods are used to classify the fake ratings. From the results, it is witnessed that SVM model works better with random and bandwagon attack models at an average of 4% higher accuracy. Similarly the decision tree method performance better at an average of 3% on average attack model.

Spatiotempo\ufeffral evolution analysis of NO2 column density before and after COVID-19 pandemic in Henan province based on SI-APSTE model

Air pollution is the result of comprehensive evolution of a dynamic and complex system composed of emission sources, topography, meteorology and other environmental factors. The establishment of spatiotemporal evolution model is of great significance for the study of air pollution mechanism, trend prediction, identification of pollution sources and pollution control. In this paper, the air pollution system is described based on cellular automata and restricted agents, and a Swarm Intelligence based Air Pollution SpatioTemporal Evolution (SI-APSTE) model is constructed. Then the spatiotemporal evolution analysis method of air pollution is studied. Taking Henan Province before and after COVID-19 pandemic as an example, the NO2 products of TROPOMI and OMI were analysed based on SI-APSTE model. The tropospheric NO2 Vertical Column Densities (VCDs) distribution characteristics of spatiotemporal variation of Henan province before COVID-19 pandemic were studied. Then the tropospheric NO2 VCDs of TROPOMI was used to study the pandemic period, month-on-month and year-on-year in 18 urban areas of Henan Province. The results show that SI-APSTE model can effectively analyse the spatiotemporal evolution of air pollution by using environmental big data and swarm intelligence, and also can establish a theoretical basis for pollution source identification and trend prediction.

A Computational Intelligence Approach to Predict Energy Demand Using Random Forest in a Cloudera Cluster

Society’s energy consumption has shot up in recent years, making the prediction of its demand a current challenge to ensure an efficient and responsible use. Artificial intelligence techniques have proven to be potential tools in handling tedious tasks and making sense of large-scale data to make better business decisions in different areas of knowledge. In this article, the use of random forests algorithms in a Big Data environment is proposed for household energy demand forecasting. The predictions are based on the use of information from different sources, confirming a fundamental role of socioeconomic data in consumer’s behaviours. On the other hand, the use of Big Data architectures is proposed to perform horizontal and vertical scaling of the solution to be used in real environments. Finally, a tool for high-resolution predictions with great efficiency is introduced, which enables energy management in a very accurate way.

Research on the Development Trend of Management Accounting in the Big Data Environment

Today, my country has entered the era of big data. In this context, my country’s management accounting has ushered in a new development opportunity. In the past, the development of accounting in my country was relatively backward, and it was not enough to support relevant staff to use comprehensive and objective information to improve and optimize management accounting. Nowadays, the development of big data enables relevant staff to obtain required information and relevant financial data through information technology and big data resources, and improve the work efficiency of relevant staff in solving management accounting problems.

A novel MapReduce-based deep convolutional neural network algorithm

Deep convolutional neural networks (DCNNs), with their complex network structure and powerful feature learning and feature expression capabilities, have been remarkable successes in many large-scale recognition tasks. However, with the expectation of memory overhead and response time, along with the increasing scale of data, DCNN faces three non-rival challenges in a big data environment: excessive network parameters, slow convergence, and inefficient parallelism. To tackle these three problems, this paper develops a deep convolutional neural networks optimization algorithm (PDCNNO) in the MapReduce framework. The proposed method first pruned the network to obtain a compressed network in order to effectively reduce redundant parameters. Next, a conjugate gradient method based on modified secant equation (CGMSE) is developed in the Map phase to further accelerate the convergence of the network. Finally, a load balancing strategy based on regulate load rate (LBRLA) is proposed in the Reduce phase to quickly achieve equal grouping of data and thus improving the parallel performance of the system. We compared the PDCNNO algorithm with other algorithms on three datasets, including SVHN, EMNIST Digits, and ISLVRC2012. The experimental results show that our algorithm not only reduces the space and time overhead of network training but also obtains a well-performing speed-up ratio in a big data environment.

Incremental Group-Level Popularity Prediction in Online Social Networks

Predicting the popularity of web contents in online social networks is essential for many applications. However, existing works are usually under non-incremental settings. In other words, they have to rebuild models from scratch when new data occurs, which are inefficient in big data environments. It leads to an urgent need for incremental prediction, which can update previous results with new data and conduct prediction incrementally. Moreover, the promising direction of group-level popularity prediction has not been well treated, which explores fine-grained information while keeping a low cost. To this end, we identify the problem of incremental group-level popularity prediction, and propose a novel model IGPP to address it. We first predict the group-level popularity incrementally by exploiting the incremental CANDECOMP/PARAFCAC (CP) tensor decomposition algorithm. Then, to reduce the cumulative error by incremental prediction, we propose three strategies to restart the CP decomposition. To the best of our knowledge, this is the first work that identifies and solves the problem of incremental group-level popularity prediction. Extensive experimental results show significant improvements of the IGPP method over other works both in the prediction accuracy and the efficiency.

5G-Oriented IoT Big Data Analysis Method System

The application degree and application scope of 5G Internet of Things technology and big data analysis technology are becoming wider and wider, bringing opportunities for the development of traditional enterprises and providing technological innovation support for the development of new enterprises. Based on 5G Internet of Things technology and big data technology, this paper designs and studies an intelligent agricultural monitoring platform. We collect crop growth data and monitor crop growth status through this platform to study the 5G-oriented IoT big data analysis method system. This paper studies the data collection and storage issues involved in the huge agricultural IoT data environment. This article analyzes the specific sources of agricultural big data, the specific methods of data collection, and the methods of various database storage technologies. Combining wireless sensor network technology, large-source data processing technology, and distributed data storage technology, a method is proposed to solve the problem of rural Internet data collection and storage in the big data environment. This paper proposes a spatiotemporal block processing TSBPS to store the first detection data. The method uses spatiotemporal preblocking, data compression, and caching to significantly improve the recording speed of near real-time storage and microdetection data. In the experimental part of this article, experiments are carried out on the key parts of the IOT-HSQM system model that may limit storage or query performance. Experimental results show that this article compares TSBPS and direct writing methods. The maximum write speed increased by 79%, and the average write speed increased by 42%. The IOT-HSQM system model can meet the requirements of compiling and query performance and statistical analysis.

Subjective Thinking on the School Running for Applied Research - oriented Colleges in the Big Data Environment

In the big data environment, the school running work of applied research-oriented colleges needs to keep pace with the times. In this process, the school should actively integrate information -based teaching elements, improve the construction of “double qualified” teachers, and compile and build the existing teaching content, so as to ensure that the teaching management for current college students has the corresponding pertinence and emphasis. In this paper, under the big data environment, the main work of running school in applied research universities is discussed and explored.

Enterprise Financial Risk Identification and Information Security Management and Control in Big Data Environment

With the deepening of business informatization, all kinds of business application data are rapidly gathering, which promotes enterprises to enter the era of big data. Enterprises begin to build the concept of big data, deepen the understanding of big data, extract potential data value, and improve the operation ability of enterprises and information systems. At the same time, big data brings internal control information to the system, which is becoming more and more challenging, so enterprises pay more and more attention to the security of the information system. This paper aims to introduce the enterprise financial risk identification and information security management and control under the big data environment and master the enterprise financial risk identification method so that the enterprise can adapt to the needs of the times competition faster and better. This paper introduces the method of identifying financial risk in the background of big data by classifying the methods of financial risk identification and designing the factor model. Through the experimental investigation of the company's financial asset rate, the enterprise financial risk situation is displayed, and the enterprise can improve the internal management to control the financial risk within a certain range. The experimental results show that from 2016 to 2020, the internal control and asset rate of the enterprise affect the financial risk of the enterprise, 82% of the operators only have a reasonable debt structure and sufficient solvency, the operator can operate in a safe state and then maintain a low financial risk, and the operator should also take measures to prevent the occurrence of risk in advance and realize the business goal of maximizing benefits.

Design of Digital Library Data Search Engine Based on Cloud Computing in Big Data Era

In the big data environment, people’s acquisition of knowledge is no longer limited to the contents of books, but more data information from the Internet. Intelligent search engine implements information retrieval based on knowledge level, with strong natural language understanding and knowledge processing ability, showing good personalized information service characteristics. he theory of information ecology uses the theory, viewpoint and method of ecology to study the constituent factors of information ecosystem. Excess information and meeting the needs of readers have become the main contradictions in library services. As an architecture model suitable for library application, cloud computing can integrate scattered internet information data, realize the intensification of digital libraries, and provide a new solution for building and sharing digital resources and intelligent search engines. This paper discusses the construction mode of digital library service platform based on cloud computing technology, and analyzes the application and design method of intelligent search engine of cloud service digital library under the background of big data.

Research and design of a data desensitization system

With the extensive implementation of SG186 Project and SG-ERP Project, State Grid has accumulated abundant data resources, and the hidden information in these data resources can create great value. But at the same time, the security of massive data has become more important. From the perspective of system design, workflow and key technology of power information desensitization system, this paper analyzes and studies the sensitive level and data type of sensitive information, matches the corresponding desensitization strategy and completes a series of operations of desensitization task, so as to realize the protection of sensitive data in power information under big data environment.

Software Design of Machine Learning Model Based on Big Data Analysis

With the advent of the information age, big data analysis technology has been popularized in various fields.Using big data analysis technology has also effectively promoted the development of various fields.This article mainly analyzes the trend of machine learning algorithms in the big data environment,it also designed and studied machine learning models.

Service Innovation of Industrial Big Data Based on Industry 4.0 Architecture

The service innovation based on the cyber-physical system is a trend in the future manufacturing industry. In this article, the servicing functions involved in the manufacturing industry under the environment of industrial big data were analysed. In addition, the combination of the needs, environment and services of industrial big data were described. Furthermore, the intelligent forecasting tools to manage industrial big data, increase the transparency of industrial production, and improve work efficiency were proposed to adopt for the application of service innovation.