Big Data Processing Research Articles

Big data and cloud computing: roles and relationships, techniques and tools

In the past years, the increase in data has been accompanied by rapid growth in various fields. It is difficult to analyze large volumes of data using traditional and relational database technology. Therefore, new databases have emerged, and for this reason, big data has become one of the new topics in IT and business today. Also, the cloud environment is increasingly used to store and process big data. Cloud processing refers to processing anything, including Big Data Analytics, on the "cloud". A "cloud" is a collection of high-powered servers from providers that can often view and query large data sets much faster than a regular computer. These two topics differ from each other in various aspects, including definition, collection references, usage method, form and format, and application. In this research, the dimensions and basic concepts, characteristics, tools and techniques, classification, and communication of data are examined. Big has been dealt with cloud computing, and in addition, storage systems, opportunities and challenges, and big data design principles in the cloud environment have been analyzed.

Assessment of it Systems Architecture in the Context of Big Data Processing for Smart Cities Development

Effective use of Big Data can significantly support the development of smart cities and the new digital economy. The aim of the article is a multi-criteria evaluation of IT systems in terms of Big Data processing, taking into account the support for the development of smart cities. The article includes theoretical and empirical research. The adopted criteria for assessing the architecture of IT systems relate to barriers to the implementation of the digital economy in smart cities and the guidelines of international data strategies. The evaluation covered, among other things, cybersecurity and the effectiveness of organizing, storing, and producing new information. The research results allowed us to identify the key factors of Big Data processing efficiency. Based on the research results, an effective model of Big Data processing in organizations was developed. In particular, various data models were analyzed as one of the main elements of software architecture of information systems. The research also focused on data processing techniques such as data warehousing, machine learning, and distributed computing. The efficiency factors of IT systems identified in the research reduce barriers to developing global data strategies and smart cities.

IFogLearn++: A new platform for fog layer's IoT attack detection in critical infrastructure using machine learning and big data processing

Nowadays, with an increasing expansion of the internet of things (IoT) that has created massive data streams, online attack detection via stream processing has become a matter of extensive attention. Current encryption and authentication methods cannot satisfy the security requirements of IoT critical infrastructures because of many heterogeneous connected devices, a wide network geographical scope, rapid software development, the possibility of security bugs, and the emergence of new attacks. Therefore, an extended method based on machine learning, which can process data stream to detect the fog layer's IoT attacks and prevent the spread of intrusion to other network segments, was proposed in this paper. The IFogLearn++ uses a fog layer to facilitate fast data stream processing in the fog layer. Using a fog layer helps the network tolerate more attacks due to the annexation of an extra security layer before the cloud layer. Based on the results, the IFogLearn++ has similar accuracy to competitors and 18 and 8 times faster performance compared to SVM and Learn++. This characteristic helps secure massive data streams in IoT.

A critical evaluation of handling uncertainty in Big Data processing

Big Data is a modern economic and social transformation driver all over the world. The selection has reached a tipping point in terms of big technological advancements that could usher in new ones. It's about making decisions, taking charge of our health, our cities, our wallets, and our education. While the amount, diversity, speed, and reliability of data has increased, so has the complexity of the data (Amzal et al., 2006). Our ability to identify the "value of information through Big Data" will have the most impact. As a result of the resolution of large data concerns, Big Data is revolutionising the IT sector Software vendors in many industries are unable to move forward with dependability and scalability, Information on the project. With the help of relational databases, the explanation for why business requires adequate atomic transactions in the business sector. As a result, to benefit from both technology, advantages, and solutions. Big Data, which refers to databases that are too huge to be processed with current database administration tools, is gaining traction in a variety of significant applications, including internet search, business, social networking, genomics, and meteorology [2]. This concept highlights key research challenges and the promise of data-driven optimization that organically integrates fuzzy, machine learning, and deep learning for decision-making under uncertainty, and identifies potential research opportunities in the business field of Bayesian optimization under uncertainty through a modern data lens. Big Data is a big issue for science and database data mining. Here, we look at the fascinating activities that my community has been doing at this conference to talk about the Big Data problem. As a result, the central focus is linking Big Data to individuals in numerous ways.

Integration of GIS and remote sensing to derive spatially continuous thermal comfort and degree days across the populated areas in Jordan.

The widespread availability of high-resolution digital elevation data and high computational capabilities, along with GIS tools, has revolutionized big data processing, management, and interpolation. The present investigation generates high spatial resolution maps of thermal comfort levels, heating (HDD), and cooling (CDD) degree days across the populated areas in Jordan. Results show that areas having indoor apparent temperature (IAT) of 26°C or above, which represents warm/hot conditions on this thermal index, cover a large portion of the study area during July and August. This thermal zone encompasses a large cluster of the major urban centers in the country. For instance, Amman, Zarqa, and Irbid, which host more than 80% of the population of the country, experience 13, 14, and 19h of warm to very warm conditions during July and August, demonstrating that cooling needs are required to bring about thermal comfort for dwellings and office buildings.Heavy cooling loads, 1700-2000 CDDs, are restricted to the Jordan Rift Valley (JRV) and other small, low-level urban centers. With the exception of the JRV, the populated areas in the country experience cold to very cold conditions during the three coldest months, December through February. Very cold conditions in winter, IAT ≤ 8°C, span more than 13-14h of the diurnal cycle in most urban centers. The HDD range from values close to zero along the JRV to ⁓1900 in the southern mountains. Heating loads for dwellings and office buildings are very demanding and represent a pressing financial challenge to bring about thermal comfort to homes and public buildings during winter. The present procedure can be integrated with auxiliary data within a GIS environment to investigate numerous climatological, environmental, and site suitability issues. The present procedure can be used for operational purposes over territorial or regional scales for a wide range of applications.

Intelligent analysis and processing technology of big data based on clustering algorithm

In order to study the big data intelligent analysis and processing technology based on clustering algorithm, an attribute category clustering method based on hierarchical clustering is proposed, which combines the attribute categories with similar fault type distribution, reduces the data dimension, and binarizes it. Aiming at the problem of more missing values of continuous data, a data completion method based on attribute distribution function is adopted. Then, from the perspective of the selection and estimation of project unit price in construction enterprises, this paper combs and summarizes the data mining process facing the characteristics of project cost data, and puts forward the method of analyzing and processing project cost data based on clustering algorithm. Finally, the processed data sets are subjected to bottom-up hierarchical clustering analysis, and finally the ideal analysis results can be obtained. The experimental results show that the preprocessing method based on attribute clustering proposed in this paper can effectively merge attributes, reduce the dimension after binary transformation and effectively reduce the amount of data under the condition of ensuring data information.

Construction of School-Enterprise Cooperation Practice Teaching System under the Big Data Internet of Things Industry Collaborative Innovation Platform.

The school-enterprise cooperation practice teaching system has increasingly become the core of the field of education research on the basis of the Internet and mobile communications. Based on the big data IoT theory and relying on the industrial collaborative innovation platform, this study designs and implements a general service platform for school-enterprise cooperation practice teaching for IoT applications. The platform is divided into two parts: the IoT data transmission part and the Web practice teaching service section. On the Internet of Things big data transmission platform, after the big data processing provided by the intelligent industry collaborative innovation end is completed, it disguises as a web practical teaching service platform built by the Internet of things framework based on B/S architecture. It solves the problem of data transmission on the order of millions. During the simulation process, the platform realizes flexible deployment and automatic integration of online upgrades. The code management script module based on scripts such as SQL, Python, and shell can complete the online platform. The automatic upgrade finally achieves the goals of the platform being easy to maintain, simple to deploy, and flexible in business logic. The experimental results show that starting from the actual situation of a digital intelligent collaborative innovation platform in a university, the practical teaching system designed and implemented adopts the object-oriented development method, and has a three-tier architecture of the collaborative innovation platform. In addition, the functional modularization and standardization reach 88.7% and 79.4%, respectively, effectively improving the performance of the practical teaching system.

Teacher Education and Management: Innovative Application of Ecological Management System of Big Data Management System

At present, with the rapid growth of China’s comprehensive national power, China wants to accelerate the construction of world-class universities and world-class disciplines to match it. Therefore, the education and management of university teachers is crucial, and the current research results at home and abroad in this area are remarkable. In this paper, we will start from the innovative application of university teachers’ management system and use the Oracle and SQL Server database processors commonly used in big data processing to process data. It is concluded that China’s teaching force is gradually balanced; the comprehensive quality of the teaching force is improving; the academic atmosphere is more open; the fields and perspectives of academic research are richer; the distribution of teachers at all levels in China is more balanced; the comprehensive quality of our teachers is improving; the system of education and scientific research in China’s universities is improving; and the ecological management system of universities is improving. In addition, in view of the current situation of education and management of teachers in China’s colleges and universities, according to the results of big data analysis, we also propose (1) to improve the academic level and establish a strict institutional threshold for the college teaching profession; (2) to effectively implement the policy of freedom and equality in academic academia and improve the appointment and evaluation system of teachers and principals in higher education institutions; (3) to insist on implementing the principle of academic fairness and optimizing the teacher evaluation system; (4) to innovate and implement the incentive mechanism for teachers in colleges and universities according to the talent training characteristics of teachers’ academic career ability; and (5) to establish the access system of college teachers’ career development and promote academic career development and other suggestions of big data management for college teachers.

Blockchain-Based Secure and Trusted Distributed International Trade Big Data Management System

As an object of rapid development of Internet technology in recent years, network security has attracted more and more attention from people from all walks of life and has gradually become a topic in international trade activities. The application of blockchain is also extremely extensive, involving many aspects, such as the development of the metaverse game engine and bank financing. With the launch of the Belt and Road strategy, trade exchanges have become more and more frequent, and at the same time, cross-border e-commerce, a new type of industry that applies Internet information technology, has emerged. The application of blockchain technology to international trade big data processing can not only improve the utilization rate of massive transaction information resources shared by enterprises, governments, financial institutions, and other institutions, but also reduce enterprise costs. The use of blockchain improves the efficiency of data storage, reduces the degree of encryption, and reduces the amount of database access and server resource occupation. This paper mainly studied the application of blockchain in international trade big data system, introduced its secure and trusted distributed network architecture, and analyzed its performance. The experimental results showed that the performance of the decentralized manager was better than that of the centralized manager when processing a large amount of data. The peak throughput of the decentralized manager was about 450 pcs/sec and the peak throughput of the centralized manager was about 150 pcs/sec. And the performance of the storage system in the local area network was better than that in the public network.

Digital Development for Music Appreciation of Information Resources Using Big Data Environment

With the continuous development of information technology and the arrival of the era of big data, music appreciation has also entered the digital development. Big data essence is highlighted by comparison with traditional data management and processing technologies. Under different requirements, the required time processing range is different. Music appreciation is an essential and important part of music lessons, which can enrich people’s emotional experience, improve aesthetic ability, and cultivate noble sentiments. Data processing of music information resources will greatly facilitate the management, dissemination, and big data analysis and processing of music resources and improve the ability of music lovers to appreciate music. This paper aims to study the digital development of music in the environment of big data, making music appreciation more convenient and intelligent. This paper proposes an intelligent music recognition and appreciation model based on deep neural network (DNN) model. The use of DNN allows this study to have significant improvement over the traditional algorithm. This paper proposes an intelligent music recognition and appreciation model based on the DNN model and improves the traditional algorithm. The improved method in this paper refers to the Dropout method on the traditional DNN model. The DNN is trained on the database and tested on the data. The results show that, in the same database, the traditional DNN model is 114 and the RNN model is 120. The PPL of the improved DNN model in this paper is 98, i.e., the lowest value. The convergence speed is faster, which indicates that the model has stronger music recognition ability and it is more conducive to the digital development of music appreciation.

A blockchain framework data integrity enhanced recommender system

AbstractRecommender system for the IoT (RSIoT) has attracted considerable attention. By leveraging emerging technologies such as the Internet of Things (IoT), artificial intelligence, and blockchain, RSIoT improves various indicators of residents' life. However, data integrity threats may affect the accuracy and consistency of the data particularly in the IoT environment where most devices are inherently dynamic and have limited resources that could fail in ensuring the quality of data transmission. Prior work has focused on processing big data and ensuring their integrity by considering cloud storage service as the popular way. In this article, we address integrity of data leveraging blockchain capabilities to ensure the integrity of the critical data. We adapted the Ethereum blockchain to our RCS for ensuring integrity of data during sharing them between doctor and patient without handling their data by third party. We build four smart contracts that enable our system of gaining more advantage of blockchain. We evaluated the performance of our smart contracts in Kovan and Rinkeby test networks. The preliminary results show the feasibility and effectiveness of the proposed solution.

Grid Supervision Path of Platform Food Safety Collaborative Governance Based on Big Data

Food safety is the focus of people’s attention. This year, there have a lot of bad pieces of news related to food safety, which has increased people’s attention to it. According to the requirements for traceability of platform food information attributes, the article aims to study the implementation of grids for food safety governance in the context of big data. This paper designs a nonconformity rate evaluation index to screen qualified food through the grid management of the area using big data processing under the DHRA copy management strategy to achieve the function of assisting food safety supervision. The experimental results of this paper show that grid management can effectively and quickly manage food safety problems. The method of this paper can also optimize the error of food detection, and the error can reach less than 2% in the end. It makes people’s lives safer and ensures that the food they consume is safer.

The Construction of Psychological Intervention Mechanism of Deep Learning in the Prevention of Legal Anomie.

The convenience of big data processing technology has played a great advantage in many scenarios, and its deep learning can effectively mine different types of data in data sets. Applying this method to mining psychological prediction data set of legal anomie behavior can effectively prevent the occurrence of illegal behavior. The effective analysis of its psychological characteristics and the changes of psychological emotions will have hidden dangers, so it is necessary to extract this kind of data in such cases. Therefore, the forest algorithm under deep learning uses weight factors, simple sampling, and other ideas to build the intervention mechanism, which will improve psychological quality to prevent anomie. The experimental results of this paper are as follows: (1) In the case of testing psychological fluctuation by three algorithms, the forest algorithm calculates the psychological change area steadily, with the increase of the number of samples. (2) In the data sample extraction of different dimensions, the test value increases with the increase of dimensions, and it is stable at 12 dimensions. (3) Preventive intervention is extremely important to construct the mechanism before the anomie behavior occurs, and the longest running time of the roll neural network is as much as 950 s. (4) The data mining technology for constructing the intervention mechanism accounts for 15.4% of counseling, which also shows that the auxiliary measures for preventing behavior are particularly important.

Multi-Objective Big Data View Materialization Using Improved Strength Pareto Evolutionary Algorithm

Big data refers to the enormous heterogeneous data being produced at a brisk pace by a large number of diverse data generating sources. Since traditional data processing technologies are unable to process big data efficiently, big data is processed using newer distributed storage and processing frameworks. Big data view materialization is a technique to process big data queries efficiently on these distributed frameworks. It generates valuable information, which can be used to take timely decisions, especially in cases of disasters. As there are a very large number of big data views, it is not possible to materialize all of them. Therefore, a subset of big data views needs to be selected for materialization, which optimizes the query response time for a given set of workload queries with minimum overheads. This big data view materialization problem, having objectives minimization of the query evaluation cost of a set of workload queries, while simultaneously minimizing the update processing costs of the materialized views, has been addressed using improved strength pareto evolutionary algorithm (SPEA-2) in this paper. The proposed big data view selection algorithm, which is able to compute a set of diverse non-dominated big data views, is shown to perform better that existing big data view selection algorithms..

Using cloud computing techniques to monitor long-term variations in ecohydrological dynamics of small seasonally-flooded wetlands in semi-arid South Africa

• Semi-arid Southern African wetlands are lost due to unsustainable use and poor management. • Ramsar Convention on wetlands encourages development of frameworks for wetlands conservation. • The implementation of these frameworks is hindered by a dearth of high-resolution data. • Google Earth Engine (GEE) offers opportunities to address data challenges. • Findings underscores the relevance of GEE in evaluating wetland ecohydrological dynamics. Wetlands in drylands have high inter- and intra-annual ecohydrological variations that are driven to a great extent by climate variability and anthropogenic influences. The Ramsar Convention on Wetlands encourages the development of frameworks for national action and international cooperation for ensuring conservation and wise use of wetlands and their resources at local, national and regional scales. However, the implementation of these frameworks remains a challenge. This is mainly due to limited availability of high-resolution data and suitable big data processing techniques for assessing and monitoring wetland ecohydrological dynamics at large spatial scales, particularly in the sub-Saharan African region. The availability of cloud computing platforms such as Google Earth Engine (GEE) offers unique big data handling and processing opportunities to address some of these challenges. In this study, we applied the GEE cloud computing platform to monitor the long-term ecohydrological dynamics of a seasonally flooded part of the Nylsvley floodplain wetland complex in north-eastern South Africa over a 20-year period (2000–2020). The specific objectives of the study were 1) to evaluate wetland ecohydrological dynamics using the 20-year multi-date Landsat composite data coupled with the Random Forest machine learning algorithm, and 2) to establish the major drivers of wetland ecohydrological changes, using selected spectral indices (i.e. Normalised Difference Vegetation Index (NDVI), Normalised Difference Water Index (NDWI) and Normalised Difference Phenology Index (NDPI)) coupled with climate data. The ecohydrology of the wetland changed over time, with some classes increasing twice when compared to the previous measurement, while others decreasing significantly during the study period. Notably, the bare surface class increased at rates of 230% and 350% between 2006–2010 and 2016–2020, respectively. Moreover, the indices showed similar trends throughout the 20-year period, with NDWI having minimum values less than zero in all cases. This implied no surface inundation, although the presence of some wetland vegetation indicated seasonal to semi-permanent soil saturation conditions. A comparative analysis of climate data and remotely sensed indices showed that annual changes of precipitation and evapotranspiration were the main drivers of wetland ecohydrological variations. The findings of the study underscore the relevance of cloud computing artificial intelligence techniques, and particularly the GEE platform, in evaluating wetland ecohydrological dynamics for semi-arid southern African systems which are deteriorating due to the unsustainable use and poor management resulting from limited knowledge about their changes over time.

Optimization of the Intelligent Sensing Model for Environmental Information in Aquaculture Waters Based on the 5G Smart Sensor Network

This paper adopts a 5G smart sensor network to design intelligent sensing of environmental information in aquaculture waters, and in this way, we study the significance of optimizing the intelligent sensing model. Microservice architecture and big data analysis and processing will become the innovation of technology in the new era. In this regard, it is very important to study a set of aquarium monitoring and management system based on microservice architecture, so that users can understand the changes in the environment in the aquarium, manage a large amount of historical data quickly, and help users solve existing problems. The design pattern of MVVM architecture, unified management of the life cycle, and microservice architecture make the system very easy to expand and maintain; secondly, it realizes the automatic monitoring of the aquarium environment while making multilevel storage optimization for a large amount of data and realizes fast retrieval of a large amount of data; finally, the system can make intelligent regulation in the environmental control of ornamental fish and can automatically adapt to the environmental requirements of different ornamental fish. Finally, the system can make intelligent control in the environment control of ornamental fish and can automatically adapt to the needs of different ornamental fish environments. After testing its server concurrent processing capability, communication stability, and Android performance, the system runs stably. This paper analyzes the architecture module reuse rate and protocol performance; the module reuse rate analysis results show that the various types of modules in the architecture are reused many times and have strong generality, which effectively reduces the development cycle and development cost, and the architecture can quickly and efficiently integrate hardware units such as sensors, actuators, and communication modules; the protocol performance analysis shows that the designed protocol has high communication performance and meets the architecture. The protocol performance analysis shows that the designed protocol has high communication performance and meets the requirements of the architecture in terms of the transmission rate. At the same time, the edge computing module of the architecture further shifts down the edge computing tasks and lays the hardware foundation for the subsequent implementation of sensor data anomaly detection, data encryption transmission, and other functions.

Enabling Transparent Acceleration of Big Data Frameworks Using Heterogeneous Hardware

The ever-increasing demand for high performance Big Data analytics and data processing, has paved the way for heterogeneous hardware accelerators, such as Graphics Processing Units (GPUs) and Field Programmable Gate Arrays (FPGAs), to be integrated into modern Big Data platforms. Currently, this integration comes at the cost of programmability since the end-user Application Programming Interface (APIs) must be altered to access the underlying heterogeneous hardware. For example, current Big Data frameworks, such as Apache Spark, provide a new API that combines the existing Spark programming model with GPUs. For other Big Data frameworks, such as Flink, the integration of GPUs and FPGAs is achieved via external API calls that bypass their execution models completely. In this paper, we rethink current Big Data frameworks from a systems and programming language perspective, and introduce a novel co-designed approach for integrating hardware acceleration into their execution models. The novelty of our approach is attributed to two key design decisions: a) support for arbitrary User Defined Functions (UDFs), and b) no modifications to the user level API. The proposed approach has been prototyped in the context of Apache Flink, and enables unmodified applications written in Java to run on heterogeneous hardware, such as GPU and FPGAs, transparently to the users. The performance evaluation of the proposed solution has shown performance speedups of up to 65x on GPUs and 184x on FPGAs for suitable workloads of standard benchmarks and industrial use cases against vanilla Flink running on traditional multi-core CPUs.

Необходимость и возможность совершенствования жилищной политики в России в условиях санкций

The article provides a brief overview of the situation in the Russian economy after the imposition of sanctions by the United States and European countries. The assessment is given of a hard situation in the housing construction sector because of the high discount rate of the Central Bank, which affects corporate loans, mortgage loans, destruction of logistics supply chains of materials, significant increase in prices for materials and services, physical absence of foreign materials, departure of migrant workers and satisfaction of future demand in 2021. Assuming a long period of sanctions, the author presents a set of measures to change housing policy, which can become one of the main drivers of Russia’s social and economic development under sanctions, including a change in the emphasis in housing policy from high-rise to low-rise individual housing construction, which will allow maintaining the volume of production of industries connected with housing construction and developing regions by merging of urban districts into an agglomeration. In addition to preferential mortgage programs for the construction of individual low-rise houses and for the purchase of secondary housing, the municipal and private partnerships and construction savings banks are suggested, also it is suggested to use digital technologies to create ecosystems in housing construction, combination and process of big data of citizens’ assets, incomes and expenses in order to reduce the cost of mortgages and the selection of specific organizations and products for the solution of the housing issue. The author’s four-level methodology of integrated assessment of efficiency of regional housing policy is suggested for setting strategic goals and monitoring, which is based on the methodological approach of S.A. Ayvazyan and on the methodology of the urban digitalization index “IQ of Cities”, which was developed by the Ministry of Construction of Russia in cooperation with Lomonosov Moscow State University.

Research on Distributed Calculation of Flight Parameters Based on Hadoop

With the improvement of launch vehicle technology and the increase of launch missions, under the intensive launch tasks, the contradiction between the large-scale calculation demand of flight parameters of launch vehicle and the traditional standalone calculation mode is increasingly prominent, which is mainly reflected in the slow calculation speed, low processing efficiency, limited bandwidth bottleneck, and single point fault. Big data processing architecture Hadoop’s distributed computing framework MapReduce, running in low cost cluster, is innovative applied in the large-scale calculation of launch vehicle’s flight parameters, relying on its distributed storage, coordination and load balancing mechanism. The method effectively improves computing efficiency, breakthroughs performance bottleneck and avoids single point failure. Compared with the traditional standalone deployment and pseudo-distributed cluster deployment based on Hadoop, the fully distributed cluster deployment based on Hadoop is the optimal deployment for calculating flight parameters. The results show that the parallel calculation of flight parameters based on Hadoop could save time used by 51% with the consistency of calculation results, which is the same as the standalone deployment.

A novel approach for power load forecast based on GAN data augmentation

At present, the concept of Smart Grid is intimately related to huge volumes of data generation, known as Big Data. These play a key role as far as the improvement of the network operation is concerned and, above all, when it comes to energy consumption. Hence the load forecast based on Big Data processing represents a reliable alternative in carrying out an accurate load prediction. As such, a new approach for the power load forecast, based on GAN data augmentation, for a hospital, within a Smart Grid, will be presented in this work.