Big Data Analysis System Research Articles

A Survey on IoT Big Data Analytic Systems: Current and Future

The Internet of Things (IoT) has become widespread around the world. Since a large number of diverse devices, such as vehicles, household electrical appliances, smart phones, and environmental sensors are connected to the Internet, we can obtain a large volume of diverse IoT data, known as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IoT big data</i> . The generation of IoT big data means that efficient analytic systems are needed for many application scenarios, for example, to optimize urban planning, solve air pollution problems, and improve business decisions. In this survey, we review current systems that can efficiently analyze IoT data. Existing systems can be categorized into batch and stream processing systems. We explore Hadoop- and Spark-based batch processing systems for spatiotemporal and trajectory data. We also review fog- and edge-aware stream processing systems. Although many existing systems can efficiently and effectively analyze specific data and tasks, no system exists that can handle all characteristics of IoT big data: volume, velocity, variety, veracity, and variability. We present some open issues and discuss the future of IoT big data analytic systems. This survey aims to help researchers and practitioners better understand current systems and develop new IoT big data analytic systems.

Rehabilitation of Sepsis Patients with Acute Kidney Injury Based on Intelligent Medical Big Data.

The objective of this study was to explore rehabilitation of patients with acute kidney injury (AKI) treated with Xuebijing injection by using intelligent medical big data analysis system. Based on Hadoop distributed processing technology, this study designed a medical big data analysis system and tested its performance. Then, this analysis system was used to systematically analyze rehabilitation of sepsis patients with AKI treated with Xuebijing injection. It is found that the computing time of this system does not increase obviously with the increase of cases. The results of systematic analysis showed that the glomerular filtration rate (59.31 ± 3.87% vs 44.53 ± 3.53%) in the experimental group was obviously superior than that in the controls after one week of treatment. The levels of urea nitrogen (9.32 ± 2.21 mmol/L vs. 14.32 ± 0.98 mmol/L), cystatin C (1.65 ± 0.22 mg/L vs. 2.02 ± 0.13 mg/L), renal function recovery time (6.12 ± 1.66 days vs. 8.66 ± 1.17 days), acute physiology and chronic health evaluation system score (8.98 ± 2.12 points vs. 12.45 ± 2.56 points), sequential organ failure score (7.22 ± 0.86 points vs. 8.61 ± 0.97 points), traditional Chinese medicine (TCM) syndrome score (6.89 ± 1.11 points vs. 11.33 ± 1.23 points), and ICU time (16.43 ± 2.37 days vs. 12.15 ± 2.56 days) in the experimental group were obviously lower than those in the controls, and the distinctions had statistical significance (P < 0.05). The significant efficiency (37.19% vs. 25.31%) and total effective rate (89.06% vs. 79.06%) in the experimental group were obviously superior than those in the controls, and distinction had statistical significance (P < 0.05). In summary, the medical big data analysis system constructed in this study has high efficiency. Xuebijing injection can improve the renal function of sepsis patients with kidney injury, and its therapeutic effect is obviously better than that of Western medicine, and it has clinical application and promotion value.

Intrusion Detection System for Big Data Analytics in IoT Environment

In the digital area, Internet of Things (IoT) and connected objects generate a huge quantity of data traffic which feeds big data analytic models to discover hidden patterns and detect abnormal traffic. Though IoT networks are popular and widely employed in real world applications, security in IoT networks remains a challenging problem. Conventional intrusion detection systems (IDS) cannot be employed in IoT networks owing to the limitations in resources and complexity. Therefore, this paper concentrates on the design of intelligent metaheuristic optimization based feature selection with deep learning (IMFSDL) based classification model, called IMFSDL-IDS for IoT networks. The proposed IMFSDL-IDS model involves data collection as the primary process utilizing the IoT devices and is preprocessed in two stages: data transformation and data normalization. To manage big data, Hadoop ecosystem is employed. Besides, the IMFSDL-IDS model includes a hill climbing with moth flame optimization (HCMFO) for feature subset selection to reduce the complexity and increase the overall detection efficiency. Moreover, the beetle antenna search (BAS) with variational autoencoder (VAE), called BAS-VAE technique is applied for the detection of intrusions in the feature reduced data. The BAS algorithm is integrated into the VAE to properly tune the parameters involved in it and thereby raises the classification performance. To validate the intrusion detection performance of the IMFSDL-IDS system, a set of experimentations were carried out on the standard IDS dataset and the results are investigated under distinct aspects. The resultant experimental values pointed out the betterment of the IMFSDL-IDS model over the compared models with the maximum accuracy 95.25% and 97.39% on the applied NSL-KDD and UNSW-NB15 dataset correspondingly.

A holistic framework to examine the impact of user, organizational and data factors on the use of big data analytics systems

Introduction. More than 80% of big data projects have failed to meet project sponsors' expectations. This study adopts the technology-organization-environment model to provide a holistic framework to examine the key factors contributing to the success of business analytics projects. Method. The paper develops a survey questionnaire consisting of thirty items on eight constructs, based on previous studies from cognitive-experiential self-efficacy, information technology mindfulness, technology-organization-environment, and information systems success theories. A total of 236 actual users in twenty organizations participated in the study. Analysis. We employed structural equation modelling with partial least squares to test the proposed hypotheses. Results. Our analysis shows that openness to novelty, the rational cognitive thinking style, organizational compatibility, organizational readiness, and data connectivity have a positive effect on the active use of big data analytics systems. Data compatibility does not emerge as a strong antecedent for frequency or duration. The active use of these types of applications is a mediating factor that can significantly improve individual job performance. Our findings provide insights into the user’s relative value, as well as organizational and data factors that can be used to promote big data analytics systems in an organization. Conclusions. Our findings strongly suggest that such an integrative approach can help an organization understand the challenges of promoting big data analytics to use and improve employees' job performance.

Key Technologies of Media Data In-Depth Analysis System Based on Artificial Intelligence-Based Big Data

At present, big data related technologies are developing rapidly, and major companies provide big data analysis services. However, the big data analysis system formed by the combination method cannot sense each other and lacks cooperation, resulting in a certain amount of waste of resources in the big data analysis system. In order to find the key technology of the data analysis system and conduct in-depth analysis of the media data, this paper proposes a scheduling algorithm based on artificial intelligence (AI) to implement task scheduling and logical data block migration. By analyzing the experimental results, we know that the performance of LAS (Logistic-Block Affinity Scheduler) is improved by 23.97%, 16.11%, and 10.56%, respectively, compared with the other three algorithms. Based on real new media data, this article analyzes the content of media data and user behavior in depth through big data analysis methods. Compared with other methods, the algorithm model in this paper optimizes the accuracy of hot topic extraction, which has important implications for media data mining. In addition, the analysis results of the emotional characteristics, audience characteristics, and hot topic communication characteristics obtained by the research also have practical value. This method improves the recall rate and F value by 5% and 4.7%, respectively, and the overall F value of emotional judgment is about 88.9%.

College Physical Education and Training in Big Data: A Big Data Mining and Analysis System.

Recently, big data has been broadly used as a research method in all aspects of analysis, prediction, and evaluation. The application of big data to college students' physical education plays a significant role in encouraging the completion of physical education at various levels. The application of the Internet and the advent of smartphones impact the way college students participate in physical exercise. At present, more and more students begin to participate in sports, and students' demand for physical training is increasing. During physical education training, a lot of data is generated every moment because of various actions and behaviors. Due to technical limitations, these data were not effectively collected and applied. In this environment, the development and management of sports data mining systems have become more and more important. This paper designs an intelligent big data system for college physical education training. The study mainly focuses on data decentralization, lack of data talents, insufficient technical support, and low utilization of venues in physical education. While designing a big data system, the data is collected based on ease of data collection, and a response framework with excellent performance in storing analytical data is selected. The design and management of this system have a certain significance for the improvement and optimization of current college physical education training.

Assessment of global solar resource development

With the increasing severity of environmental problems, many countries have set energy transition targets to promote the realization of the Paris Agreement. There has been a global consensus on utilizing solar energy resources as alternatives to conventional sources to support this energy transition. In this regard, analyzing the “location,” “quantity,” and “quality” of global solar energy resources will not only assist an individual country to efficiently utilize these resources but also promote the realization of large-scale intercontinental resource utilization and complementation. This study established the basic database, model methods, and platform tools for global solar energy assessment, Then, a global solar energy resource assessment was conducted, which included the theoretical reserves (TRs), technical installed potential capacity (TPIC), and average development cost (ADC). A comparative analysis of the assessment results for all continents was also performed. After that, based on big data analysis and geographic information system (GIS) calculations, the distribution characteristics of the global solar power TPIC were calculated with the two core indicators, namely the capacity factor and ADC. Furthermore, a data-driven quantitative evaluation of the refined development potential of solar energy resources was performed. Finally, the reasonableness and coincidence analysis of the resource assessment results were verified using data from global and specifically Chinese photovoltaic (PV) bases.

STL-HDL: A new hybrid network intrusion detection system for imbalanced dataset on big data environment

The ability to process large amounts of data in real time using big data analytics tools brings many advantages that can be used in intrusion detection systems. Deep learning approaches have also been increasingly used in big data analysis and intrusion detection systems in recent years. In this study, a new classification-based network attack detection system is proposed on network flow traffic generating big data. In the proposed system, a Hybrid Deep Learning (HDL) network consisting of Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) is used for a better intrusion detection system. In addition, data imbalance processing consisting of Synthetic Minority Oversampling Technique (SMOTE) and Tomek-Links sampling methods called STL was used to reduce the effects of data imbalance on system performance. In the study, PySpark providing Python support on Apache Spark platform in Google Colab environment was used. The multiclass evaluation of the model was made on the CIDDS-001 data set, and the binary classification evaluation was made on the UNS-NB15 data set. Nine different machine learning and deep learning algorithms have been compared to the proposed method. The results obtained were evaluated using the parameters of Accuracy, F-Measure, Precision, Recall, ROC Curve and Precision-Recall Curve. As a result, the proposed method has reached 99.83% accuracy in multiclass classification and 99.17% accuracy in binary classification. According to the results, the proposed method has achieved quite successful results in detecting network attacks in imbalanced data sets compared to current methods.

Call for Special Issue Papers: Big Data Analytics and Intelligent Systems for Cybersecurity.

Call for Special Issue Papers: Big Data Analytics and Intelligent Systems for Cybersecurity.

Images in Space and Time

Medical imaging diagnosis is mostly subjective, as it depends on medical experts. Hence, the service provided is limited by expert opinion variations and image complexity as well. However, with the increasing advancements in deep learning field, techniques are developed to help in the diagnosis and risk assessment processes. In this article, we survey different types of images in healthcare. A review of the concept and research methodology of Radiomics will highlight the potentials of integrated diagnostics. Convolutional neural networks can play an important role in next generations of automated imaging biomarker extraction and big data analytics systems. Examples are provided of what is already feasible today and also describe additional technological components required for successful clinical implementation.

Improved efficiency in automated acquisition of ultra-high-resolution electron holograms using automated target detection.

An automated hologram acquisition system for big-data analysis and for improving the statistical precision of phase analysis has been upgraded with automated particle detection technology. The coordinates of objects in low-magnification images are automatically detected using zero-mean normalized cross-correlation with preselected reference images. In contrast with the conventional scanning acquisitions from the whole area of a microgrid and/or a thin specimen, the new method allows efficient data collections only from the desired fields of view including the particles. The acquisition time of the cubic/triangular nanoparticles that were observed was shortened by about one-fifty eighth that of the conventional scanning acquisition method because of efficient data collections. The developed technology can improve statistical precision in electron holography with shorter acquisition time and is applicable to the analysis of electromagnetic fields for various kinds of nanoparticles.

Prediction of Air Pressure Change Inside the Chamber of an Oscillating Water Column–Wave Energy Converter Using Machine-Learning in Big Data Platform

Wave power is an eco-friendly power generation method. Owing to the highly volatile nature of wave energy, the application of prediction techniques for power generation, failure diagnosis, and operational efficiency plays a key role in the successful operation of wave power plants (WPPs). To this end, we propose the following approaches: (i) deriving the correlation between highly volatile data such as wave height data and sensor data in an oscillating water column (OWC) chamber; (ii) development of an optimal training model capable of accurate prediction of the state of the wave energy converter (WEC) based on the collected sensor data. In this study, we developed a big data analysis system that can utilize the machine learning framework in KNIME (an open analysis platform), and to enable smart operation, we designed a training model using a digital twin of an OWC–WEC that is currently in operation. Using various machine learning models, the pressure of the OWC chamber was predicted, and the results obtained were tested and evaluated to confirm its validity. Furthermore, the prediction performance was comparatively analyzed, demonstrating the excellent performance of the proposed CNN-LSTM-based prediction model.

Role of the Big Data Analytic framework in Business Intelligence and its Impact: Need and Benefits

Big Data is mixed with huge, autonomous sources like decentralized and distributed control system. For organizations that use the conventional data processing mechanism to handle and archive these large data sets, these capabilities pose an extreme obstacle. A new model has to be defined and the existing framework must be re-evaluated for the analysis and management of big data. The word Business Intelligence (BI) applies to applications, technology and activities for commercial knowledge gathering, review, incorporation and presentation. Business Intelligence’s primary aim is to facilitate quicker and stronger business decision-making process.” Therefore, the strategic review of the literature must study the trend of Information System (IS) adoption factors as a well-designed strategic diagnostic tool that can be used for essential decision-making systems to enable more effective and reliable action plans. We started with discussing some frameworks required for strategic excellence by examining the potential approaches of Big Data Analytics (BDA) and Business Intelligence (BI). In the end, we would design an integrated application that functions as an organization's strategic performance management diagnostic tool. “In general, the emphasis and reach are on the corporate decision-making mechanism, there are some specifics on the analysis, but every conceivable tool and instrument is not specified, the concept is sufficiently concrete to assist in the creation of steps. This research paper therefore examines the position of the system for big data analytics and market intelligence.

Big Data Analytics Performance Enhancement For Covid-19 Data Using Machine Learning And Cloud

The exponential rise in software computing, internet and web-services has broadened the horizon for BigData that demands robust and highly efficient analytics system to serve timely and accurate distributed data support. The distributed frameworks with parallelized computing have been found key driving force behind the contemporary BigData analytics systems; however, the lack of optimal data pre-processing, feature sensitive computation and more importantly feature learning makes major at-hand solutions inferior, especially in terms of time and accuracy. Unlike major at hand methods employing machine learning for BigData analytics, in this paper the key emphasis was made on improving pre-processing, low-dimensional semantic feature extraction and lightweight improved machine learning based feature learning for BigData analytics. Noticeably, the proposed model hypothesizes that an analytics solution with BigData characteristics must have the potential to process humongous, heterogenous, unstructured and multi-dimensional features to yield time-efficient and accuracy analytical outputs. In this reference, we proposed a state-of-art new and robust BigData analytics model, specially designed for Spark distributed framework. To process analytical task our proposed model at first employs tokenization, followed by Word2Vec based semantic feature extraction using CBOW and N-Skip-Gram methods. Our proposed model was found more effective with Skip-Gram Word2Vec feature extraction. Simulation results with a publicly available COVID-19 data exhibited better performance than existing K-Means based MapReduce distributed data frameworks.

Design of Open Knowledge Platform Based On Knowledge Base Utilization Model And Service Scenario To Support Solutions Of Regional Issues

Open knowledge platform can provide a purified knowledge base. Thus, we build a platform for several application areas in a cloud computing that supports APIs for various data based on a knowledge utilization model. The goal of this platform is to maximize the utilization of the knowledge base. In order to achieve this goal, we designed the structure of this platform as an open knowledge platform. The targets of the design are to maximize the utilization of data linkage, to expand it to national common knowledge and to increase its usability by providing services with knowledge graphs. In order to design the platform we identified users, information sources, and infrastructures. In the process, we found it is crucial to specify roles and services to the users of the platform. The requirements are induced from a utilization model and scenario of the service based on the knowledge graph. With the service scenario, stakeholders of the platform started narrow down function modules needed to support the service. One of the modules is a national common knowledge in the knowledge base, which provide essential connected knowledge to support solving regional problems of government such as earthquake, flooding. To increase the usability of data scattered by departments and agencies, data linkage, and knowledge between fragmented data sets is included in this platform. Subsequently, we designed modules to support the effective utilization of this knowledge information. Also, we found that a cloud infrastructure instead of in-house hardware and software could provide flexible and compatible services for the platform. Moreover, the cloud system has advantages on big data analysis and distributed system interconnection. Utilization model and scenario-based process modeling provide a systematic approach to design an open knowledge platform that supports many required components enabling interoperability, compatibility, and connectivity among other knowledge bases..

Call for Special Issue Papers: Big Data Analytics and Intelligent Systems for Cybersecurity.

Call for Special Issue Papers: Big Data Analytics and Intelligent Systems for Cybersecurity.

Generating High-Performance FPGA Accelerator Designs for Big Data Analytics with Fletcher and Apache Arrow

As big data analytics systems are squeezing out the last bits of performance of CPUs and GPUs, the next near-term and widely available alternative industry is considering for higher performance in the data center and cloud is the FPGA accelerator. We discuss several challenges a developer has to face when designing and integrating FPGA accelerators for big data analytics pipelines. On the software side, we observe complex run-time systems, hardware-unfriendly in-memory layouts of data sets, and (de)serialization overhead. On the hardware side, we observe a relative lack of platform-agnostic open-source tooling, a high design effort for data structure-specific interfaces, and a high design effort for infrastructure. The open source Fletcher framework addresses these challenges. It is built on top of Apache Arrow, which provides a common, hardware-friendly in-memory format to allow zero-copy communication of large tabular data, preventing (de)serialization overhead. Fletcher adds FPGA accelerators to the list of over eleven supported software languages. To deal with the hardware challenges, we present Arrow-specific components, providing easy-to-use, high-performance interfaces to accelerated kernels. The components are combined based on a generic architecture that is specialized according to the application through an extensive infrastructure generation framework that is presented in this article. All generated hardware is vendor-agnostic, and software drivers add a platform-agnostic layer, allowing users to create portable implementations.

Analysis on the Countermeasures of Optimizing Urban Tourism Public Service System under the Background of Smart City

At present, there are problems in the construction of the tourism public service system such as poor coordination and coordination among various departments, lack of resource integration mechanism, and low level of information of tourism public services. Therefore, by analysing the problems existing in the tourism public service system in the context of smart cities, this paper puts forward strategies to optimize the tourism public service system. First, we build a smart tourism public service system platform for overall development and optimize the tourism public service platform. Secondly, by focusing on the behaviour of tourists, we construct a big data analysis and management system for the behaviour of smart tourism tourists. Finally, for the traffic situation in the process of tourism, a smart traffic line network based on pheromone ant colony is constructed. The results of the questionnaire survey on tourists show that the optimized public service system platform has achieved great satisfaction.

Ambient backscatter communication-based smart 5G IoT network

In this paper, we propose an ambient backscatter communication-based smart 5G IoT network. The network consists of two parts, namely a real-time data transmission system based on ambient backscatter communication and a real-time big data analysis system based on the combination of shallow neural networks and deep neural networks. The real-time data transmission system based on ambient backscatter communication can extend the standby time of data collection equipment, reduce the size of the equipment, and increase the comfort of wearing. The real-time big data analysis system combining the shallow neural network and the deep neural network can greatly reduce the pressure caused by the frequent deep neural network calculations of the MEC and greatly reduce the energy consumed by the MEC for remote real-time monitoring.

A Framework for Forecasting Outbreak of Infectious Diseases Based on Climate Variability and Social Media Content

The amount of data has risen significantly over the last few years, due to the popularity of some of the data generation sources like social media, electronic health records, sensors and online shopping sites. Analyzing, processing and storing this data is very prominent since it helps to uncover hidden patterns and unknown correlations. A big data analysis and prediction System is proposed in this context, which combines weather observations, health data and social media content in order to forecast the outbreaks of infectious diseases in a locality. Finding information about the determinants of disease outbreaks are required to reduce its effects on populations. An In-mapper combiner based MapReduce algorithm is used to calculate the mean of daily measurements of various climate parameters like temperature, atmospheric pressure, relative humidity, solar and wind. The climatic parameter that may leads to the outbreak of a disease is identified by finding the correlation between the parameters and disease incidence count. To evaluate how user’s tweeting patterns and sentiments matched with the outbreak of diseases, all tweets containing keywords related to diseases are collected using twitter streaming APIs and are analyzed and processed using Spark framework. The performance of proposed model is improved due to the presence of tweet processing. This indicates that the real-time analysis of social media data can provide more effective result rather than working on the historical data.