Big Data Analytics Platform Research Articles

Construction and Management of Astronomical Big Data Analysis and Multi-dimensional Information Visualization Platform

The development of modern astronomy is rapidly and astronomical data increases exponentially. The HPC architecture based on GPU provides an efficient way of astronomic big data computing. Based on secure Ipv6 network environment, PMO has constructed the Big Data Analysis and Multi-dimensional Information Visualization Platform, which can reach the peak computing speed of 352Tflops and the totally storage capacity of 288TB. The platform is composed of 25 computing nodes, one management node and 5 storage nodes. The use of user-friendly, centralized cluster management software, the deployment of proprietary environmental control settings and multi-dimensional visualization of safety management systems form a multi-level, tridimensional and efficient management structure. A high-speed, high-capacity, highly reliable, secure and efficient high-performance computing cluster is finally constructed. The platform has a fully redundant, self-healing, highly scalable distributed storage system, a high-performance InfiniBand parallel computing and storage network, a complete job scheduling system, a cuda parallel computing architecture, and a variety of physical software tools for astronomical applications. It offers a great help to astronomical research topics such as astronomical image processing, moving target extraction, space target orbit calculation, numerical cosmology, cosmology simulation, galaxy fluid simulation. Thus it provides an important information support for the research work of 3 major breakthroughs and 5 key cultivation directions in the "One Three Five" plan of Purple Mountain Observatory.

Hawkeye: Adaptive Straggler Identification on Heterogeneous Spark Cluster With Reinforcement Learning

It is a common sense that people harbors the belief that stragglers exert huge influence upon the performance conducted by the analysis systems of big data for the reason of poor performance made by some computing nodes, data skew and so on. Accordingly, stragglers have been billed as an indispensable bottleneck in Map-Reduce framework processing. However, existing studies on stragglers identification are targeting coarse-grained detection, schedule level optimization and off-line log based cause analysis. Accuracy identifying the stragglers in time for each job, however, is an extremely tough because (1) Auite a number of root causes for stragglers in data analytics frameworks;(2) The number of key parameters affecting stragglers identification; and (3) The different clusters configurations, and their impact on the stragglers detection, vary among different job types and sizes. Either existing solutions adopt a “tweak-and-pray” manual tuning approach, which is complex, time-consuming and error-prone, or only most of them fix theirs eyes upon coarse-grained straggler detection. In this paper, we systematically conduct the exploration on the fundamental problem of automatic, adaptive straggler identification on big data analytics platform. Under the inspiration of the recent triumphs over implementing Reinforcement Learning (RL) techniques for solving complex online optimal problems, we conducted investigation that Reinforcement learning are reasonably employed to adaptively opt the optimal parameters to identify stragglers free of the intervention of human beings. Specifically, we propose Hawkeye, a general adaptive speculative execution system which identifies stragglers by reinforcement are learning to launch speculative tasks on heterogeneous cluster at runtime. In accordance with the experimental conclusions, Hawkeye manages to cut down the job completion time over the distinct type applications. An instance is that it reveals as many as nearly 37% decrease average job completion time based on an improvement of 23% on the preciseness of the present resolutions to the heterogeneous cluster.

A Brief Survey on Big Data in Healthcare

This article presents a brief introduction to big data and big data analytics and also their roles in the healthcare system. A definite range of scientific researches about big data analytics in the healthcare system have been reviewed. The definition of big data, the components of big data, medical big data sources, used big data technologies in present, and big data analytics in healthcare have been examined under the different titles. Also, the historical development process of big data analytics has been mentioned. As a known big data analytics technology, Apache Hadoop technology and its core components with tools have been explained briefly. Moreover, a glance of some of the big data analytics tools or platforms apart from Hadoop eco-system were given. The main goal is to help researchers or specialists with giving an opinion about the rising importance of used big data analytics in healthcare systems.

Design Of For Multivariable Big Data Mobile Analysis Platform Based On Collaborative Filtering Recommendation Algorithm

In order to overcome the problems of poor accuracy and high data redundancy in the current big data analysis platform, this paper proposes and designs a multivariable big data mobile analysis platform based on collaborative filtering recommendation algorithm. The platform is divided into data acquisition layer, storage layer, processing and analysis layer and scheduling layer, introduces two ways of dimensionality reduction and recommendation to realise multivariable big data mining and analysis. User behaviour analysis and data item behaviour analysis of the dimension-reduced data are carried out, and multi-level coordination is used to complete the construction of multivariable big data mobile analysis platform. The experimental results show that the accuracy of the platform's big data analysis is always above 97%, and the accuracy of data mining analysis is stronger. The acceleration ratio is always above 2, the response speed is faster, the user satisfaction is about 96%.

Components and Development in Big Data System: A Survey

With the growth of distributed computing systems, the modern Big Data analysis platform products often have diversified characteristics. It is hard for users to make decisions when they are in early contact with Big Data platforms. In this paper, we discussed the design principles and research directions of modern Big Data platforms by presenting research in modern Big Data products. We provided a detailed review and comparison of several state-of-the-art frameworks and concluded into a typical structure with five horizontal and one vertical. According to this structure, this paper presents the components and modern optimization technologies developed for Big Data, which helps to choose the most suitable components and architecture from various Big Data technologies based on requirements.

Design and Research of Big Data Collection and Analysis Platform Based on Cloud Computing

More new technologies and products emerged in the era of big data. In the era of big data, the paper proposes a big data collection and analysis platform based on cloud computing. The paper first designed the software architecture and network architecture of the big data analysis platform. Secondly, comparing the data platform with the traditional data analysis platform, the platform can quickly process high-correlation and complex data, and reduce the response time of data analysis and processing. The platform has good practical value in big data service processing.

Research on Big Data Analysis Platform of Power Grid Enterprise Accounting Based on Cloud Computing

With the development of smart grids, big data is generated in all aspects of the power system, and the traditional financial system is no longer able to accurately analyze the large amount of financial data. In this paper, based on the problems arising from the traditional financial system decision-making process, the decision-making of grid financial system based on big data and cloud computing is proposed. Through the analysis of the big data characteristics of the power system, the application of big data and cloud computing in the smart grid is clarified, and the external application oriented to the customer group and the internal application oriented to the system flow are explored. Taking the operation and maintenance cost of an actual substation in the power grid as an example, the paper uses the idea of big data to conduct scientific research on its operation and maintenance cost, which intuitively reflects the application of big data, and makes the readers clearer and vividly see big data. How to apply it to accelerate the application development of big data in China.

Automatic Deep Extraction of Robust Dynamic Features for Industrial Big Data Modeling and Soft Sensor Application

Dynamic is one of the main bottlenecks in the industrial soft sensor application, due to the difficulties in representing and extracting dynamic data features. Meanwhile, an end-to-end deep network owns the ability to characterize sequence data information, but its fitting ability requires improvements in practical applications. In this article, an ensemble tree model with transferable and robust dynamic features extracted by a newly developed automatic dynamic feature extractor is proposed. First, the dynamic feature extractor with an encoding–decoding structure can provide effective dynamic features, which is equivalent to crossing and nonlinear mapping of sequences under the supervision of a decoder. Meanwhile, a new “regularization” method by smoothing dynamic features based on attention weights is proposed to denoise and alleviate the overfitting of the regressor after adding new features. Then, the extracted dynamic features can be transferred to the regressor with strong generalization ability, which takes into account the feature extraction of the deep network and the generalization of strong models. Finally, application results on a debutanizer distillation process show that the incorporation of robust dynamic features can significantly improve the soft sensing performance, compared to traditional methods. Moreover, the proposed model is further implemented through a cloud computing platform for industrial big data analytics.

A Survey on Vertical and Horizontal Scaling Platforms for Big Data Analytics

There is no doubt that we are entering the era of big data. The challenge is on how to store, search, and analyze the huge amount of data that is being generated per second. One of the main obstacles to the big data researchers is how to find the appropriate big data analysis platform. The basic aim of this work is to present a complete investigation of all the available platforms for big data analysis in terms of vertical and horizontal scaling, and its compatible framework and applications in detail. Finally, this article will outline some research trends and other open issues in big data analytics

Open Source Platform for Big Data Exploration and Analysis

Despite the enormous potential of big data, it is a relatively new issue for many companies, particularly for those of smaller size that looks at this as a challenge, is unattainable and only possible for companies with high financial capacity. However, open source software presents itself as an excellent alternative for these companies, which will allow them to exploit the high volume of data they have at their disposal. In this sense, this study presents a proposal for an architecture based exclusively on open source software that includes the entire value chain of big data, from data collection to data analysis. This architecture was tested considering three emerging scenarios in which big data become very relevant and challenging, namely for mobile analytics, network analytics, and mobile analytics.

IoT Analysis and Processing System using Sensed Data for Laboratory Safety Management

This study proposes and implements a real-time message analysis processing system using sensed data for safety management of a laboratory composed of Internet of Things (IoT) sensor devices. Existing IoT systems have difficulty storing data in a relational database system due to the different data formats between sensor devices, and they also have preprocessing overhead due to nonstandard communications protocols. Furthermore, they are centralized systems with unstable service continuity, and it is difficult to add new functions or extend the system because of the complex connections among sensor devices and services. Studies on real-time analysis and processing of large volumes of sensor data are also insufficient. To address these problems, the proposed system uses IoT sensor devices, based on standard communications protocols, and message queue brokers based on standard messaging protocols. The message queue brokers can facilitate adding sensor devices or services by delivering the sensor data from the gateway and central server to the services. Services can continue, even in the event of network trouble, because the services that had been provided from the central server are now provided through smart gateways. The central server can provide risk analysis and prevention services for laboratory safety management through Apache Spark, a real-time big data analysis and processing platform. Through the experiments in this study, the proposed real-time message analysis processing system for laboratory safety management was implemented. The message queue brokers between the standard Message Queuing Telemetry Transport (MQTT) protocol and the smart gateway receiving the data were built into sensor devices equipped with network and sensor modules. Exchange routing keys are also specified by topic and service for each sensor device in line with the laboratory safety management environment. Data transmission processing is handled by a federated connection of the gateway brokers and the central server brokers.

Yugong

Companies like Alibaba operate tens of data centers (DCs) across geographically distributed locations. These DCs collectively provide the storage space and computing power for the company, storing EBs of data and serving millions of batch analytics jobs every day. In Alibaba, as our businesses grow, there are more and more cross-DC dependencies caused by jobs reading data from remote DCs. Consequently, the precious wide area network bandwidth becomes a major bottleneck for operating geo-distributed DCs at scale. In this paper, we present Yugong --- a system that manages data placement and job placement in Alibaba's geo-distributed DCs, with the objective to minimize cross-DC bandwidth usage. Yugong uses three methods, namely project placement, table replication, and job outsourcing, to address the issues of high bandwidth consumption across the DCs. We give the details of Yugong's design and implementation for the three methods, and describe how it cooperates with other systems (e.g., Alibaba's big data analytics platform and cluster scheduler) to improve the productivity of the DCs. We also report comprehensive performance evaluation results, which validate the design of Yugong and show that significant reduction in cross-DC bandwidth usage has been achieved.

BigData Analysis in Healthcare: Apache Hadoop , Apache spark and Apache Flink

Introduction: Health care data is increasing. The correct analysis of such data will improve the quality of care and reduce costs. This kind of data has certain features such as high volume, variety, high-speed production, etc. It makes it impossible to analyze with ordinary hardware and software platforms. Choosing the right platform for managing this kind of data is very important. The purpose of this study is to introduce and compare the most popular and most widely used platform for processing big data, Apache Hadoop MapReduce, and the two Apache Spark and Apache Flink platforms, which have recently been featured with great prominence.Material and Methods: This study is a survey whose content is based on the subject matter search of the Proquest, PubMed, Google Scholar, Science Direct, Scopus, IranMedex, Irandoc, Magiran, ParsMedline and Scientific Information Database (SID) databases, as well as Web reviews, specialized books with related keywords and standard. Finally, 80 articles related to the subject of the study were reviewed.Results: The findings showed that each of the studied platforms has features, such as data processing, support for different languages, processing speed, computational model, memory management, optimization, delay, error tolerance, scalability, performance, compatibility, Security and so on. Overall, the findings showed that the Apache Hadoop environment has simplicity, error detection, and scalability management based on clusters, but because its processing is based on batch processing, it works for slow complex analyzes and does not support flow processing, Apache Spark is also distributed as a computational platform that can process a big data set in memory with a very fast response time, the Apache Flink allows users to store data in memory and load them multiple times and provide a complex Fault Tolerance mechanism Continuously retrieves data flow status.Conclusion: The application of big data analysis and processing platforms varies according to the needs. In other words, it can be said that each technology is complementary, each of which is applicable in a particular field and cannot be separated from one another and depending on the purpose and the expected expectation, and the platform must be selected for analysis or whether custom tools are designed on these platforms.

Big Data Analytics Platforms for Electric Vehicle Integration in Transport Oriented Smart Cities

Electric vehicles (EVs) are key players for transport oriented smart cities (TOSC) powered by smart grids (SG) because they help those cities to become greener by reducing vehicle emissions and carbon footprint. In this article, the authors analyze different use-cases to show how big data analytics (BDA) can play vital role for successful electric vehicle (EV) to smart grid (SG) integration. Followed by this, this article presents an edge computing model and highlights the advantages of employing such distributed edge paradigms towards satisfying the store, compute and networking (SCN) requirements of smart EV applications in TOSCs. This article also highlights the distinguishing features of the edge paradigm, towards supporting BDA activities in EV to SG integration in TOSCs. Finally, the authors provide a detailed overview of opportunities, trends, and challenges of both these computing techniques. In particular, this article discusses the deployment challenges and state-of-the-art solutions in edge privacy and edge forensics.

Augmented Artificial Intelligence Improves Data Analytics in Heavy-Oil Reservoirs

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper SPE 193650, “Augmented-Artificial-Intelligence Solutions for Heavy-Oil Reservoirs: Innovative Work Flows That Build From Smart Analytics, Machine Learning, and Expert-Based Systems,” by David Castineira, Xiang Zhai, and Hamed Darabi, Quantum Reservoir Impact Group, prepared for the 2018 SPE International Heavy Oil Conference and Exhibition, Kuwait City, Kuwait, 10–12 December. The paper has not been peer reviewed. Recently, many heavy-oil fields have seen exponentially higher volumes of data made available as a result of omnipresent connectivity. Existing data platforms have focused traditionally on solving the problem of data storage and access. The more-complex problem of true knowledge discovery and systematic value creation from the massive amount of data is less frequently addressed. The authors of this paper propose a novel work flow for the problem of building intelligent data analytics in heavy-oil fields. Introduction Optimal reservoir management for heavy-oil reservoirs requires systematic solutions that combine both engineering ability and advanced analytics. The authors believe that this requirement is addressed by what they call augmented artificial intelligence (AAI), a process inspired by the intelligence-amplification concept in which machine learning and human expertise are combined to improve solutions derived by systems that learn without any type of input from engineers or geoscientists. Practical deployment of AAI will involve automated work flows that use solid technical expertise and proven processes to transform field data into more-effective reservoir-management solutions. Even with rapid data-preprocessing solutions in place, developing an optimal reservoir-management framework for heavy-oil assets is inherently complex. Identifying key recovery obstacles (KROs) and field-development plans (FDPs) typically takes many months, involving a large team of experts and the construction of sophisticated full-field simulation models. The recommendation is that automated work flows and AAI solutions are combined to identify those KROs rapidly and prepare robust FDPs that increase production and optimize current operations. Perhaps the less-intuitive step in developing systematic solutions for heavy-oil fields is the process of developing a quantitative reservoir diagnostic framework. This process must build from big-data analytics platforms and an array of analytical, numerical, and empirical models combined to deliver a catalog of KROs affecting field performance. To this end, the entire historical set of well, field, and reservoir data must be processed and input into this diagnostics platform. Once the KROs are understood, the next step is to translate the diagnostics into detailed action plans in the field that can generate production, reserves, or capital-efficiency improvements. This paper aims to offer an alternative approach to traditional work flows that identify recovery obstacles and development opportunities in heavy-oil fields by labor-intensive solutions. In contrast, the authors propose a systematic framework that provides three key advantages: Execution time is fast, and an initial opportunity inventory can be generated. The user can choose from multiple algorithms and methods to customize the technology to unique field/reservoir complexities. The core algorithms are data-driven, integrate multidisciplinary data sets, and leave little room for the biases of the user, which allows for a consistent and repeatable analysis.

Cloud-based big data analytics platform using algorithm templates for the manufacturing industry

ABSTRACTSpecialised knowledge on analytics algorithms is necessary to conduct a data analysis; thus, there are limits to how a manufacturing manager who is not a data analysis expert conducts analysis tasks. This study proposed a manufacturing-specialised data analytics algorithm template and a cloud-based big data analytics platform to resolve such a problem and to enable those not specialising in data analysis to perform analysis tasks. The manufacturing-specialised data analytics algorithm template is a concept in which data characteristics are considered according to manufacturing problems to systematically provide the pre-processing, data mining, and visualisation algorithms required for conducting analysis tasks. A cloud-based big data analytics platform is designed to use algorithm templates for analysis and is capable of not only storing and managing data per company but also creating analysis models and confirming results. The platform consists of the following: inter-connected middleware, data storage, a batch manager, a data analytics modeller, and an analysis engine. This study applied the proposed manufacturing-specialised data analytics algorithm template and cloud-based big data analytics platform to the machine problem-solving process executed by mobile phone component production companies. It also sought study limitations and offered development plans for future research.

Orthodontics in the era of big data analytics

The objective of this report was to provide an overview of the current landscape of big data analytics in the healthcare sector, introduce various approaches of machine learning and discuss potential implications in the field of orthodontics. With the increasing availability of data from various sources, the traditional analytical methods may not be conducive anymore for examining clinical outcomes. Machine-learning approaches, which are algorithms trained to identify patterns in large data sets, are ideally suited to facilitate data-driven decision making. The field of orthodontics is particularly ripe for embracing the big data analytics platform to improve decision making in clinical practice. The availability of omics data, state-of-the-art imaging and potential for establishing large clinical data repositories have favourably positioned the specialty of orthodontics to deliver personalized and precision orthodontic care. Specifically, we discuss about next-generation sequencing, radiomics in the context of CBCT imaging, and how centralized data repositories can enable real-time data pooling from multiple sources.

Analyzing the impacts of urban expansion on air pollution in Vietnam using the SEAP platform

The relationship between urbanization and air pollution was discovered in many studies. In this study, we analyzed the impacts of urban expansion on air pollution in Vietnam using remotely-sensed data from 2004 to 2015. In this period of time, Vietnam urban square was increased from 4623 km2 in 2004 to 5094 km2 in 2015. Besides, there is a clear difference between the average PM2.5 concentration value of urban areas and non-urban areas in Vietnam, urban PM2.5 values are generally higher than in rural areas for years. In this study, we use the SEAP (big Spatial data Exploration and Analysis Platform) platform to analyze and store data.

Haze alarm visual map (HazeViz): an intelligent haze forecaster

The haze problem has intensified in recent years. The particulate matter of less than 10 microns in size, PM10 is the dominant air pollutant during haze. In this paper, we present the development of HazeViz, a Haze Alarm Visual Map forecaster, which is based on PM10. The intelligent web application allows users to visualize the pattern of PM10 in a region, forecasts PM10 value and alarms bad haze condition. HazeViz was developed using HTML, Java Script, PHP, MySQL, R Programming and Fusionex Giant. The SARIMA statistical forecasting models that underlie the application were developed using R. The PM10 trend analysis, and the consequential map and chart visualizations were implemented on the Fusionex GIANT Big Data Analytics platform. HazeViz was developed in the context of the Klang Valley, our case study. The dataset was obtained from Department of Environment Malaysia, which contains a total of 157,680 hourly PM10 data for six stations in Klang Valley, for the years 2013 to 2015. The SARIMA models were developed using maximum daily PM10 data for 2013 and 2014, and the 2015 data was used to validate the model. The fitting models were determined based on the Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE). While the selected models were implemented in HazeViz and successfully deployed on the web, the results show that the selected models have MAPE ranging between 35 percent and 45 percent, which implies that the models are still far from robust. Future work can consider augmented SARIMA models that can yield improved results.

Research on Key Technologies of E-commerce Big Data Analysis Platform

This paper designs and implements the E-commerce big data analysis platform, which is mainly based on the data of commodities and sales of E-commerce sales platforms, providing a platform for intelligent analysis. The platform provides an interface for users to operate. It can efficiently obtain the necessary business information for e-commerce users and provide decision support. The system can be divided into four parts: crawler system, storage system, offline data analysis system and user interaction system. The storage system adopts the Hadoop ecosystem. Hadoop’s HDFS are highly fault-tolerant, suitable for processing GB, TB, or even PB-level data, and can be expanded horizontally, and deployed on multiple inexpensive machines, the platform is Hadoop-based. HBase and Hive will handle and use data more efficiently. The offline analysis system adopts the Spark framework. That Spark is based on RDD (Elastic Distributed) set data processing. It can connect various data sources of HBase and Hive. Because the data are loaded into the memory of the cluster host, and quickly iterated, it is suitable for multiple rounds of computing tasks such as machine learning. Spark-SQL can provide SQL-like operations on structured or semi-structured data, which can greatly improve the efficiency of off-line data analysis.