Hadoop Database Research Articles

Query Service for New ATLAS EventIndex System

The ATLAS EventIndex system consists of the catalogue of all events collected, processed or generated by the ATLAS experiment at the CERN LHC accelerator, and all associated software tools. The new system, developed for LHC Run 3, makes use of Apache HBase—the Hadoop database—and Apache Phoenix—an SQL/relational database layer for HBase—to store and access all the event metadata. The Query Service allows searches for and retrieval of information of interest. Its design is based on a server-client architecture with the sides communicating via a RESTful API. The OpenAPI Specification is used to describe the API and generate parts of the source code of the server and client. Based on selection criteria received from the client, the server executes SQL queries against the Phoenix tables, retrieves the data, and returns it to the client. The client—a command line utility following the UNIX/POSIXconforming program design style—can output results in various formats including plain text and JSON. The server and the client are developed in Java 8, with the use of the Jersey 2 JAX-RS API implementation and the Spring Framework. The service performance is suitable for interactive use with queries for up to 1000 events at a time.

Introducing an intelligent multi-level retrieval method for mineral resource potential evaluation result data

Abstract The geological data of the mineral resource potential evaluation results (MRPERs) are diverse and extremely large; efficiently retrieving data remains a challenging problem. In this work, a new way of using the Hadoop platform is proposed. The Hadoop distributed file system is used to store the massive data and construct the data storage model of geological and mineral resources. Using a distributed Hadoop database (HBase) that supports the fast query of a single record, it manages its metadata and retrieves the data of MRPERs quickly. At the same time, a multi-level index directory is designed to support the non-main key query on the HBase. This overcomes the shortcoming that the HBase only supports the simple index based on the main key and realizes the intelligent, efficient retrieval of MRPERs. The validity and feasibility of the proposed method are further verified by experiments using the MRPER data in the Institute of Mineral Resources, Chinese Academy of Geological Sciences.

Knowledge graph construction and Internet of Things optimisation for power grid data knowledge extraction

Abstract Problems exist in power grid data management that have unclear relationships, weak security and low accuracy. By analysing the knowledge graph construction characteristics of smart grid data information and knowledge extraction, the grid data management platform is reshaped architecturally, and the knowledge graph construction technology is embedded in the grid data management framework. For the aforementioned problems, the knowledge graph construction and Internet of Things optimisation framework of power grid data knowledge extraction are proposed in this article. Firstly, the semantic search (KGSS) algorithm based on the knowledge graph is used. The KGSS algorithm extracts knowledge from structured, semi-structured and unstructured grid data through the massively parallel processing acquisition model and Hadoop database, and constructs knowledge entities, attributes and inter-entity relationships. Then, it optimises and predicts through the knowledge graph construction and Internet of Things optimisation framework extracted from power grid data knowledge. Finally, the experimental results show that the accuracy rate of the KGSS algorithm is 92%. The experimental results also show that it provides new ideas and research directions for power grid data under big data in the future.

Using Hadoop, Analyze Airport Authority

A modern airport has access to a vast quantity of information, including the number of flights, arrival and departure times, flight routes, the number of airports that operate in each nation, a list of the current active airlines in each nation, etc. DBMS has so far been utilized to store and analyze the airport database. An airport database management system uses SQL to apply queries to all of the records. B-tree or distributed hash tables are used by My SQL to analyze the database. It is unable to analyze a large volume of data. Data is stored in a parallel database. This system's disadvantage is that it can't handle an increase in massive amounts of data to a significant degree. Therefore, the Hadoop database file system is employed to solve this issue. Hadoop uses HDFS and the Map-Reduce processing architecture to store vast amounts of data and provide the data in parallel. We made an effort to examine in-depth research of airline data sets, such as a list of airports in India, a list of airlines with no stops, a list of codeshare airlines by country with the most airports, and a list of active airlines in the United States. Here, we concentrated on processing large data sets in a distributed setting utilizing the hive component of the Hadoop ecosystem. The developers and business analysts will be helped by this effort when it comes to obtaining and handling their user requests

Building a novel physical design of a distributed big data warehouse over a Hadoop cluster to enhance OLAP cube query performance

Improving OLAP (Online Analytical Processing) query performance in a distributed system on top of Hadoop is a challenging task. An OLAP Cube query comprises several relational operations, such as selection, join, and group-by aggregation. It is well-known that star join and group-by aggregation are the most costly operations in a Hadoop database system. These operations indeed increase network traffic and may overflow memory; to overcome these difficulties, numerous partitioning and data load balancing techniques have been proposed in the literature. However, some issues remain questionable, such as decreasing the Spark stages and the network I/O for an OLAP query being executed on a distributed system. In a precedent work, we proposed a novel data placement strategy for a big data warehouse over a Hadoop cluster. This data warehouse schema enhances the projection, selection, and star-join operations of an OLAP query, such that the system’s query-optimizer can perform a star join process locally, in only one spark stage without a shuffle phase. Also, the system can skip loading unnecessary data blocks when executing the predicates. In this paper, we extend our previous work with further technical details and experiments, and we propose a new dynamic approach to improve the group-by aggregation. To evaluate our approach, we conduct some experiments on a cluster with 15 nodes. Experimental results show that our method outperforms existing approaches in terms of OLAP query evaluation time.

Analysis and Research Based on Instrument Drift Data

Instruments and meters are used to inspect, measure, control, analyze, calculate and display the physical quantity, chemical quantity, biomass, electrical parameters, geometric quantity, and movement status of the object under test. Its applications cover a wide range of industries, agriculture, transportation, science, and technology, etc., and can be divided into electrical instruments, optical instruments, laboratory instruments, analytical instruments, testing machines, industrial instruments, etc. It assumes the task of gatekeeper and guide in the construction of the national economy. The analysis and research techniques of this article are based on the professional data of power plant industrial instrumentation. This paper presents an overview of various analysis methods of instrument data, and in-depth analysis of time correlation and coverage analysis, outlier analysis, etc. We give the statistical analysis process of drift data, focusing on the process of the AFAL method; We also described the drift algorithm of random behavior and deviation behavior and explored and discussed the analysis tools and data analysis results of the power plant industry professional data. We have designed a set of application systems for instrument data import, export, storage, and analysis, which can store instrument calibration data onto a Hadoop database in a prescribed format. Which provides great reference values for the calibration of instrument data. Finally, it looks forward to the analysis and research direction and development trend of industrial big data.

An improved join‐free snowflake schema for ETL and OLAP of data warehouse

SummaryThe emergence of big data makes more and more enterprise change data management strategy, from simple data storage to OLAP query analysis; meanwhile, NoSQL‐based data warehouse receive more increasing attention than traditional SQL‐based database. By improving the JFSS model for ETL, this paper proposes the uniform distribution code (UDC), model identification code (MIC), standard dimension code (SDC), and attribute dimensional code (ADC); defines the data storage format of <Rowkey Compound Code, Sec (Value)>; and identifies the extraction, transformation, and loading strategies of data warehouse. Several experiments are carried out to analyze single record and range record queries as typical OLAP based on Hadoop database (HBase). The results show the proposed scheme can provide lower overhead than the traditional SQL‐based database while facilitating the scope and flexibility of data warehouse services.

Research on Visual Analysis of Agricultural Food Big Data Based on CiteSpace III

This study makes visual analysis on agricultural food big data retrieval literature by using the information visualization tool CiteSpace III and with the Web of Science™ core collection as data sources. The spatial and temporal distribution, research focus, major fields of study, research fronts and evolution paths on the research field of big data were analyzed by knowledge maps and literature research. The results of the research show that the research focus in future may include Hadoop Distributed File System, Hadoop Database, performance evaluation and medical research.

MHBase: A Distributed Real-Time Query Scheme for Meteorological Data Based on HBase

Meteorological technology has evolved rapidly in recent years to provide enormous, accurate and personalized advantages in the public service. Large volumes of observational data are generated gradually by technologies such as geographical remote sensing, meteorological radar satellite, etc. that makes data analysis in weather forecasting more precise but also poses a threat to the traditional method of data storage. In this paper, we present MHBase, (Meteorological data based on HBase (Hadoop Database), a distributed real-time query scheme for meteorological data based on HBase. The calibrated data obtained from terminal devices will be partitioned into HBase and persisted to HDFS (the Hadoop Distributed File System). We propose two algorithms (the Indexed Store and the Indexed Retrieve Algorithms) to implement a secondary index using HBase Coprocessors, which allow MHbase to provide high performance data querying on columns other than rowkey. Experimental results show that the performance of MHBase can satisfy the basic demands of meteorological business services.

A Hyperconnected Manufacturing Collaboration System Using the Semantic Web and Hadoop Ecosystem System

With the explosive growth of digital data communications in synergistic operating networks and cloud computing service, hyperconnected manufacturing collaboration systems face the challenges of extracting, processing, and analyzing data from multiple distributed web sources. Although semantic web technologies provide the solution to web data interoperability by storing the semantic web standard in relational databases for processing and analyzing of web-accessible heterogeneous digital data, web data storage and retrieval via the predefined schema of relational / SQL databases has become increasingly inefficient with the advent of big data. In response to this problem, the Hadoop Ecosystem System is being adopted to reduce the complexity of moving data to and from the big data cloud platform. This paper proposes a novel approach in a set of the Hadoop tools for information integration and interoperability across hyperconnected manufacturing collaboration systems. In the Hadoop approach, data is “Extracted” from the web sources, “Loaded” into a set of the NoSQL Hadoop Database (HBase) tables, and then “Transformed” and integrated into the desired format model with Hive's schema-on-read. A case study was conducted to illustrate that the Hadoop Extract-Load-Transform (ELT) approach for the syntax and semantics web data integration could be adopted across the global smartphone value chain.

Integrating of Data Using the Hadoop and R

Abstract The article offers a data integration model, which must be supported by a unified view of disparate data sources, management of integrity constraints, management of data manipulation and query executing operations, matching data from various sources, the ability to expand and set up new data sources. The proposed approach is the integration of Hadoop-based data and R, which is popular for processing statistical information. Hadoop database contains libraries, Distributed File System (HDFS), and resource management platform and implements a version of the MapReduce programming model for processing large-scale data. This model allows us to integrate various data sources at any level, by setting arbitrary links between circuit elements, constraints and operations.

Big Data Hadoop for Cloud-Based Enterprise Collaboration Systems

With the growing popularity of cloud computing, enterprises are turning their collaboration platforms towards Software as a Service (SaaS) applications over the Internet. Companies hosting cloud-computing services face the challenges of multi-dimensional information and knowledge from a variety of distributed sources. Semantic web technologies provide the solution to semantic interoperability of heterogeneous data. In addition, the NoSQL Hadoop database (HBase) is being adopted dramatically on the cloud for semantic data integration. This paper investigates a Hadoop Database (HBase) approach for semantic web information representation across distributed and collaborative enterprises systems.

대용량 스트리밍 센서데이터 환경에서 RDFS 규칙기반 병렬추론 기법

Recently, large-scale streaming sensor data have emerged due to explosive supply of smart phones, diffusion of IoT and Cloud computing technology, and generalization of IoT devices. Also, researches on combination of semantic web technology are being actively pushed forward by increasing of requirements for creating new value of data through data sharing and mash-up in large-scale environments. However, we are faced with big issues due to large-scale and streaming data in the inference field for creating a new knowledge. For this reason, we propose the RDFS rule based parallel reasoning scheme to service by processing large-scale streaming sensor data with the semantic web technology. In the proposed scheme, we run in parallel each job of Rete network algorithm, the existing rule inference algorithm and sharing data using the HBase, a hadoop database, as a public storage. To achieve this, we implement our system and evaluate performance through the AWS data of the weather center as large-scale streaming sensor data.

Research of Hadoop-based data flow management system

The data flow is very important at the field of network management, network security and network experiments. But with the rapidly growing of computer network data, traditional data flow management system has been unable to meet the current needs. Combined the features of data flow management systems and the advantages of Hadoop cloud computing platforms, this paper design and implement a Hadoop-based platform for distributed data flow management systems, using MapReduce to process the user request, using Hadoop distributed file system (HDFS) to manage the data flow files, and using Hadoop database (HBase) to manage the data flow information. Test shows that this system is better than the traditional data flow management system in aspects of efficiency, scalability and reliability. And it can meet the large demands for data flow management better.