Traditional Data Warehouse Research Articles

A Model for Enhancing Unstructured Big Data Warehouse Execution Time

Traditional data warehouses (DWs) have played a key role in business intelligence and decision support systems. However, the rapid growth of the data generated by the current applications requires new data warehousing systems. In big data, it is important to adapt the existing warehouse systems to overcome new issues and limitations. The main drawbacks of traditional Extract–Transform–Load (ETL) are that a huge amount of data cannot be processed over ETL and that the execution time is very high when the data are unstructured. This paper focuses on a new model consisting of four layers: Extract–Clean–Load–Transform (ECLT), designed for processing unstructured big data, with specific emphasis on text. The model aims to reduce execution time through experimental procedures. ECLT is applied and tested using Spark, which is a framework employed in Python. Finally, this paper compares the execution time of ECLT with different models by applying two datasets. Experimental results showed that for a data size of 1 TB, the execution time of ECLT is 41.8 s. When the data size increases to 1 million articles, the execution time is 119.6 s. These findings demonstrate that ECLT outperforms ETL, ELT, DELT, ELTL, and ELTA in terms of execution time.

Temporal Graph Cube

Data warehouse and OLAP (Online Analytical Processing) are effective tools for decision support on traditional relational data and static multidimensional network data. However, many real-world multidimensional networks are often modeled as temporal multidimensional networks, where the edges in the network are associated with temporal information. Such temporal multidimensional networks typically cannot be handled by traditional data warehouse and OLAP techniques. To fill this gap, we propose a novel data warehouse model, named <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathsf {Temporal{\kern3.0pt}Graph{\kern3.0pt}Cube}$</tex-math></inline-formula> , to support OLAP queries on temporal multidimensional networks. Through supporting OLAP queries in any time range, users can obtain summarized information of the network in the time range of interest, which cannot be derived by using traditional static graph OLAP techniques. We propose a segment-tree based indexing technique to speed up the OLAP queries, and also develop an index-updating technique to maintain the index when the temporal multidimensional network evolves over time. In addition, we also propose a novel concept called <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathsf {similarity{\kern3.0pt}of{\kern3.0pt}snapshots}$</tex-math></inline-formula> which shows a strong correlation with the efficiency of indexing technique and can provide a good reference on the necessity of building the index. The results of extensive experiments on two large real-world datasets demonstrate the effectiveness and efficiency of the proposed method.

Traditional Vs Cloud Data warehouse – Comparative Analysis &amp; Survey paper

In recent years due to the emerging of technologies, data amount is getting increase day by day. The mode and methods of data handling are getting upgraded. Prediction analysis seems to be very difficult but it generates interesting results. Various sectors like health, transport, education and economics are providing large amount of data. Recent developments in web technologies made it possible for operator and researcher to analyze and forecast that huge amount of data. The domain of BI (business intelligence) is core technology which help its users to extract meaningful information for decision making regarding any business. Data warehouse offers an insight view of business processes using previously provided data. Still, this traditional data warehouse system is not suitable for data analysis as it doesn't meet the requirements of evolving industries of this age. It cannot provide the up or down scale. Plus, it cannot handle the growing number of users. A new form of data warehouse has developed having better design and features, known as cloud data warehouse. This system has evolved in recent years, which directly affects the business and application domain as well. It has evolved in such a way to control large-scale data and provide up and down scale of business anytime. Moreover, it can handle a large number of users. In this review paper, comparison of traditional and cloud data warehouse is provided.

Design and Implementation of an Efficient Electronic Bank Management Information System Based Data Warehouse and Data Mining Processing

The quantity of electronic bank data grows exponentially with development of Information Technology (IT). The size of these data is impossible for traditional database and human analyst to come up with interesting information that will help in process of decision making. Management Information System (MIS) based Data warehouse (DW) and Data Mining (DM) techniques support the development of IT and process of management decision-making. But the traditional DW size make the query complex, which may cause unacceptable delay in decision support queries. Thus, in this paper an Efficient Electronic Bank MIS based DW and Mining Processing (EEBMIS-DWMP) was developed with cluster and non-cluster indexed view to provide decision-makers with both best response time and precise information. Also, analysis of the multilayer perception neural network, naïve Bayes, random forest, logistic regression, support vector machine and C5.0 on a real-world data of bank was done to improve effectiveness for campaign by analyzing the most useful features that influence campaign success. Results offer how the proposed EEBMIS-DWMP developed bank organizations by comparing performance of system with and without index view in terms of balance accuracy, accuracy, precision, recall, mean absolute error, root mean square error, F measure and running time. Conclusions from results offers that EEBMIS-DWMP can construct a database for each customer, a storage system that integrates data from a variety of sources into a single unified framework, decrease errors and time required to prepare financial reports, quickly access for information, analysis of data in multivariate, accurate prediction of competent, profitability segmentation.

A heterogeneous multi-modal medical data fusion framework supporting hybrid data exploration.

Industry 4.0 era has witnessed that more and more high-tech and precise devices are applied into medical field to provide better services. Besides EMRs, medical data include a large amount of unstructured data such as X-rays, MRI scans, CT scans and PET scans, which is still continually increasing. These massive, heterogeneous multi-modal data bring the big challenge to finding valuable data sets for healthcare researchers and other users. The traditional data warehouses are able to integrate the data and support interactive data exploration through ETL process. However, they have high cost and are not real-time. Furthermore, they lack of the ability to deal with multi-modal data in two phases—data fusion and data exploration. In the data fusion phase, it is difficult to unify the multi-modal data under one data model. In the data exploration phase, it is challenging to explore the multi-modal data at the same time, which impedes the process of extracting the diverse information underlying multi-modal data. Therefore, in order to solve these problems, we propose a highly efficient data fusion framework supporting data exploration for heterogeneous multi-modal medical data based on data lake. This framework provides a novel and efficient method to fuse the fragmented multi-modal medical data and store their metadata in the data lake. It offers a user-friendly interface supporting hybrid graph queries to explore multi-modal data. Indexes are created to accelerate the hybrid data exploration. One prototype has been implemented and tested in a hospital, which demonstrates the effectiveness of our framework.

Data variety, come as you are in multi-model data warehouses

Multi-model DBMSs (MMDBMSs) have been recently introduced to store and seamlessly query heterogeneous data (structured, semi-structured, graph-based, etc.) in their native form, aimed at effectively preserving their variety. Unfortunately, when it comes to analyzing these data, traditional data warehouses (DWs) and OLAP systems fall short because they rely on relational DBMSs for storage and querying, thus constraining data variety into the rigidity of a structured, fixed schema. In this paper, we investigate the performances of an MMDBMS when used to store multidimensional data for OLAP analyses. A multi-model DW would store each of its elements according to its native model; among the benefits we envision for this solution, that of bridging the architectural gap between data lakes and DWs, that of reducing the cost for ETL, and that of ensuring better flexibility, extensibility, and evolvability thanks to the combined use of structured and schemaless data. To support our investigation we define a multidimensional schema for the UniBench benchmark dataset and an ad-hoc OLAP workload for it. Then we propose and compare three logical solutions implemented on the PostgreSQL multi-model DBMS: one that extends a star schema with JSON, XML, graph-based, and key–value data; one based on a classical (fully relational) star schema; and one where all data are kept in their native form (no relational data are introduced). As expected, the full-relational implementation generally performs better than the multi-model one, but this is balanced by the benefits of MMDBMSs in dealing with variety. Finally, we give our perspective view of the research on this topic.

The contribution of linked open data to augment a traditional data warehouse

The arrival of Big Data has contributed positively to the evolution of the data warehouse (DW ) technology. This gives birth of augmented DW s that aim at maximizing the effectiveness of existing ones. Various augmentation scenarios have been proposed and adopted by firms and industry covering several aspects such as new data sources (e.g., Linked Open Data (LOD), social, stream and IoT data), data ingestion, advanced deployment infrastructures, programming paradigms, data visualization. These scenarios allow companies reaching valuable decisions. By examining traditional DW s, we realized that they do not fulfill all decision-maker requirements since data sources alimenting a target DW are not rich enough to capture Big Data. The arrival of LOD era is an excellent opportunity to enrich traditional DW s with a new V dimension: Value. In this paper, we first conceptualize the variety of internal and external sources and study its effect on the ETL phase to ease the value capturing. Secondly, a Value-driven approach for the DW design is discussed. Thirdly, three realistic scenarios for integrating LOD in the DW landscape are given. Finally, experiments are conducted showing the added value by augmenting the existing DW environment with LOD.

Lake Data Warehouse Architecture for Big Data Solutions

Traditional Data Warehouse is a multidimensional repository. It is nonvolatile, ‎subject-oriented, integrated, time-variant, and non-‎operational data. It is gathered from multiple ‎heterogeneous data ‎sources. We need to adapt traditional Data Warehouse architecture to deal with the new ‎challenges imposed by the abundance of data and the current big data characteristics, containing ‎volume, value, variety, validity, volatility, visualization, variability, and venue. The new ‎architecture also needs to handle existing drawbacks, including availability, scalability, and ‎consequently query performance. This paper introduces a novel Data Warehouse architecture, named Lake ‎Data Warehouse Architecture, to provide the traditional Data Warehouse with the capabilities to ‎overcome the challenges. ‎Lake Data Warehouse Architecture depends on merging the traditional Data Warehouse architecture ‎with big data technologies, like the Hadoop framework and Apache Spark. It provides a hybrid ‎solution in a complementary way. The main advantage of the proposed architecture is that it ‎integrates the current features in ‎traditional Data Warehouses and big data features acquired ‎through integrating the ‎traditional Data Warehouse with Hadoop and Spark ecosystems. Furthermore, it is ‎tailored to handle a tremendous ‎volume of data while maintaining availability, reliability, and ‎scalability.‎

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter.

A THEORITICAL EXPLORATION OF DATA MANAGEMENT AND INTEGRATION IN ORGANISATION SECTORS

Big data development is a disturbing issue that will affect enterprise across various sectors. The increase of data volume, high speed of data generation and increasing rate of different data from heterogeneous sources have led to difficulties in data management. This paper first reviews different aspects of big data management, including data integration and traditional data warehouse, and their associated challenges. The problems include increase of redundant data, data accessibility, time consumption in data modelling and data movement from heterogeneous sources into a central database, especially in the big data environment. We then propose a logical data management approach using RESTview technology to integrate and analyse data, without fully adopting traditional ETL processes. Data that for governance, corporate, security or other restriction reasons cannot be copied or moved, can easily be accessed, integrated and analysed, without creating a central repository. Data can be kept in its original form and location, eliminating the movement of data, significantly speeding up the process and allowing for live data interrogation. It may not be the practical solution for every situation but, it is a feasible solution that is comparably cost effective.

Design a Data Warehouse Schema from Document-Oriented database

Traditional data warehouses are unable to meet the growing needs of the modern enterprise to integrate and analyze a wide variety of data generated by social, mobile and sensor sources. What is remarkable is that many companies have changed their data storage in NoSQL databases. Given the important role played by this type of databases, it is necessary to study NoSQL databases as a data source for the modeling and the implementation of data warehouses. This paper proposes a method to design the data warehouse schema from schema free databases known as NoSQL databases. Our proposal starts with the extraction of schemes from document-oriented databases as an example of NoSQL database. This extraction is performed based on the MapReduce paradigm. Then, it defines the structure identification graph from the extracted schemes. This graph will help the designer to identify the multidimensional concepts for each schema extracted in order to design the global data warehouse schema.

Datawarehouse Versus Datalake

For decades, datawarehouses functioned as fundamental repository of integrated data which facilitated informed decisions. In the recent past, the field of data management has witnessed a cataclysmic change with the arrival of big data. Big data has created a need to process high volume and variety of data coupled with the desire for faster access to data. Datalakes are widely employed to meet these requirements and touted as a one stop shop to fulfill all the data requirements of an organization. This assumption has questioned the role of the traditional datawarehouse. This paper describes the essential characteristics of datawarehouses and datalakes. It also compares and contrasts the two systems and concludes that the two are not competing but complementary technologies. Together they can effectively and proficiently serve the data management needs of an organization.

Multiple Decisional Query Optimization in Big Data Warehouse

Data warehousing (DW) area has always motivated a plethora of hard optimization problem that cannot be solved in polynomial time. Those optimization problems are more complex and interesting when it comes to multiple OLAP queries. In this article, the authors explore the potential of distributed environment for an established data warehouse, database-related optimization problem, the problem of Multiple Query Optimization (MQO). In traditional DW materializing views is an optimization technic to solve such problem by storing pre-computed join or frequently asked queries. In this era of big data this kind of view materialization is not suitable due to the data size. In this article, the authors tackle the problem of MQO on distributed DW by using a multiple, small, shared and easy to maintain shared data. The evaluation shows that, compared to available default execution engine, the authors' approach consumes on average 20% less memory in the Map-scan task and it is 12% faster regarding the execution time of interactive and reporting queries from TPC-DS.

A Comparative Analysis of Traditional and Cloud Data Warehouse

In the age of emerging technologies, the amount of data is increasing very rapidly. With the passage of time, the methods of data handling are getting improved. Prediction analysis is quite a tough task, but it also yields interesting results. Different sectors like financial services, transportation, health and education are generating large amount of data. The emergence of web 2.0 (social web) made it possible for users and researchers to analyze and predict huge amount of data. The domain of Business Intelligence is core technology for users who want to extract useful information for decision making regarding their businesses. Data warehouse provides an insight into the business processes using the historical data. However, traditional data warehouse may not be suitable for the data analysis needs because of the evolving requirement of industry. It cannot be scaled up or down. Moreover, it cannot handle the increasing number of users. A new kind of data warehouse with design and implementation aspects has been emerged, called as cloud data warehouse. The cloud data warehouse model has evolved with the passage of time, which affects the application and business domains as well. The cloud data warehouse has evolved to control the large scale data. It can be scaled up or down at any time and also it has no limitation on increasing number of users. In this review paper, we have compared traditional and cloud data warehouse. We can conclude that the ultimate future of data warehouse is cloud data warehouse.

Survey data and metadata modelling using document-oriented NoSQL

Survey data that are collected from year to year have metadata change. However it need to be stored integratedly to get statistical data faster and easier. Data warehouse (DW) can be used to solve this limitation. However there is a change of variables in every period that can not be accommodated by DW. Traditional DW can not handle variable change via Slowly Changing Dimension (SCD). Previous research handle the change of variables in DW to manage metadata by using multiversion DW (MVDW). MVDW is designed using relational model. Some researches also found that developing nonrelational model in NoSQL database has reading time faster than the relational model. Therefore, we propose changes to metadata management by using NoSQL. This study proposes a model DW to manage change and algorithms to retrieve data with metadata changes. Evaluation of the proposed models and algorithms result in that database with the proposed design can retrieve data with metadata changes properly. This paper has contribution in comprehensive data analysis with metadata changes (especially data survey) in integrated storage.

A bottom-up approach for creating goal hierarchy using OLAP query recommendation technique

Traditionally, data warehouses have been designed guided by an analysis of the underlying data sources. On the other hand, goal-oriented approaches build a goal decomposition structure which is further mapped to a data warehouse model. Goal-oriented approaches address the requirements of the decision makers. In this paper, we propose to build a goal decomposition structure for traditional data warehouses. As a result it can be assessed whether the existing data warehouse model satisfies the decision makers' requirements. The approach is bottom-up wherein the tasks are identified first and the path to the strategic goal is built later. The tasks themselves are identified using OLAP query recommendation technique.

Is Datawarehouse Relevant in the Era of Big Data?

For a long time, datawarehouses were built as a central repository of integrated data that were used to make informed decisions. However, there is an upheaval in the data management landscape in the last few years with the advent of big data that has questioned the need to build a traditional datawarehouse. This paper presents the idea that the traditional concept of datawarehousing is still relevant in the era of big data. However, it needs to be evolved and augmented with big data technologies to better serve the data integration needs of an organization and cope with increased variety, volume and velocity with which the data is being generated and analyzed. This paper also discusses the challenges that are currently faced by traditional datawarehouses and provides suggestions on how big data technologies can be leveraged upon to mitigate few of these challenges.

E-Science technologies in a workflow for personalized medicine using cancer screening as a case study.

Objective:We provide an e-Science perspective on the workflow from risk factor discovery and classification of disease to evaluation of personalized intervention programs. As case studies, we use personalized prostate and breast cancer screenings. Materials and Methods:We describe an e-Science initiative in Sweden, e-Science for Cancer Prevention and Control (eCPC), which supports biomarker discovery and offers decision support for personalized intervention strategies. The generic eCPC contribution is a workflow with 4 nodes applied iteratively, and the concept of e-Science signifies systematic use of tools from the mathematical, statistical, data, and computer sciences. Results:The eCPC workflow is illustrated through 2 case studies. For prostate cancer, an in-house personalized screening tool, the Stockholm-3 model (S3M), is presented as an alternative to prostate-specific antigen testing alone. S3M is evaluated in a trial setting and plans for rollout in the population are discussed. For breast cancer, new biomarkers based on breast density and molecular profiles are developed and the US multicenter Women Informed to Screen Depending on Measures (WISDOM) trial is referred to for evaluation. While current eCPC data management uses a traditional data warehouse model, we discuss eCPC-developed features of a coherent data integration platform. Discussion and Conclusion:E-Science tools are a key part of an evidence-based process for personalized medicine. This paper provides a structured workflow from data and models to evaluation of new personalized intervention strategies. The importance of multidisciplinary collaboration is emphasized. Importantly, the generic concepts of the suggested eCPC workflow are transferrable to other disease domains, although each disease will require tailored solutions.

Big Data and Cloud Computing: Trends and Challenges

Big data is currently one of the most critical emerging technologies. Big Data are used as a concept that refers to the inability of traditional data architectures to efficiently handle the new data sets. The 4V’s of big data – volume, velocity, variety and veracity makes the data management and analytics challenging for the traditional data warehouses. It is important to think of big data and analytics together. Big data is the term used to describe the recent explosion of different types of data from disparate sources. Analytics is about examining data to derive interesting and relevant trends and patterns, which can be used to inform decisions, optimize processes, and even drive new business models. Cloud computing seems to be a perfect vehicle for hosting big data workloads. However, working on big data in the cloud brings its own challenge of reconciling two contradictory design principles. Cloud computing is based on the concepts of consolidation and resource pooling, but big data systems (such as Hadoop) are built on the shared nothing principle, where each node is independent and selfsufficient. The integrating big data with cloud computing technologies, businesses and education institutes can have a better direction to the future. The capability to store large amounts of data in different forms and process it all at very large speeds will result in data that can guide businesses and education institutes in developing fast. Nevertheless, there is a large concern regarding privacy and security issues when moving to the cloud which is the main causes as to why businesses and educational institutes will not move to the cloud. This paper introduces the characteristics, trends and challenges of big data. In addition to that, it investigates the benefits and the risks that may rise out of the integration between big data and cloud computing.

Research in Big Data Warehousing using Hadoop

Traditional data warehouses have played a key role in decision support system until the recent past. However, the rapid growing of the data generation by the current applications requires new data warehousing systems: volume and format of collected datasets, data source variety, integration of unstructured data and powerful analytical processing. In the age of the Big Data, it is important to follow this pace and adapt the existing warehouse systems to overcome the new issues and challenges. In this paper, we focus on the data warehousing over big data. We discuss the limitations of the traditional ones. We present its alternative technologies and related future work for data warehousing