Big Data Solutions Research Articles

Research on Big Data Mining and Solutions Based on Computer Cloud Computing

The application of computer cloud computing has profoundly changed the way the world works and brought people an everlasting experience. In order to adapt to the development direction of information technology, we need to deeply understand the current situation of big data application and study the innovative application of data mining technology.

Big Data and Digital Solutions: Laying the Foundation for Cardiovascular Population Management CME.

There are huge gaps in evidence-based cardiovascular care at the national, organizational, practice, and provider level that can be attributed to variation in provider attitudes, lack of incentives for positive change and care standardization, and observed uncertainty in clinical decision making. Big data analytics and digital application platforms-such as patient care dashboards, clinical decision support systems, mobile patient engagement applications, and key performance indicators-offer unique opportunities for value-based healthcare delivery and efficient cardiovascular population management. Successful implementation of big data solutions must include a multidisciplinary approach, including investment in big data platforms, harnessing technology to create novel digital applications, developing digital solutions that can inform the actions of clinical and policy decision makers and relevant stakeholders, and optimizing engagement strategies with the public and information-empowered patients.

GeoMatch

We develop GeoMatch as a novel, scalable, and efficient big-data pipeline for large-scale map matching on Apache Spark. GeoMatch improves existing spatial big-data solutions by utilizing a novel spatial partitioning scheme inspired by Hilbert space-filling curves. Thanks to its partitioning scheme, GeoMatch can effectively balance operations across different processing units and achieve significant performance gains. GeoMatch also incorporates a dynamically adjustable error-correction technique that provides robustness against positioning errors. We demonstrate the effectiveness of GeoMatch through rigorous and extensive empirical benchmarks that consider large-scale urban spatial datasets ranging from 166,253 to 3.78B location measurements. We separately assess execution performance and accuracy of map matching and develop a benchmark framework for evaluating large-scale map matching. Results of our evaluation show up to 27.25-fold performance improvements compared to previous works while achieving better processing accuracy than current solutions. We also showcase the practical potential of GeoMatch with two urban management applications. GeoMatch and our benchmark framework are open-source.

CLOUD COMPUTING AND BIG DATA: A COMPREHENSIVE ANALYSIS

The Internet plays an important role in the modern world. With the increasing usage of internet and web-based applications, various enterprises turn to big data solutions to cope up with the changing demand. In this big data era, data is the most valuable entity as it is mainly used by businesses for future prediction and decision making. With a substantial increase in the scale of big data, storing and managing the same is a great challenge. However, technologies such as cloud computing offer cost-effective, well-suited and consistent on-demand services that can be adopted by industries for big data and analytics. This paper aims to investigate various features, technologies available in big data and cloud computing, how cloud computing is used in par with big data. Also, the challenges faced in a cloud environment and big data and why the cloud is needed for big data is been discussed. Finally, some of the future directions for our research is been highlighted.

Suppression of Coherence Matrix Bias for Phase Linking and Ambiguity Detection in MTInSAR

Phase decorrelation, as one of the main error sources, limits the capability of interferometric synthetic aperture radar (InSAR) for deformation mapping over areas with low coherence. Although several methods have been realized to reduce decorrelation noise, for example, by phase linking and spatial and temporal filters, their performances deteriorate when coherence estimation bias exists. We present an arc-based approach that allows reconstructing unwrapped interval phase time-series based on iterative weighted least squares (WLS) in temporal and spatial domains. The main features of the method are that phase optimization and unwrapping can be jointly conducted by spatial and temporal iterative WLS and coherence matrix bias has negligible effects on the estimation. In addition, the linear formation makes the implementation suitable with small subset of interferograms, providing an efficient solution for future big SAR data. We demonstrate the effectiveness of the proposed method using simulated and real data with different decorrelation mechanisms and compare our approach with the state-of-art phase reconstruction methods. Substantial improvement can be achieved in terms of reduced root-mean-square error (RMSE) in the simulation data and increased density of coherent measurements in the real data.

How do job vacancy rates predict firm performance? A web crawling massive data perspective

Traditionally, the relationship between a firm's performance and its business strategy is studied using structured data taken from proxy statements and financial reports. However, there have been increasing efforts to explore the linkages between corporate outcomes and unstructured information, such as text or image/audio/video files. Until recently, semi-structured data had been largely overlooked. Given that a substantial amount of such data can be extracted using web crawler techniques and then processed using big data solutions, the current study employed this procedure to investigate whether dynamic job vacancy postings by Taiwanese publicly listed companies are associated with subsequent stock returns and operating ratios. We report that new job openings foreshadow a firm's operating performance, both indirectly, by boosting stock prices, and directly, by signaling positive developments. This finding remains robust to tests addressing endogeneity concerns and the adoption of alternative specifications. We thus shed light on the role of metadata in financial analysis.

Big Data in the Field of Logistics: A Retrospective Manner to Resolve

Big data a term created a huge change in the under currentan upcoming revolutionizing supply chain industry . The data is the new oil and gas for the modern word .SCM has also not left untouched with its Midas touch . The upcoming techniques of making decision to upgrade the profitability and data reverences. The algorithms of big data and its analytical excellence tools helped making better decisions to the upper hand decision maker and researchers .The situation of dealing a humongous and heterogenous data has been changed by these techniques . The old school SCM methods have taken a back seat in dealing these data sources. This paper is in advocacy of the present day techniques and create a path way to the explore the possibilities of the success of the big data solutions

Ultra-short-term forecasting for photovoltaic power plants and real-time key performance indicators analysis with big data solutions. Two case studies - PV Agigea and PV Giurgiu located in Romania

Nowadays, plenty of data is continuously pouring from the PhotoVoltaic Power Plants (PV) monitoring systems and sensors that could be successfully handled by big data technologies. This paper proposes a methodology that automatically collects the data logs from sensors installed on PV arrays, inverters and weather stations, checks the health status of the PV components, forecasts the generated power for each inverter based on its real operating conditions and the predicted irradiance and finally provides useful insights of the PV system based on the Key Performance Indicators (KPI) using big data technologies. The Ultra-Short-Term Forecast (USTF) algorithm provides the estimations of irradiance and generated power for the next 30 min and is applied on a sliding time window interval. The algorithm uses a Feed-Forward Artificial Neural Network (FF-ANN) and, to significantly reduce the number of iterations, we propose a backtracking adjustment of the learning rate that enables faster convergence reducing the computational time that is essential for USTF. Two data sets from PV Agigea 0.5 MW and PV Giurgiu 7.5 MW, located in the South-East and South of Romania, that consist in data logs from inverters and arrays, are used for simulation. The exhaustive analyses are performed for PV Agigea (including KPI calculation), while PV Giurgiu data set was mainly used to check the scalability and replicability of the algorithm.

Available techniques in hadoop small file issue

Hadoop is an optimal solution for big data processing and storing since being released in the late of 2006, hadoop data processing stands on master-slaves manner [1] that’s splits the large file job into several small files in order to process them separately, this technique was adopted instead of pushing one large file into a costly super machine to insights some useful information. Hadoop runs very good with large file of big data, but when it comes to big data in small files it could facing some problems in performance, processing slow down, data access delay, high latency and up to a completely cluster shutting down [2]. In this paper we will high light on one of hadoop’s limitations, that’s affects the data processing performance, one of these limits called “big data in small files” accrued when a massive number of small files pushed into a hadoop cluster which will rides the cluster to shut down totally. This paper also high light on some native and proposed solutions for big data in small files, how do they work to reduce the negative effects on hadoop cluster, and add extra performance on storing and accessing mechanism.

A multi-dimension framework for value creation through Big Data

Big Data represents a promising area for value creation and frontier research. The potential to extract actionable insights from Big Data has gained increasing attention of both academics and practitioners operating in several industries. Marketing domain has become from the start a field for experiments with Big Data approaches, even if the adoption of Big Data solutions does not always generate effective value for the adopters. Therefore, the gap existing between the potential of value creation embedded in the Big Data paradigm and the current limited exploitation of this value represents an area of investigation that this paper aims to explore. In particular, by following a systematic literature review, this study aims at presenting a framework that outlines the multiple value directions that the Big Data paradigm can generate for the adopting organizations. Eleven distinct value directions have been identified and then grouped in five dimensions (Informational, Transactional, Transformational, Strategic, Infrastructural Value), which constitute the pillars of the proposed framework. Finally, the framework has been also preliminarily applied in three case studies conducted within three Italian based companies operating in different industries (e-commerce, fast-moving consumer goods, and banking) in the final aim to see its applicability in real business scenarios.

F1 스코어를 이용한 한국어 감정 지수 연구

Digital companies are processing and managing their business relationships between their customers, suppliers and employees in digitalized way. These companies innovate the traditional social structure by applying digital technology to society. Enterprises build and utilize information and communication technology (ICT) as a platform, including cloud computing, artificial intelligence (AI), and big data solutions. Transformation into a digital company and using data analysis processes in an organization will create new values or services. In addition, analyzing the flow of data within an organization in an e-business environment is a necessary activity for corporate survival.<BR> The purpose of this study is to analyze the emotions of internal and external customers, which are diversified in atypical data analysis. Wheel of emotions(Plutchik, 1980) is used to classify emotions and prioritize emotion words within the categorized emotion sets. The Korean emotional word index process was studied by classifying and ranking the emotional words in the emotion set using the F1 score, a tool for measurement of effect. The emotion index was ranked in 200,000 comments using 190 generalized emotional words in the emotion set. Emotions such as anger and joy, fear and happiness can be identified through statistical values. In addition, it is confirmed that customer needs or product preference analysis can be applied to get more insights in the company"s point of view.

Lawyers and the Machine. Contemplating the Future of Litigation in the Age of AI

The possible impacts of artificial intelligence (AI) on the modern world constitute a complex field of study. In our analysis, we attempt to explore some possible consequences of the utilization of AI in the judicial field both as regarding adjudication, formerly exclusively reserved for human judges, and in the rendering of legal services by attorneys-at-law. We list the main factors influencing technology adoption and analyse the possible paths the automated management and solution of disputes may take. We conclude that the optimal outcome would be a cooperation of human and artificially intelligent factors. We also outline the conditions in which, following the abandonment of the principle of procedural fairness, AI may be directly utilized in judicial procedure. We conclude that big data solutions, such as social rating systems, are particularly concerning as they constitute a conceivable modality of deploying AI to solve litigious disputes without regard to fundamental human rights as understood today.

A multi-objective trade-off framework for cloud resource scheduling based on the Deep Q-network algorithm

Cloud computing, now a mature computing service, provides an efficient and economical solution for processing big data. As such, it attracts a lot of attention in academia and plays an important role in industrial applications. With the recent increase in the scale of cloud computing data centers, and improvements in user service quality requirements, the structure of the whole cloud system has become more complex, which has also made the resource scheduling management of these systems more challenging. Thus, the goal of this research was to resolve the conflict between cloud service providers (CSPs) who aim to minimize energy costs and those who seek to optimize service quality. Based on the excellent environmental awareness and online adaptive decision-making ability of deep reinforcement learning (DRL), we proposed an online resource scheduling framework based on the Deep Q-network (DQN) algorithm. The framework could make a trade-off of the two optimization objectives of energy consumption and task makespan by adjusting the proportion of the reward of different optimization objectives. Experimental results showed that this framework could effectively be used to make a trade-off of the two optimization objectives of energy consumption and task makespan, and exhibited obvious optimization effects compared with the baseline algorithm. Therefore, our proposed framework can dynamically adjust the optimization objective of the system according to the different requirements of the cloud system.

Big Data: Analytical Solutions, Research Challenges and Trends

The term Big Data refers to an extensive collections of digital data generating every second. Produced datasets come in structured, semi-structured, and unstructured formats throughout the world, which is difficult for the traditional database management systems to analyze. Recently, big data analytics emerges as an essential research area due to the popularity of the Internet and the advent of new Web technologies. This growing area of research represents a multi-disciplinary that attracts researchers from various research fields. Interested researchers are invited to design, develop, and implement several tools, technologies, architecture, and platforms for analyzing these large volumes of data. This paper begins with a brief introduction to big data and related concepts, including the main characteristics of big data, followed by discussions of the most significant open research challenges and emerging trends. Next, reviewing a study of big data analytics, the advantages of using big data solutions, and the preliminary assessments required before migrating from traditional solutions. Finally, presenting a review of the recent main applications to obtain a broad perspective of big data analytics.

Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues

The proliferation of multimedia devices over the Internet of Things (IoT) generates an unprecedented amount of data. Consequently, the world has stepped into the era of big data. Recently, on the rise of distributed computing technologies, video big data analytics in the cloud has attracted the attention of researchers and practitioners. The current technology and market trends demand an efficient framework for video big data analytics. However, the current work is too limited to provide a complete survey of recent research work on video big data analytics in the cloud, including the management and analysis of a large amount of video data, the challenges, opportunities, and promising research directions. To serve this purpose, we present this study, which conducts a broad overview of the state-of-the-art literature on video big data analytics in the cloud. It also aims to bridge the gap among large-scale video analytics challenges, big data solutions, and cloud computing. In this study, we clarify the basic nomenclatures that govern the video analytics domain and the characteristics of video big data while establishing its relationship with cloud computing. We propose a service-oriented layered reference architecture for intelligent video big data analytics in the cloud. Then, a comprehensive and keen review has been conducted to examine cutting-edge research trends in video big data analytics. Finally, we identify and articulate several open research issues and challenges, which have been raised by the deployment of big data technologies in the cloud for video big data analytics. To the best of our knowledge, this is the first study that presents the generalized view of the video big data analytics in the cloud. This paper provides the research studies and technologies advancing video analyses in the era of big data and cloud computing.

Big data solutions for CMS computing monitoring and analytics

The CMS computing infrastructure is composed of several subsystems that accomplish complex tasks such as workload and data management, transfers, submission of user and centrally managed production requests. Till recently, most subsystems were monitored through custom tools and web applications, and logging information was scattered over several sources and typically accessible only by experts. In the last year, CMS computing fostered the adoption of common big data solutions based on open-source, scalable, and no-SQL tools, such as Hadoop, InfluxDB, and ElasticSearch, available through the CERN IT infrastructure. Such systems allow for the easy deployment of monitoring and accounting applications using visualisation tools such as Kibana and Grafana. Alarms can be raised when anomalous conditions in the monitoring data are met, and the relevant teams are automatically notified. Data sources from different subsystems are used to build complex workflows and predictive analytics (such as data popularity, smart caching, transfer latency), and for performance studies. We describe the full software architecture and data flow, the CMS computing data sources and monitoring applications, and show how the stored data can be used to gain insights into the various subsystems by exploiting scalable solutions based on Spark.

How Do Job Vacancy Rates Predict Firm Performance? A Web Crawling Massive Data Perspective

Traditionally, the relationship between a firm’s performance and its business strategy is studied using structured data taken from proxy statements and financial reports. However, there have been increasing efforts to explore the linkages between corporate outcomes and unstructured information, such as text or image/audio/video files. Until recently, semi-structured data had been largely overlooked. Given that a substantial amount of such data can be extracted using web crawler techniques and then processed using big data solutions, the current study employed this procedure to investigate whether dynamic job vacancy postings by Taiwanese publicly listed companies are associated with subsequent stock returns and operating ratios. We report that new job openings foreshadow a firm’s operating performance, both indirectly, by boosting stock prices, and directly, by signaling positive developments. This finding remains robust to tests addressing endogeneity concerns and the adoption of alternative specifications. We thus shed light on the role of metadata in financial analysis.

MR-BIRCH: A scalable MapReduce-based BIRCH clustering algorithm

Many classical clustering algorithms have been fitted into MapReduce, which provides a novel solution for clustering big data. However, several iterations are required to reach an acceptable result in most of the algorithms. For each iteration, a new MapReduce job must be executed to load the dataset into main memory, which results in high I/O overhead and poor efficiency. BIRCH algorithm stores only the statistical information of objects with CF entries and CF tree to cluster big data, but with the increase of the tree nodes, the main memory will be insufficient to contain more objects. Hence, BIRCH has to reduce the tree, which will degrade the clustering quality and decelerate the whole execution efficiency. To deal with the problem, BIRCH was fitted into MapReduce called MR-BIRCH in this paper. In contrast to a great number of MapReduce-based algorithms, MR-BIRCH loads dataset only once, and the dataset is processed parallel in several machines. The complexity and scalability were analyzed to evaluate the quality of MR-BIRCH, and MR-BIRCH was compared with Python sklearn BIRCH and Apache Mahout k-means on real-world and synthetic datasets. Experimental results show, most of the time, MR-BIRCH was better or equal to sklearn BIRCH, and it was competitive to Mahout k-means.

Amber

A long-running analytic task on big data often leaves a developer in the dark without providing valuable feedback about the status of the execution. In addition, a failed job that needs to restart from scratch can waste earlier computing resources. An effective method to address these issues is to allow the developer to debug the task during its execution, which is unfortunately not supported by existing big data solutions. In this paper we develop a system called Amber that supports responsive debugging during the execution of a workflow task. After starting the execution, the developer can pause the job at will, investigate the states of the cluster, modify the job, and resume the computation. She can also set conditional breakpoints to pause the execution when certain conditions are satisfied. In this way, the developer can gain a much better understanding of the run-time behavior of the execution and more easily identify issues in the job or data. Amber is based on the actor model, a distributed computing paradigm that provides concurrent units of computation using actors. We give a full specification of Amber, and implement it on top of the Orleans system. Our experiments show its high performance and usability of debugging on computing clusters.

Lake Data Warehouse Architecture for Big Data Solutions

Traditional Data Warehouse is a multidimensional repository. It is nonvolatile, ‎subject-oriented, integrated, time-variant, and non-‎operational data. It is gathered from multiple ‎heterogeneous data ‎sources. We need to adapt traditional Data Warehouse architecture to deal with the new ‎challenges imposed by the abundance of data and the current big data characteristics, containing ‎volume, value, variety, validity, volatility, visualization, variability, and venue. The new ‎architecture also needs to handle existing drawbacks, including availability, scalability, and ‎consequently query performance. This paper introduces a novel Data Warehouse architecture, named Lake ‎Data Warehouse Architecture, to provide the traditional Data Warehouse with the capabilities to ‎overcome the challenges. ‎Lake Data Warehouse Architecture depends on merging the traditional Data Warehouse architecture ‎with big data technologies, like the Hadoop framework and Apache Spark. It provides a hybrid ‎solution in a complementary way. The main advantage of the proposed architecture is that it ‎integrates the current features in ‎traditional Data Warehouses and big data features acquired ‎through integrating the ‎traditional Data Warehouse with Hadoop and Spark ecosystems. Furthermore, it is ‎tailored to handle a tremendous ‎volume of data while maintaining availability, reliability, and ‎scalability.‎