Size Of Big Data Research Articles

Efficient big data assimilation through sparse representation: A 3D benchmark case study in petroleum engineering.

Data assimilation is an important discipline in geosciences that aims to combine the information contents from both prior geophysical models and observational data (observations) to obtain improved model estimates. Ensemble-based methods are among the state-of-the-art assimilation algorithms in the data assimilation community. When applying ensemble-based methods to assimilate big geophysical data, substantial computational resources are needed in order to compute and/or store certain quantities (e.g., the Kalman-gain-type matrix), given both big model and data sizes. In addition, uncertainty quantification of observational data, e.g., in terms of estimating the observation error covariance matrix, also becomes computationally challenging, if not infeasible. To tackle the aforementioned challenges in the presence of big data, in a previous study, the authors proposed a wavelet-based sparse representation procedure for 2D seismic data assimilation problems (also known as history matching problems in petroleum engineering). In the current study, we extend the sparse representation procedure to 3D problems, as this is an important step towards real field case studies. To demonstrate the efficiency of the extended sparse representation procedure, we apply an ensemble-based seismic history matching framework with the extended sparse representation procedure to a 3D benchmark case, the Brugge field. In this benchmark case study, the total number of seismic data is in the order of . We show that the wavelet-based sparse representation procedure is extremely efficient in reducing the size of seismic data, while preserving the salient features of seismic data. Moreover, even with a substantial data-size reduction through sparse representation, the ensemble-based seismic history matching framework can still achieve good estimation accuracy.

Selecting User Influence on Twitter Data Using Skyline Query under MapReduce Framework

The aim of this research was to select and identify user influence on Twitter data. In identification stage, the method proposed in this study was matrix Twitter approach, sentiment analysis, and characterization of the opinion leader. The importan characteristics included external communication, accessibility, and innovation. Based on these characteristics and information from Twitter data through matrix Twitter and sentiment analysis, a algorithm of skyline query was constructed for the selection stage. Algorithm of skyline query selected user influence by comparing with other users according to values of each characteristic. Thus, user influence was indicated as user that was not influenced by other users in any combination of skyline objects. The use of MapReduce framework model in identification and selection stage, support whole operation where Twitter had big size data and rapid changes. The results in identification and selection of user influence exhibited that MapReduce framework minimized the execution time, whereas in parallel skyline query could reveal user influence on the data.

Fast Fourier transform based efficient data processing technique for big data processing speed enhancement in P2P computing environment

Recently, as the mobile phone technology has been developed, apps used on mobile phones have also been developed rapidly in P2P cloud computing environment. In particular, as the IoT(Internet of Things) technology has been grafted on mobile phones, large and small data have been diversified centering on P2P computing environments. However, due to the diverse sizes and uses of big data used in P2P environments, users have many complaints regarding the low accuracy of big data search results and service delay. Previous studies have been actively researched to develop a technology that can cheaply construct and efficiently utilize IT infrastructure for big data processing. In particular, research has been conducted on technologies in which a large number of servers distribute and manage a huge amount of data generated or processed from a portable device such as a smartphone. However, existing researchers do not only store and manage big data, but also need additional data latency and service delay prevention technology to cope with various problems flexibly and to prevent service interruption. Currently, there are many researchers, but until now, there has been no research to satisfy user’s search delay time and accuracy by using size and usage of big data. This paper proposes a fast Fourier transform-based efficient data processing scheme so that users can accurately search for desired data out of different kinds (size, use, type, etc.) of big data in P2P cloud computing environment. The proposed scheme utilizes the keywords used by users to search big data as coefficients of polynomial expressions with a view to enhancing polynomial expression transformation speeds. In addition, the proposed scheme organized the coefficients of polynomial expressions that constitute subnets in pairs with probability values for processing in linkage with each other to enhance data accessibility. In particular, the proposed scheme transforms the vectors shown using the coefficients of polynomial expressions in pairs with the polynomial expressions so that searched data can be quickly identified thereby minimizing user service delay time. As a result of the performance evaluation, the data processing time was improved by 7.3% on average compared with the existing techniques and the server’s data processing rate per unit time was improved by 11.1% on average compared to existing techniques. In addition, according to the data size, the communication delay time between the server and the user improved by an average of 8.9% and server overhead was 10.4% lower than the existing techniques on average.

BDEv 3.0: Energy efficiency and microarchitectural characterization of Big Data processing frameworks

As the size of Big Data workloads keeps increasing, the evaluation of distributed frameworks becomes a crucial task in order to identify potential performance bottlenecks that may delay the processing of large datasets. While most of the existing works generally focus only on execution time and resource utilization, analyzing other important metrics is key to fully understanding the behavior of these frameworks. For example, microarchitecture-level events can bring meaningful insights to characterize the interaction between frameworks and hardware. Moreover, energy consumption is also gaining increasing attention as systems scale to thousands of cores. This work discusses the current state of the art in evaluating distributed processing frameworks, while extending our Big Data Evaluator tool (BDEv) to extract energy efficiency and microarchitecture-level metrics from the execution of representative Big Data workloads. An experimental evaluation using BDEv demonstrates its usefulness to bring meaningful information from popular frameworks such as Hadoop, Spark and Flink.

Big Data and Shipping-managing vessel performance

Shipping business is staggering the trade by a substantial number which portrays the usage of leading technologies to deliver formative and reliable performance to deal with the increasing demand. Technologies like AIS, machine learning, and IoT are making a shift in shipping industry by introducing robots and more sensor equipped devices. The hitch big data originates as a technology which is proficient for assembling and transforming the colossal and divergent figures of data providing organizations with meaningful insights for better decision-making. The size of data is increasing at a higher rate because of the procreation of peripatitic gadgets and sensors attached. Big data is accustomed to delineate technologies and techniques which are used to store, manage, distribute and analyze huge data sheets with a high rate of data occurrence. This gigantic data is allowing to terminate the business by developing meaningful and valuable insights by processing the data. Hadoop is the fundamental basic for composing big data and furnishes with convenient judgments through analysis. It enables the processing of large sets of data by providing a higher degree of fault-tolerance. Parallelism is adapted to process big size of data in the efficient and inexpensive way. Contending massive bulk of data is a determined and vigorous assignment that needs an enormous crunching armature to guaranty affluent data processing and analysis.Â

MINING BIG DATA FOR EFFICIENT DATA RETRIEVAL: A SURVEY

The vast majority of work related to different data handling tasks is confronting a blast in the information that they need to gather and process so as to direct research. This is valid for both: investigative areas managing trial information, e.g. science, human science, stargazing and so on, additionally logical spaces managing reproduction information, e.g. seismology, material science and so on. To augment the potential result of investigative information examination, individual information administration applications need to satisfy the accompanying coarse undertakings: quick on-interest information handling, and compelling stockpiling and merging of various information accumulations. So handling the big data size along with the processing time is the major concern today. In this paper we have concentrated on big data challenges and analyzed the aspects. For this reason we have studies and suggest the future possibilities.

Technological challenges in Magnetic Resonance Imaging: enhancing sensitivity, moving to quantitative imaging and searching for disease biomarkers

Diagnostic imaging based on the Nuclear Magnetic Resonance phenomenon has increasingly spread in the recent few decades, mainly owing to its exquisite capability in depicting a contrast between soft tissues, to its generally non-invasive nature, and to the priceless advantage of using non-ionizing radiation. Magnetic Resonance (MR)-based acquisition techniques allow gathering information on the structure (through Magnetic Resonance Imaging— MRI), the metabolic composition (through Magnetic Resonance Spectroscopy—MRS), and the functioning (through functional MRI —fMRI) of the human body. MR investigations are the methods of choice for studying the brain in vivo, including anatomy, structural wiring and functional connectivity, in healthy and pathological conditions. Alongside the efforts of the clinical research community in extending the acquisition protocols to allow the exploration of a large variety of pathologies affecting diverse body regions, some relevant technological improvements are on the way to maximize the impact of MR in medical diagnostic. The development of MR scanners operating at ultra-high magnetic field (UHF) strength (⩾ 7 tesla), is pushing forward the spatial resolution of MRI and the spectral resolution of MRS, and it is increasing the specificity of fMRI to grey matter signal. UHF MR systems are currently in use for research purposes only; nevertheless, UHF technological advances are positively affecting MR investigations at clinical field strengths. To overcome the current major limitation of MRI, which is mostly based on contrast between tissues rather than on absolute measurements of physical quantities, a new acquisition modality is under development, which is referred as Magnetic Resonance Fingerprinting technique. Finally, as neuroimaging data acquired worldwide are reaching the typical size of Big Data, dedicated technical solutions are required to mine large amount of information and to identify specific biomarkers of pathological conditions.

Una base de datos espacial integrada en un Sistema de Información Geográfica para la gestión del terroir: un nuevo sistema consistente e interactivo

Methodology of the Integrated Zoning of the Terroir involve the generation and management of a high volume of big size data that are interrelated conceptually and spatially, of which its analysis and storage is necessary. These actions are critical in the investigation process and, therefore, there is a special interest in an effectively and efficiently data management. The databases are widely used because of their capabilities on integrity, consistency, redundancy and independence of data management as a method for optimizing resources, so the main goal is to develop a spatial database that facilitates the access, management and analysis of the data of the ZIT. The result is TEZISdb, an acronym for Terroir Zoning Information Service Database, a multilocal and multiscalar modeling that allows to collect thematic and spatial data of all environmental factors from terroir as well as functions for its analysis.

GT-WGS: an efficient and economic tool for large-scale WGS analyses based on the AWS cloud service

BackgroundWhole-genome sequencing (WGS) plays an increasingly important role in clinical practice and public health. Due to the big data size, WGS data analysis is usually compute-intensive and IO-intensive. Currently it usually takes 30 to 40 h to finish a 50× WGS analysis task, which is far from the ideal speed required by the industry. Furthermore, the high-end infrastructure required by WGS computing is costly in terms of time and money. In this paper, we aim to improve the time efficiency of WGS analysis and minimize the cost by elastic cloud computing.ResultsWe developed a distributed system, GT-WGS, for large-scale WGS analyses utilizing the Amazon Web Services (AWS). Our system won the first prize on the Wind and Cloud challenge held by Genomics and Cloud Technology Alliance conference (GCTA) committee. The system makes full use of the dynamic pricing mechanism of AWS. We evaluate the performance of GT-WGS with a 55× WGS dataset (400GB fastq) provided by the GCTA 2017 competition. In the best case, it only took 18.4 min to finish the analysis and the AWS cost of the whole process is only 16.5 US dollars. The accuracy of GT-WGS is 99.9% consistent with that of the Genome Analysis Toolkit (GATK) best practice. We also evaluated the performance of GT-WGS performance on a real-world dataset provided by the XiangYa hospital, which consists of 5× whole-genome dataset with 500 samples, and on average GT-WGS managed to finish one 5× WGS analysis task in 2.4 min at a cost of $3.6.ConclusionsWGS is already playing an important role in guiding therapeutic intervention. However, its application is limited by the time cost and computing cost. GT-WGS excelled as an efficient and affordable WGS analyses tool to address this problem. The demo video and supplementary materials of GT-WGS can be accessed at https://github.com/Genetalks/wgs_analysis_demo.

Service composition in the multi cloud environment

PurposeUser requests over the cloud are not achievable with one single service, multiple services need to be executed to fulfill what a user asks for. Typically, such services are composed and presented as one global service. Moreover, the same operation can be achieved by multiple services available at different clouds, which can result in different possibilities in composing them. This paper aims to decrease the number of clouds involved in the composition process, so that user requests are satisfied with minimal cost (communication costs, execution time and financial charges).Design/methodology/approachThis paper investigates the use of an intelligent water drops (IWDs) optimization-based algorithm, and an integer linear programming model to optimize the number of cloud bases involved in the composition process. A comparison of the solutions found by these two techniques is presented in the paper.FindingsThe obtained results show that the number of cloud bases can be decreased without affecting user satisfaction.Originality/valueThe paper is a first attempt to use the IWDs algorithm for service composition, tested with big-size data.

Taxi-Passenger-Demand Modeling Based on Big Data from a Roving Sensor Network

Investigating passenger demand is essential for the taxicab business. Existing solutions are typically based on offline data collected by manual investigations, which are often dated and inaccurate for real-time analysis. To address this issue, we propose Dmodel, employing roving taxicabs as real-time mobile sensors to (i) infer passenger arriving moments by interactions of vacant taxicabs, and then (ii) infer passenger demand by customized online training with both historical and real-time data. Dmodel utilizes a novel parameter called pickup pattern based on an entropy of pickup events (accounts for various real-world logical information, e.g., bad weather) to reduce the size of big historical taxicab data to be processed. We evaluate Dmodel with a real-world 450 GB dataset of 14,000 taxicabs for a half year, and results show that compared to the ground truth, Dmodel achieves 83 percent accuracy and outperforms a statistical model by 42 percent. We further present an application where Dmodel is used to dispatch vacant taxicabs to achieve an equilibrium between passenger demand and taxicab supply across urban regions.

Cloud computing in a distributed e‐infrastructure using the web processing service standard

SummaryNew Science paradigms have recently evolved to promote open publication of scientific findings as well as multi‐disciplinary collaborative approaches to scientific experimentation. These approaches can face modern scientific challenges but must deal with large quantities of data produced by industrial and scientific experiments. These data, so‐called Big Data, require to introduce new computer science systems to help scientists cooperate, extract information, and possibly produce new knowledge out of the data. E‐infrastructures are distributed computer systems that foster collaboration between users and can embed distributed and parallel processing systems to manage big data. However, in order to meet modern Science requirements, e‐Infrastructures impose several requirements to computational systems in turn, eg, being economically sustainable, managing community‐provided processes, using standard representations for processes and data, managing big data size and heterogeneous representations, supporting reproducible Science, collaborative experimentation, and cooperative online environments, managing security and privacy for data and services. In this paper, we present a cloud computing system (gCube DataMiner) that meets these requirements and operates in an e‐Infrastructure, while sharing characteristics with state‐of‐the‐art cloud computing systems. To this aim, DataMiner uses the web processing service standard of the open geospatial consortium and introduces features like collaborative experimental spaces, automatic installation of processes and services on top of a flexible and sustainable cloud computing architecture. We compare DataMiner with another mature cloud computing system and highlight the benefits our system brings, the new paradigms requirements it satisfies, and the applications that can be developed based on this system.

Smart cloud system with image processing server in diagnosing brain diseases dedicated for hospitals with limited resources.

The use of medical imaging in diagnosing brain disease is growing. The challenges are related to the big size of data and complexity of the image processing. High standard of hardware and software are demanded, which can only be provided in big hospitals. Our purpose was to provide a smart cloud system to help diagnosing brain diseases for hospital with limited infrastructure. The expertise of neurologists was first implanted in cloud server to conduct an automatic diagnosis in real time using image processing technique developed based on ITK library and web service. Users upload images through website and the result, in this case the size of tumor was sent back immediately. A specific image compression technique was developed for this purpose. The smart cloud system was able to measure the area and location of tumors, with average size of 19.91 ± 2.38 cm2 and an average response time 7.0 ± 0.3 s. The capability of the server decreased when multiple clients accessed the system simultaneously: 14 ± 0 s (5 parallel clients) and 27 ± 0.2 s (10 parallel clients). The cloud system was successfully developed to process and analyze medical images for diagnosing brain diseases in this case for tumor.

Imaging plus X: multimodal models of neurodegenerative disease.

Purpose of reviewThis article argues that the time is approaching for data-driven disease modelling to take centre stage in the study and management of neurodegenerative disease. The snowstorm of data now available to the clinician defies qualitative evaluation; the heterogeneity of data types complicates integration through traditional statistical methods; and the large datasets becoming available remain far from the big-data sizes necessary for fully data-driven machine-learning approaches. The recent emergence of data-driven disease progression models provides a balance between imposed knowledge of disease features and patterns learned from data. The resulting models are both predictive of disease progression in individual patients and informative in terms of revealing underlying biological patterns.Recent findingsLargely inspired by observational models, data-driven disease progression models have emerged in the last few years as a feasible means for understanding the development of neurodegenerative diseases. These models have revealed insights into frontotemporal dementia, Huntington's disease, multiple sclerosis, Parkinson's disease and other conditions. For example, event-based models have revealed finer graded understanding of progression patterns; self-modelling regression and differential equation models have provided data-driven biomarker trajectories; spatiotemporal models have shown that brain shape changes, for example of the hippocampus, can occur before detectable neurodegeneration; and network models have provided some support for prion-like mechanistic hypotheses of disease propagation. The most mature results are in sporadic Alzheimer's disease, in large part because of the availability of the Alzheimer's disease neuroimaging initiative dataset. Results generally support the prevailing amyloid-led hypothetical model of Alzheimer's disease, while revealing finer detail and insight into disease progression.SummaryThe emerging field of disease progression modelling provides a natural mechanism to integrate different kinds of information, for example from imaging, serum and cerebrospinal fluid markers and cognitive tests, to obtain new insights into progressive diseases. Such insights include fine-grained longitudinal patterns of neurodegeneration, from early stages, and the heterogeneity of these trajectories over the population. More pragmatically, such models enable finer precision in patient staging and stratification, prediction of progression rates and earlier and better identification of at-risk individuals. We argue that this will make disease progression modelling invaluable for recruitment and end-points in future clinical trials, potentially ameliorating the high failure rate in trials of, e.g., Alzheimer's disease therapies. We review the state of the art in these techniques and discuss the future steps required to translate the ideas to front-line application.

A review on document image analysis techniques directly in the compressed domain

The rapid growth of digital libraries, e-governance, and internet based applications has caused an exponential escalation in the volume of ‘Big-data’ particularly due to texts, images, audios and videos that are being both archived and transmitted on a daily basis. In order to make their storage and transfer efficient, different data compression techniques are used in the literature. The ultimate motive behind data compression is to transform a big size data into small size data, which eventually implies less space while archiving, and less time in transferring. However, in order to operate/analyze compressed data, it is usually necessary to decompress it, so as to bring back the data to its original form, which unfortunately warrants an additional computing cost. In this backdrop, if operating upon the compressed data itself can be made possible without going through the stage of decompression, then the advantage that could be accomplished due to compression would escalate. Further due to compression, from the data structure and storage perspectives, the original visibility structure of the data also being lost, it turns into a potential challenge to trace the original information in the compressed representation. This challenge is the motivation behind exploring the idea of direct processing on the compressed data itself in the literature. The proposed survey paper specifically focuses on compressed document images and brings out two original contributions. The first contribution is that it presents a critical study on different image analysis and image compression techniques, and highlights the motivational reasons for pursuing document image analysis in the compressed domain. The second contribution is that it summarizes the different compressed domain techniques in the literature so far based on the type of compression and operations performed by them. Overall, the paper aims to provide a perspective for pursuing further research in the area of document image analysis and pattern recognition directly based on the compressed data.

Tracing the Spatial-Temporal Evolution of Events Based on Social Media Data

Social media data provide a great opportunity to investigate event flow in cities. Despite the advantages of social media data in these investigations, the data heterogeneity and big data size pose challenges to researchers seeking to identify useful information about events from the raw data. In addition, few studies have used social media posts to capture how events develop in space and time. This paper demonstrates an efficient approach based on machine learning and geovisualization to identify events and trace the development of these events in real-time. We conducted an empirical study to delineate the temporal and spatial evolution of a natural event (heavy precipitation) and a social event (Pope Francis’ visit to the US) in the New York City—Washington, DC regions. By investigating multiple features of Twitter data (message, author, time, and geographic location information), this paper demonstrates how voluntary local knowledge from tweets can be used to depict city dynamics, discover spatiotemporal characteristics of events, and convey real-time information.

An adaptive lack of fit test for big data

abstractNew technological advancements combined with powerful computer hardware and high-speed network make big data available. The massive sample size of big data introduces unique computational challenges on scalability and storage of statistical methods. In this paper, we focus on the lack of fit test of parametric regression models under the framework of big data. We develop a computationally feasible testing approach via integrating the divide-and-conquer algorithm into a powerful nonparametric test statistic. Our theory results show that under mild conditions, the asymptotic null distribution of the proposed test is standard normal. Furthermore, the proposed test benefits from the use of data-driven bandwidth procedure and thus possesses certain adaptive property. Simulation studies show that the proposed method has satisfactory performances, and it is illustrated with an analysis of an airline data.

Magnet: Real-Time Trace Stream Analytics Framework for 5G Operations Support Systems

The era of petabyte data has arrived as the digital big data universe continues its expansion toward exascale with massive volumes of data generated by diverse distributed sources. The size of big data makes it very difficult to gain insight into the meaning of data. In industrial applications, in order to explore both the meaning of data and the complex relationship between data components, big data needs to be processed and reduced enabling further deeper analysis in a timely manner. In this article an integrated data analytics framework is presented designed to extract the set of instances exhibiting statistical dependency from the massive volume of data in a pre-defined quasi real-time manner. The parallel computing model of MapReduce is enhanced to realize Magnet. The solution presented in this article is applicable to the telecommunications market where it optimizes next-generation network management systems for heterogeneous radio access technologies.

Complex Networks Unveiling Spatial Patterns in Turbulence

Numerical and experimental turbulence simulations are nowadays reaching the size of the so-called big data, thus requiring refined investigative tools for appropriate statistical analyses and data mining. We present a new approach based on the complex network theory, offering a powerful framework to explore complex systems with a huge number of interacting elements. Although interest in complex networks has been increasing in the past years, few recent studies have been applied to turbulence. We propose an investigation starting from a two-point correlation for the kinetic energy of a forced isotropic field numerically solved. Among all the metrics analyzed, the degree centrality is the most significant, suggesting the formation of spatial patterns which coherently move with similar vorticity over the large eddy turnover time scale. Pattern size can be quantified through a newly-introduced parameter (i.e. average physical distance) and varies from small to intermediate scales. The network analysis allows a systematic identification of different spatial regions, providing new insights into the spatial characterization of turbulent flows. Based on present findings, the application to highly inhomogeneous flows seems promising and deserves additional future investigation.

Graphing trillions of triangles.

The increasing size of Big Data is often heralded but how data are transformed and represented is also profoundly important to knowledge discovery, and this is exemplified in Big Graph analytics. Much attention has been placed on the scale of the input graph but the product of a graph algorithm can be many times larger than the input. This is true for many graph problems, such as listing all triangles in a graph. Enabling scalable graph exploration for Big Graphs requires new approaches to algorithms, architectures, and visual analytics. A brief tutorial is given to aid the argument for thoughtful representation of data in the context of graph analysis. Then a new algebraic method to reduce the arithmetic operations in counting and listing triangles in graphs is introduced. Additionally, a scalable triangle listing algorithm in the MapReduce model will be presented followed by a description of the experiments with that algorithm that led to the current largest and fastest triangle listing benchmarks to date. Finally, a method for identifying triangles in new visual graph exploration technologies is proposed.