Petabytes Of Data Research Articles

Gravity Wave Momentum Fluxes from 1 km Global ECMWF Integrated Forecast System

Progress in understanding the impact of mesoscale variability, including gravity waves (GWs), on atmospheric circulation is often limited by the availability of global fine-resolution observations and simulated data. This study presents momentum fluxes due to atmospheric GWs extracted from four months of an experimental “nature run", integrated at a 1 km resolution (XNR1K) using the Integrated Forecast System (IFS) model. Helmholtz decomposition is used to compute zonal and meridional flux of vertical momentum from ~1.5 petabytes of data; quantities often emulated by climate model parameterization of GWs. The fluxes are validated using ERA5 reanalysis, both during the first week after initialization and over the boreal winter period from November 2018 to February 2019. The agreement between reanalysis and IFS demonstrates its capability to generate reliable flux distributions and capture mesoscale dynamic variability in the atmosphere. The dataset could be valuable in advancing our understanding of GW-planetary wave interactions, GW evolution around atmospheric extremes, and as high-quality training data for machine learning (ML) simulation of GWs.

Gapr for large-scale collaborative single-neuron reconstruction.

Whole-brain analysis of single-neuron morphology is crucial for unraveling the complex structure of the brain. However, large-scale neuron reconstruction from terabyte and even petabyte data of mammalian brains generated by state-of-the-art light microscopy is a daunting task. Here, we developed 'Gapr' (Gapr accelerates projectome reconstruction) that streamlines deep learning-based automatic reconstruction, 'automatic proofreading' that reduces human workloads at high-confidence sites, and high-throughput collaborative proofreading by crowd users through the Internet. Furthermore, Gapr offers a seamless user interface that ensures high proofreading speed per annotator, on-demand conversion for handling large datasets, flexible workflows tailored to diverse datasets and rigorous error tracking for quality control. Finally, we demonstrated Gapr's efficacy by reconstructing over 4,000 neurons in mouse brains, revealing the morphological diversity in cortical interneurons and hypothalamic neurons. Here, we present Gapr as a solution for large-scale single-neuron reconstruction projects.

Connecting separate monitoring programs through the SoundCoop

Passive acoustic monitoring (PAM) data collection has been growing exponentially, resulting in petabytes of data that document ocean soundscapes, how they change over time, and what animals use these ecosystems at varying timescales. Efficiently extracting this critical information and comparing it to other datasets in the context of ecosystem-based management is a Big Data challenge that traditional desktop processing methods cannot address. The curation, management, and dissemination of PAM datasets is another challenge in need of collaborative progress. To meet these exigencies, a multi-agency funded Sound Cooperative (SoundCoop) project is building community-focused, national cyberinfrastructure capability for PAM data to promote improved, scalable and sustainable accessibility and applications for management and science. Driven by partnerships and framed by four case studies, the SoundCoop has established guidance on the standardized processing of sound level metrics using free software toolkits and begun developing core cyberinfrastructure components that future PAM projects can leverage. U.S. and international scientists contributed PAM data collected across 10 long-term monitoring projects to operationalize the production of hybrid-millidecade spectra across a diversity of labs/instruments. Collectively, the contributed data demonstrate the value of standardized processing that enables the creation of comparable results from disparate monitoring efforts.

Mapping oil palm expansion within the protected lowland rainforest of Nigeria using Google Earth engine

Increasing demand for Elaeis guineensis (African Oil Palm) products both for domestic and industrial use has led to its continuous expansion. The influence of oil palm plantation establishment on the economic well-being of communities and ecosystems cannot be over-emphasised. The study focuses on the rapid expansion of oil palm plantations within all protected areas and forest reserves in the lowland rainforests of Ondo State, Nigeria using. Object-Based Image Analysis (OBIA) was used to map oil palm expansion using 10-metre resolution Sentinel-2A images for 2015 and 2020 in Google Earth Engine (GEE). We found expansion of both smallholder and commercial oil palm plantations within eight of the thirteen protected areas with three protected areas (Ipele, Onisere and Akure Ofosu) showing a significant increase in oil palm plantation establishment. The use of object-based classification techniques, which combines contextual information within the image domain to discriminate landscape features such as oil palm canopy features, was effective in delineating oil palm from the forest canopy and other crops. While Google Earth Engine, a server-based remote sensing domain with petabytes of data, is effective for monitoring large-scale tropical forests.

A Comparative Study of Parallel and Distributed Big Data programming models: Methodologies, Challenges and Future Directions

According to a survey conducted in 2021, users share about 4 petabytes of data on Facebook daily. The exponential increase in data (called big data) plays a vital role in machine learning, internet of things (IoT), and business intelligence applications. Due to the rapid increase in big data, research in big data programming models gained much interest in the past decade. Today, many programming paradigms exist to handle big data, and selecting an appropriate model for a project is critical for its success. This study provides an in-depth analysis of big data programming models such as MapReduce, Directed Acyclic Graph (DAG), Bulk Synchronous Parallel (BSP), and SQL. We conduct a comparative study of distributed and parallel big data programming models and categorize these models into three classes: traditional data processing, graph-based processing, and query-based processing models. Furthermore, we evaluate these programming models based on different parameters like performance, data processing, storage, fault-tolerant, suitable language, and machine learning support. Finally, we highlight the benchmark datasets used for big data programming models and discuss the challenges of models along with future directions for the research community.

Design of Vessel Data Lakehouse with Big Data and AI Analysis Technology for Vessel Monitoring System

The amount of data in the maritime domain is rapidly increasing due to the increase in devices that can collect marine information, such as sensors, buoys, ships, and satellites. Maritime data is growing at an unprecedented rate, with terabytes of marine data being collected every month and petabytes of data already being made public. Heterogeneous marine data collected through various devices can be used in various fields such as environmental protection, defect prediction, transportation route optimization, and energy efficiency. However, it is difficult to manage vessel related data due to high heterogeneity of such marine big data. Additionally, due to the high heterogeneity of these data sources and some of the challenges associated with big data, such applications are still underdeveloped and fragmented. In this paper, we propose the Vessel Data Lakehouse architecture consisting of the Vessel Data Lake layer that can handle marine big data, the Vessel Data Warehouse layer that supports marine big data processing and AI, and the Vessel Application Services layer that supports marine application services. Our proposed a Vessel Data Lakehouse that can efficiently manage heterogeneous vessel related data. It can be integrated and managed at low cost by structuring various types of heterogeneous data using an open source-based big data framework. In addition, various types of vessel big data stored in the Data Lakehouse can be directly utilized in various types of vessel analysis services. In this paper, we present an actual use case of a vessel analysis service in a Vessel Data Lakehouse by using AIS data in Busan area.

EMPIAR: the Electron Microscopy Public Image Archive.

Public archiving in structural biology is well established with the Protein Data Bank (PDB; wwPDB.org) catering for atomic models and the Electron Microscopy Data Bank (EMDB; emdb-empiar.org) for 3D reconstructions from cryo-EM experiments. Even before the recent rapid growth in cryo-EM, there was an expressed community need for a public archive of image data from cryo-EM experiments for validation, software development, testing and training. Concomitantly, the proliferation of 3D imaging techniques for cells, tissues and organisms using volume EM (vEM) and X-ray tomography (XT) led to calls from these communities to publicly archive such data as well. EMPIAR (empiar.org) was developed as a public archive for raw cryo-EM image data and for 3D reconstructions from vEM and XT experiments and now comprises over a thousand entries totalling over 2 petabytes of data. EMPIAR resources include a deposition system, entry pages, facilities to search, visualize and download datasets, and a REST API for programmatic access to entry metadata. The success of EMPIAR also poses significant challenges for the future in dealing with the very fast growth in the volume of data and in enhancing its reusability.

Machine learning applied to big data from marine cabled observatories: A case study of sablefish monitoring in the NE Pacific

Ocean observatories collect large volumes of video data, with some data archives now spanning well over a few decades, and bringing the challenges of analytical capacity beyond conventional processing tools. The analysis of such vast and complex datasets can only be achieved with appropriate machine learning and Artificial Intelligence (AI) tools. The implementation of AI monitoring programs for animal tracking and classification becomes necessary in the particular case of deep-sea cabled observatories, as those operated by Ocean Networks Canada (ONC), where Petabytes of data are now collected each and every year since their installation. Here, we present a machine-learning and computer vision automated pipeline to detect and count sablefish (Anoplopoma fimbria), a key commercially exploited species in the N-NE Pacific. We used 651 hours of video footage obtained from three long-term monitoring sites in the NEPTUNE cabled observatory, in Barkley Canyon, on the nearby slope, and at depths ranging from 420 to 985 m. Our proposed AI sablefish detection and classification pipeline was tested and validated for an initial 4.5 month period (Sep 18 2019-Jan 2 2020), and was a first step towards validation for future processing of the now decade-long video archives from Barkley Canyon. For the validation period, we trained a YOLO neural network on 2917 manually annotated frames containing sablefish images to obtain an automatic detector with a 92% Average Precision (AP) on 730 test images, and a 5-fold cross-validation AP of 93% (± 3.7%). We then ran the detector on all video material (i.e., 651 hours from a 4.5 month period), to automatically detect and annotate sablefish. We finally applied a tracking algorithm on detection results, to approximate counts of individual fishes moving on scene and obtain a time series of proxy sablefish abundance. Those proxy abundance estimates are among the first to be made using such a large volume of video data from deep-sea settings. We discuss our AI results for application on a decade-long video monitoring program, and particularly with potential for complementing fisheries management practices of a commercially important species.

How innovations in methodology offer new prospects for volume electron microscopy

Detailed knowledge of biological structure has been key in understanding biology at several levels of organisation, from organs to cells and proteins. Volume electron microscopy (volume EM) provides high resolution 3D structural information about tissues on the nanometre scale. However, the throughput rate of conventional electron microscopes has limited the volume size and number of samples that can be imaged. Recent improvements in methodology are currently driving a revolution in volume EM, making possible the structural imaging of whole organs and small organisms. In turn, these recent developments in image acquisition have created or stressed bottlenecks in other parts of the pipeline, like sample preparation, image analysis and data management. While the progress in image analysis is stunning due to the advent of automatic segmentation and server‐based annotation tools, several challenges remain. Here we discuss recent trends in volume EM, emerging methods for increasing throughput and implications for sample preparation, image analysis and data management.

A scalable and modular automated pipeline for stitching of large electron microscopy datasets.

Serial-section electron microscopy (ssEM) is the method of choice for studying macroscopic biological samples at extremely high resolution in three dimensions. In the nervous system, nanometer-scale images are necessary to reconstruct dense neural wiring diagrams in the brain, so -called connectomes. The data that can comprise of up to 108 individual EM images must be assembled into a volume, requiring seamless 2D registration from physical section followed by 3D alignment of the stitched sections. The high throughput of ssEM necessitates 2D stitching to be done at the pace of imaging, which currently produces tens of terabytes per day. To achieve this, we present a modular volume assembly software pipeline ASAP (Assembly Stitching and Alignment Pipeline) that is scalable to datasets containing petabytes of data and parallelized to work in a distributed computational environment. The pipeline is built on top of the Render Trautman and Saalfeld (2019) services used in the volume assembly of the brain of adult Drosophila melanogaster (Zheng et al. 2018). It achieves high throughput by operating only on image meta-data and transformations. ASAP is modular, allowing for easy incorporation of new algorithms without significant changes in the workflow. The entire software pipeline includes a complete set of tools for stitching, automated quality control, 3D section alignment, and final rendering of the assembled volume to disk. ASAP has been deployed for continuous stitching of several large-scale datasets of the mouse visual cortex and human brain samples including one cubic millimeter of mouse visual cortex (Yin et al. 2020); Microns Consortium et al. (2021) at speeds that exceed imaging. The pipeline also has multi-channel processing capabilities and can be applied to fluorescence and multi-modal datasets like array tomography.

Machine Learning-Based Intrusion Detection System: An Experimental Comparison

Recently, networks are moving toward automation and getting more and more intelligent. With the advent of big data and cloud computing technologies, lots and lots of data are being produced on the internet. Every day, petabytes of data are produced from websites, social media sites, or the internet. As more and more data are produced, a continuous threat of network attacks is also growing. An intrusion detection system (IDS) is used to detect such types of attacks in the network. IDS inspects packet headers and data and decides whether the traffic is anomalous or normal based on the contents of the packet. In this research, ML techniques are being used for intrusion detection purposes. Feature selection is also used for efficient and optimal feature selection. The research proposes a hybrid feature selection technique composed of the Pearson correlation coefficient and random forest model. For the machine learning (ML) model, decision tree, AdaBoost, and K-nearest neighbor are trained and tested on the TON_IoT dataset. The dataset is new and contains new and recent attack types and features. For deep learning (DL), multilayer perceptron (MLP) and long short-term memory are trained and tested. Evaluation is done on the basis of accuracy, precision, and recall. It is concluded from the results that the decision tree for ML and MLP for DL provides optimal accuracy with fewer false-positive and false-negative rates. It is also concluded from the results that the ML techniques are effective for detecting intrusion in the networks.

On-demand state separation for cloud data warehousing

Moving data analysis and processing to the cloud is no longer reserved for a few companies with petabytes of data. Instead, the flexibility of on-demand resources is attracting an increasing number of customers with small to medium-sized workloads. These workloads do not occupy entire clusters but can run on single worker machines. However, picking the right worker for the job is challenging. Abstracting from worker machines, e.g., using stateless architectures, introduces overheads impacting performance. Solutions without stateless architectures resort to query restarts in the event of an adverse worker matching, wasting already achieved progress. In this paper, we propose migrating queries between workers by introducing on-demand state separation. Using state separation only when required enables maximum flexibility and performance while keeping already achieved progress. To derive the requirements for state separation, we first analyze the query state of medium-sized workloads on the example of TPC-DS SF100. Using this, we analyze the cost and describe the constraints necessary for state separation on such a workload. Furthermore, we describe the design and implementation of on-demand state separation in a compiling database system. Finally, using this implementation, we show the feasibility of our approach on TPC-DS and give a detailed analysis of the cost of query migration and state separation.

Deepfakes Classification of Faces Using Convolutional Neural Networks

In the recent years, petabytes of data is being generated and uploaded online every second. To successfully detect fake contents, a deepfake detection technique is used to determine whether the uploaded content is real or fake. In this paper, a convolutional neural network-based model is proposed to detect the fake face images. The generative adversarial networks and data augmentation are used to generate the face dataset for real and fake face classification. Transfer learning techniques from pretrained deep models such as VGG16 and ResNet50 are employed in the proposed model. The proposed model is evaluated on three benchmark datasets, namely 140k Real and Fake Faces, Real and Fake Face Detection, and Fake Faces. The proposed model attained accuracies over the three datasets are 95.85%, 53.25%, and 88.63%, respectively. Moreover, to improve the obtained results of the proposed model, we combine it with other pretrained models of VGG16 and ResNet50 to construct deep ensembles. The overall performance is greatly improved with the ensemble model achieving accuracies on the three datasets as 98.79%, 75.79%, and 95.52%, respectively. Furthermore, the obtained results also show that the proposed models have superior performance than existing models.

A Heterogeneous Access Metamodel for Efficient IoT Remote Sensing Observation Management: Taking Precision Agriculture as an Example

Standard remote sensing observation (RSO) access and formulization is essential to Internet of Things (IoT) data management, such as in precision agriculture (PA). Because of the heterogeneous characteristics and the petabyte data size of RSO, massive remote sensing processing in RSO management has been hampered. Here, we present a heterogeneous access metamodel for efficient RSO management (HAMERM) and verify it in PA. The structure of basic metadata components is defined. A five-tuple metadata structure based on the metaobject facility is designed. HAMERM consists of identification, platform, observation, product, and access, which represent the five aspects of RSO metadata information. In addition, the flatMap/reduceByKey algorithms and the table structure have been proposed under Sensor Web and Geographic Information Science (GIS) techniques. Intensive experiments in Guangdong Province, China are conducted to test the proposed method. Two RSO metadata formulization instances were conducted to examine the ability of sheltering the differences of multisource and heterogeneous RSO. Experiments containing data storage and data soil moisture (SM) mapping were performed. The results suggest that the HAMERM method achieved a performance 30.1 times higher than that of Hadoop and three times higher than that of Spark (stand-alone). Consequently, the proposed HAMERM can be applied to achieve efficient SM mapping within PA, which is helpful for efficient RSO management for the IoT.

Human and Technical Aspects of Content Regulation

Facebook had 2.8 billion users in 2020, and the company generates four petabytes of data a day. In addition to the legal regulation, one pays less attention to the issues of human moderators and artificial intelligence although the classification of content is done by them through the combined use of artificial intelligence and human intervention. Together they are the first line of defence in the war on unwanted content. The solution is probably not a decision between human moderation and moderation by artificial intelligence but for using them jointly in the future.

Identifying cancer sub-types from genomic scale data sets using confidence based integration (CBI)

Precision medicine is a method involving refined diagnosis of patients and searching for causes that are unseen in their patient cohorts who otherwise have largely similar health conditions. As the technology evolved to extract features from a wide variety of sources including genetics, a large quantum of data is available to the researchers for conducting micro studies in the field of disease and cures. In cancer research, integrative methods using genomic data sets has become a major area of interest. The petabytes of data that is available at The Cancer Genome Atlas (TCGA), a program jointly under NCI and National Human Genome Research Institute, has made possible more nuanced research in cancer genomics. Our method, Confidence Based Integration (CBI) is an integration method to extract similar as well as complementing information from the genomic data sets. This information will provide insight into the status of patients and their prospects. We used the expression data sets of gene, miRNA and DNA methylation in our fusion experiments on five different cancer types. These data sets, after fusion, are clustered using 'Spectral Clustering' algorithm, which derives clusters that form the disease sub types. Survival properties of each sub type demonstrates the reasons to consider the samples inside them highly similar. The performance of CBI, we report, is better, in terms of P-value in log-rank test, than other methods like similarity network fusion or SNF in forming clusters of significance. Individual features clustered extremely poor compared to CBI in most of the experiments.

Big data analytics in Cloud computing: an overview

Big Data and Cloud Computing as two mainstream technologies, are at the center of concern in the IT field. Every day a huge amount of data is produced from different sources. This data is so big in size that traditional processing tools are unable to deal with them. Besides being big, this data moves fast and has a lot of variety. Big Data is a concept that deals with storing, processing and analyzing large amounts of data. Cloud computing on the other hand is about offering the infrastructure to enable such processes in a cost-effective and efficient manner. Many sectors, including among others businesses (small or large), healthcare, education, etc. are trying to leverage the power of Big Data. In healthcare, for example, Big Data is being used to reduce costs of treatment, predict outbreaks of pandemics, prevent diseases etc. This paper, presents an overview of Big Data Analytics as a crucial process in many fields and sectors. We start by a brief introduction to the concept of Big Data, the amount of data that is generated on a daily bases, features and characteristics of Big Data. We then delve into Big Data Analytics were we discuss issues such as analytics cycle, analytics benefits and the movement from ETL to ELT paradigm as a result of Big Data analytics in Cloud. As a case study we analyze Google’s BigQuery which is a fully-managed, serverless data warehouse that enables scalable analysis over petabytes of data. As a Platform as a Service (PaaS) supports querying using ANSI SQL. We use the tool to perform different experiments such as average read, average compute, average write, on different sizes of datasets.

Communication-Efficient Federated Learning With Compensated Overlap-FedAvg

While petabytes of data are generated each day by a number of independent computing devices, only a few of them can be finally collected and used for deep learning (DL) due to the apprehension of data security and privacy leakage, thus seriously retarding the extension of DL. In such a circumstance, federated learning (FL) was proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning with periodic model averaging (FedAvg) introduced massive communication overhead as the synchronized data in each iteration is about the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant proposals focusing on the communication rounds reduction and data compression were proposed to decrease the communication overhead of FL. In this article, we propose Overlap-FedAvg, an innovative framework that loosed the chain-like constraint of federated learning and paralleled the model training phase with the model communication phase (i.e., uploading local models and downloading the global model), so that the latter phase could be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg was further developed with a hierarchical computing strategy, a data compensation mechanism, and a nesterov accelerated gradients (NAG) algorithm. In Particular, Overlap-FedAvg is orthogonal to many other compression methods so that they could be applied together to maximize the utilization of the cluster. Besides, the theoretical analysis is provided to prove the convergence of the proposed framework. Extensive experiments conducting on both image classification and natural language processing tasks with multiple models and datasets also demonstrate that the proposed framework substantially reduced the communication overhead and boosted the federated learning process.

The next wave of passive acoustic data management: How centralized access can enhance science

Passive acoustic data collection has grown exponentially over the past decade resulting in petabytes of data that document our ocean soundscapes. This effort has resulted in two big data challenges: the curation, management, and global dissemination of passive acoustic datasets and efficiently extracting critical information and comparing it to other datasets in the context of ecosystem-based research and management. To address the former, the NOAA National Centers for Environmental Information established a passive acoustic data archive, which contains over 100 TB of audio files mainly collected from stationary recorders throughout waters in the U.S. These datasets are documented with standards-based metadata and are freely available to the public. To begin to address the latter, through standardized processing and centralized stewardship and access, we will present a previously unattainable comparison of first order sound level-patterns from archived data collected across three distinctly separate long-term passive acoustic monitoring efforts conducted at regional and national scales: NOAA/National Park Service Ocean Noise Reference Station Network, the NOPP-funded Atlantic Deepwater Ecosystem Observatory Network, and the NOAA-Navy Sanctuary Soundscape Monitoring Project. Further, we will propose the next frontier for scalable data stewardship, access, and processing flow to help the community collaboratively move forward.

A Reliable Large Distributed Object Store Based Platform for Collecting Event Metadata

The Large Hadron Collider (LHC) is about to enter its third run at unprecedented energies. The experiments at the LHC face computational challenges with enormous data volumes that need to be analysed by thousands of physics users. The ATLAS EventIndex project, currently running in production, builds a complete catalogue of particle collisions, or events, for the ATLAS experiment at the LHC. The distributed nature of the experiment data model is exploited by running jobs at over one hundred Grid data centers worldwide. Millions of files with petabytes of data are indexed, extracting a small quantity of metadata per event, that is conveyed with a data collection system in real time to a central Hadoop instance at CERN. After a successful first implementation based on a messaging system, some issues suggested performance bottlenecks for the challenging higher rates in next runs of the experiment. In this work we characterize the weaknesses of the previous messaging system, regarding complexity, scalability, performance and resource consumption. A new approach based on an object-based storage method was designed and implemented, taking into account the lessons learned and leveraging the ATLAS experience with this kind of systems. We present the experiment that we run during three months in the real production scenario worldwide, in order to evaluate the messaging and object store approaches. The results of the experiment show that the new object-based storage method can efficiently support large-scale data collection for big data environments like the next runs of the ATLAS experiment at the LHC.