GB Dataset Research Articles

Kmer-Node2Vec: a Fast and Efficient Method for Kmer Embedding from the Kmer Co-occurrence Graph, with Applications to DNA Sequences.

Learning low-dimensional continuous vector representation for short k-mers divided from long DNA sequences is key to DNA sequence modeling that can be utilized in many bioinformatics investigations, such as DNA sequence retrieval and classification. DNA2Vec is the most widely used method for DNA sequence embedding. However, it poorly scales to large data sets due to its extremely long training time in kmer embedding. In this paper, we propose a novel efficient graph-based kmer embedding method, named Kmer-Node2Vec, to tackle this concern. Our method converts the large DNA corpus into one kmer co-occurrence graph, and extracts kmer relation on the graph by random walks to learn fast and high-quality kmer embedding. Extensive experiments show that our method is faster than DNA2Vec by 29 times for training on a 4GB data set, and on par with DNA2Vec in terms of task-specific accuracy of sequence retrieval and classification.

A hands-on gaze on HTTP/3 security through the lens of HTTP/2 and a public dataset

Following QUIC protocol ratification on May 2021, the third major version of the Hypertext Transfer Protocol, namely HTTP/3, was published around one year later in RFC 9114. In light of these consequential advancements, the current work aspires to provide a full-blown coverage of the following issues, which to our knowledge have received feeble or no attention in the literature so far. First, we provide a complete review of attacks against HTTP/2, and elaborate on if and in which way they can be migrated to HTTP/3. Second, through the creation of a testbed comprising the at present six most popular HTTP/3-enabled servers, we examine the effectiveness of a quartet of attacks, either stemming directly from the HTTP/2 relevant literature or being entirely new. This scrutiny led to the assignment of at least one CVE ID with a critical base score by MITRE. No less important, by capitalizing on a realistic, abundant in devices testbed, we compiled a voluminous, labeled corpus containing traces of ten diverse attacks against HTTP and QUIC services. An initial evaluation of the dataset mainly by means of machine learning techniques is included as well. Given that the 30 GB dataset is made available in both pcap and CSV formats, forthcoming research can easily take advantage of any subset of features, contingent upon the specific network topology and configuration.

Parallel-FST: A feature selection library for multicore clusters

Feature selection is a subfield of machine learning focused on reducing the dimensionality of datasets by performing a computationally intensive process. This work presents Parallel-FST, a publicly available parallel library for feature selection that includes seven methods which follow a hybrid MPI/multithreaded approach to reduce their runtime when executed on high performance computing systems. Performance tests were carried out on a 256-core cluster, where Parallel-FST obtained speedups of up to 229x for representative datasets and it was able to analyze a 512 GB dataset, which was not previously possible with a sequential counterpart library due to memory constraints.

DRIIS: MapReduce Parameter Optimization of Hadoop Using Genetic Algorithm

In this era of massive knowledge, Hadoop, jointly of the foremost unremarkably used massive processing platforms, features a variety of parameters that are closely associated with resource utilization, particularly mainframe and memory. Calibration of these parameters through improvement may increase Hadoop’s resource utilization. Manual calibration of those parameters is just about not possible, thanks to the time price. Within the massive knowledge business, there’s a requirement to mechanically set up parameters and thereby maximize resource usage. The previous automatic calibration strategies take an extended time to realize the optimum configuration, reducing the cluster’s overall performance. With the help of a novel perceptive procedure, each genetic programming and a genetic rule are supported with a view to enhancing the performance of a Hadoop MapReduce work, we propose an assistant in nursing the best configuration finder. We have a tendency to use the algorithms on top to search out the simplest values for parameter settings. Experiments were performed on four common applications, WordCount, TeraSort, Index and Grep, and eight virtual machines (VMs) in a very typical Hadoop cluster. Our projected methodology will increase MapReduce job potency by 53.63% for a one GB dataset and by 67.4% for a five GB dataset, in keeping with the findings, and by 73.68% on a 10 GB dataset; additionally, for a TeraSort programmed application, MapReduce job potency will increase by 52.62% for an one GB dataset, 61.2% for a five GB dataset, and 55.17% for a 10[Formula: see text]GB dataset. MapReduce jobs boost the performance of Grep applications by 44.4% for a one GB dataset, 56.25% for a five GB dataset, and 49.44% for a 10 GB dataset.

A Novel Intelligent Hybrid Optimized Analytics and Streaming Engine for Medical Big Data.

Medical data processing is exponentially increasing day by day due to the frequent demand for many applications. Healthcare data is one such field, which is dynamically growing day by day. In today's scenario, an enormous amount of sensing devices and data collection units have been employed to generate and collect medical data all over the world. These healthcare devices will result in big real-time data streams. Hence, healthcare-based big data analytics and monitoring have gained hawk-eye importance but needs improvisation. Recently, machine and deep learning algorithms have gained importance to analyze huge amounts of medical data, extract the information, and even predict the future insights of diseases and also cope with the huge volume of data. But applying the learning models to handle big/medical data streams remains to be a challenge among the researchers. This paper proposes the novel deep learning electronic record search engine algorithm (ERSEA) along with firefly optimized long short-term memory (LSTM) model for better data analytics and monitoring. The experimentations have been carried out using Apache Spark using the different medical respiratory data. Finally, the proposed framework results are contrasted with existing models. It shows the accuracy, sensitivity, and specificity like 94%, 93.5%, and 94% for less than 5 GB dataset, and also, more than 5 GB it provides 94%, 92%, and 93% to prove the extraordinary performance of the proposed framework.

Distributed Healthcare Framework Using MMSM-SVM and P-SVM Classification

With the modernization of machine learning techniques in healthcare, different innovations including support vector machine (SVM) have predominantly played a major role in classifying lung cancer, predicting coronavirus disease 2019, and other diseases. In particular, our algorithm focuses on integrated datasets as compared with other existing works. In this study, parallel-based SVM (P-SVM) and multiclass-based multiple submodels (MMSM-SVM) were used to analyze the optimal classification of lung diseases. This analysis aimed to find the optimal classification of lung diseases with id and stages, such as key-value pairs in MapReduce combined with P-SVM and MMSVM for binary and multiclasses, respectively. For non-linear classification, kernel clustering-based SVM embedded with multiple submodels was developed. Both algorithms were developed using Apache spark environment, and data for the analysis were retrieved from microscope lab, UCI, Kaggle, and General Thoracic surgery database along with some electronic health records related to various lung diseases to increase the dataset size to 5 GB. Performance measures were conducted using a 5 GB dataset with five nodes. Dataset size was finally increased, and task analysis and CPU utilization were measured.

CEM500K – A large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning.

Abstract Automated segmentation of cellular electron microscopy (EM) datasets remains a challenge. Supervised deep learning (DL) methods that rely on region-of-interest (ROI) annotations yield models that fail to generalize to unrelated datasets. Newer unsupervised DL algorithms require relevant pre-training images, however, pre-training on currently available EM datasets is computationally expensive and shows little value for unseen biological contexts, as these datasets are large and homogeneous. To address this issue, we present CEM500K, a nimble 25 GB dataset of 0.5 x 106 unique cellular EM images curated from nearly 600 three-dimensional (3D) and 10,000 two-dimensional (2D) images from >100 unrelated imaging projects. We show that models pre-trained on CEM500K learn features that are biologically relevant and resilient to meaningful image augmentations. Critically, we evaluate transfer learning from these pre-trained models on six publicly available and one newly derived benchmark segmentation task and report state-of-the-art results on each. We release the CEM500K dataset, pre-trained models and curation pipeline for model building and further expansion by the EM community. Data and code are available at https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10592/ and https://git.io/JLLTz.

Classifying UAVs With Proprietary Waveforms via Preamble Feature Extraction and Federated Learning

Small unmanned aerial vehicles (UAVs) are deployed in a number of different emerging market segments as well as for recreational hobby flying. Driven by their ubiquitous availability, a large number of manufacturers offer UAV models in different form factors, control and load carrying capacities. This paper proposes a deep learning method to detect the type/model of the UAV using the transmitted RF signals, even when these signals follow proprietary medium access protocols whose headers cannot be decoded. The main contributions are as follows: (i) We show how the preamble portion of the packet is better suited for learning subtle protocol-specific differences, instead of randomly selecting any subset of the transmitted packet, (ii) we propose a pre-processing scheme that generates cross-correlation feature maps to enhance the classification accuracy, (iii) we develop a deep convolutional neural architecture that can be trained in data collected from static scenarios and then tested in practical hovering conditions with 98.2% accuracy of UAV model classification, demonstrating robustness to channel variations, and (iv) we extend this model towards a federated learning paradigm where sensors send individually trained models back to a central controller that combines them, without any appreciable loss of accuracy. Our evaluations are performed on an 8.9 GB dataset collected from static and flying UAVs, which we also release as part of the technical contributions of the work.

CEM500K, a large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning.

Automated segmentation of cellular electron microscopy (EM) datasets remains a challenge. Supervised deep learning (DL) methods that rely on region-of-interest (ROI) annotations yield models that fail to generalize to unrelated datasets. Newer unsupervised DL algorithms require relevant pre-training images, however, pre-training on currently available EM datasets is computationally expensive and shows little value for unseen biological contexts, as these datasets are large and homogeneous. To address this issue, we present CEM500K, a nimble 25 GB dataset of 0.5 × 106 unique 2D cellular EM images curated from nearly 600 three-dimensional (3D) and 10,000 two-dimensional (2D) images from >100 unrelated imaging projects. We show that models pre-trained on CEM500K learn features that are biologically relevant and resilient to meaningful image augmentations. Critically, we evaluate transfer learning from these pre-trained models on six publicly available and one newly derived benchmark segmentation task and report state-of-the-art results on each. We release the CEM500K dataset, pre-trained models and curation pipeline for model building and further expansion by the EM community. Data and code are available at https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10592/ and https://git.io/JLLTz.

Cache-aware volume rendering methods with dynamic data reorganization

We present two cache-aware methods for accelerating volume rendering on graphics processing units (GPUs). The proposed methods extend a previous method such that they improve the worst frame rate typically observed at oblique viewing angles. The key technique for realizing this extension is a dynamic reorganization of the volume data that improves the data locality based on the expected data access pattern. The proposed methods are: (1) a basic method which requires three times the amount of memory to store the volume and (2) an adaptive method which utilizes on-the-fly data reorganization to achieve similar performance without increasing the memory consumption. Experimental results indicate that the GPU texture cache hit rate increased from 58.8 to 71.8% for oblique viewing angles using a 1 GB data set, compared to the previous method. Furthermore, the maximum speedup of the proposed adaptive method was 3.6 $$\times$$ that of the previous method.

A Repository of Vanilla Long-term Integrations of the Solar System

Abstract We share the source code and a 121 GB data set of 96 long-term N-body simulations of the solar system. This data set can be analyzed by itself to study the dynamics of the solar system. In addition, our simulations can be the starting point for future studies wishing to explore different initial conditions or additional physical effects on the solar system. Our simulations can also be used as a comparison and benchmark for new numerical algorithms.

Distributed Subtrajectory Join on Massive Datasets

Joining trajectory datasets is a significant operation in mobility data analytics and the cornerstone of various methods that aim to extract knowledge out of them. In the era of Big Data, the production of mobility data has become massive and, consequently, performing such an operation in a centralized way is not feasible. In this article, we address the problem of Distributed Subtrajectory Join processing by utilizing the MapReduce programming model. Compared to traditional trajectory join queries, this problem is even more challenging since the goal is to retrieve all the “maximal” portions of trajectories that are “similar.” We propose three solutions: (i) a well-designed basic solution, coined DTJb ; (ii) a solution that uses a preprocessing step that repartitions the data, labeled DTJr ; and (iii) a solution that, additionally, employs an indexing scheme, named DTJi . In our experimental study, we utilize a 56GB dataset of real trajectories from the maritime domain, which, to the best of our knowledge, is the largest real dataset used for experimentation in the literature of trajectory data management. The results show that DTJi performs up to 16× faster compared with DTJb , 10× faster than DTJr , and 3× faster than the closest related state-of-the-art algorithm.

Succinct Range Filters

We present the Succinct Range Filter (SuRF), a fast and compact data structure for approximate membership tests. Unlike traditional Bloom filters, SuRF supports both single-key lookups and common range queries. SuRF is based on a new data structure called the Fast Succinct Trie (FST) that matches the point and range query performance of state-of-the-art order-preserving indexes, while consuming only 10 bits per trie node. The false positive rates in SuRF for both point and range queries are tunable to satisfy different application needs. We evaluate SuRF in RocksDB as a replacement for its Bloom filters to reduce I/O by filtering requests before they access on-disk data structures. Our experiments on a 100 GB dataset show that replacing RocksDB's Bloom filters with SuRFs speeds up open-seek (without upper-bound) and closed-seek (with upper-bound) queries by up to 1.5× and 5× with a modest cost on the worst-case (all-missing) point query throughput due to slightly higher false positive rate.

EntropyDB: a probabilistic approach to approximate query processing

We present, an interactive data exploration system that uses a probabilistic approach to generate a small, query-able summary of a dataset. Departing from traditional summarization techniques, we use the Principle of Maximum Entropy to generate a probabilistic representation of the data that can be used to give approximate query answers. We develop the theoretical framework and formulation of our probabilistic representation and show how to use it to answer queries. We then present solving techniques, give two critical optimizations to improve preprocessing time and query execution time, and explore methods to reduce query error. Lastly, we experimentally evaluate our work using a 5 GB dataset of flights within the USA and a 210 GB dataset from an astronomy particle simulation. While our current work only supports linear queries, we show that our technique can successfully answer queries faster than sampling while introducing, on average, no more error than sampling and can better distinguish between rare and nonexistent values. We also discuss extensions that can allow for data updates and linear queries over joins.

DAG-CPM Scheduler for Parallel Execution of Critical Jobs

Map Reduce applications having multiple jobs may be dependent on each other such as iterative Page View application [2] performs the required operation in several iterations before generating the result. Each iteration is considered as single job. Conventional Hadoop MapReduce schedules the jobs sequentially, but not customized to handle multi job application. Also, it will not perform the parallel execution of the dependent jobs. This prolongs the execution time to complete all the jobs. Therefore a new scheduler DAG–CPM Scheduler uses the critical path job scheduling model, to identify the jobs present in the critical path. Critical path job scheduling is optimized to offer support for multi job applications, critical path job is a series of jobs, if execution of a job is delayed, then time required to execute all jobs will be prolonged. DAG–CPM Scheduler schedules multiple jobs by dynamically constructing the job dependency in DAG for the currently running job based on the input and output of a job. DAG represents the dependency among the jobs, this dependency graph is used to insert a pipeline between the output of one job as input for map tasks of another job and it executes the dependent jobs in parallel which results into a substantial reduction in the execution time of an application. Experimental analysis on the proposed approach has been carried out on Page View application on Academic and research web server log file, such as, NASA and rnsit.ac.in of 10 GB data set. PigMix2 is executed on 8GB data set. Experimental results reveal that the average execution time is decreased by 41% compared to Hadoop in respect Page View application and Execution speed is 37.7% faster compared to Pig and DAG–CPM Scheduler can run 24.3% faster when compared to DAG–CPM Scheduler without pipeline.

Understanding Pictograph with Facial Features: End-to-End Sentence-Level Lip Reading of Chinese

With the breakthrough of deep learning, lip reading technologies are under extraordinarily rapid progress. It is well-known that Chinese is the most widely spoken language in the world. Unlike alphabetic languages, it involves more than 1,000 pronunciations as Pinyin, and nearly 90,000 pictographic characters as Hanzi, which makes lip reading of Chinese very challenging. In this paper, we implement visual-only Chinese lip reading of unconstrained sentences in a two-step end-to-end architecture (LipCH-Net), in which two deep neural network models are employed to perform the recognition of Pictureto-Pinyin (mouth motion pictures to pronunciations) and the recognition of Pinyin-to-Hanzi (pronunciations to texts) respectively, before having a jointly optimization to improve the overall performance. In addition, two modules in the Pinyin-to-Hanzi model are pre-trained separately with large auxiliary data in advance of sequence-to-sequence training to make the best of long sequence matches for avoiding ambiguity. We collect 6-month daily news broadcasts from China Central Television (CCTV) website, and semi-automatically label them into a 20.95 GB dataset with 20,495 natural Chinese sentences. When trained on the CCTV dataset, the LipCH-Net model outperforms the performance of all stateof-the-art lip reading frameworks. According to the results, our scheme not only accelerates training and reduces overfitting, but also overcomes syntactic ambiguity of Chinese which provides a baseline for future relevant work.

Analysing billion-objects catalogue interactively: Apache Spark for physicists

Abstract Apache Spark is a Big Data framework for working on large distributed datasets. Although widely used in the industry, it remains rather limited in the academic community or often restricted to software engineers. The goal of this paper is to show with practical uses-cases that the technology is mature enough to be used without excessive programming skills by astronomers or cosmologists in order to perform standard analyses over large datasets, as those originating from future galaxy surveys. To demonstrate it, we start from a realistic simulation corresponding to 10 years of LSST data taking (6 billions of galaxies). Then, we design, optimize and benchmark a set of Spark python algorithms in order to perform standard operations as adding photometric redshift errors, measuring the selection function or computing power spectra over tomographic bins. Most of the commands execute on the full 110 GB dataset within tens of seconds and can therefore be performed interactively in order to design full-scale cosmological analyses. A jupyter notebook summarizing the analysis is available at https://github.com/astrolabsoftware/1807.03078 .

Tera-scale coordinate descent on GPUs

In this work we propose an asynchronous, GPU-based implementation of the widely-used stochastic coordinate descent algorithm for convex optimization. We define the class of problems that can be solved using this method, and show that it includes many popular machine learning applications. For three such applications, we demonstrate at least a 10× training speed-up relative to a state-of-the-art implementation that uses all available resources of a modern CPU. In order to train on very large datasets that do not fit inside the memory of a single GPU, we then consider techniques for distributed learning. We show that while such techniques do not necessarily allow one to achieve further speed-up, they do allow one to train on datasets that would otherwise not fit into memory. We thus propose a distributed learning system that uses the synchronous CoCoA framework to distribute the global optimization across GPUs, and our novel asynchronous algorithm to solve the corresponding local optimizations within each GPU. We benchmark such a system using a 200 GB dataset that consists of 1 billion training examples. We show by scaling out across 16 GPUs, we can train an SVM model to a high degree of accuracy in around 1 min: a 15× speed-up in training time compared to a state-of-the-art CPU-based implementation that uses 640 threads distributed across 8 CPUs.

Taxi-Passenger-Demand Modeling Based on Big Data from a Roving Sensor Network

Investigating passenger demand is essential for the taxicab business. Existing solutions are typically based on offline data collected by manual investigations, which are often dated and inaccurate for real-time analysis. To address this issue, we propose Dmodel, employing roving taxicabs as real-time mobile sensors to (i) infer passenger arriving moments by interactions of vacant taxicabs, and then (ii) infer passenger demand by customized online training with both historical and real-time data. Dmodel utilizes a novel parameter called pickup pattern based on an entropy of pickup events (accounts for various real-world logical information, e.g., bad weather) to reduce the size of big historical taxicab data to be processed. We evaluate Dmodel with a real-world 450 GB dataset of 14,000 taxicabs for a half year, and results show that compared to the ground truth, Dmodel achieves 83 percent accuracy and outperforms a statistical model by 42 percent. We further present an application where Dmodel is used to dispatch vacant taxicabs to achieve an equilibrium between passenger demand and taxicab supply across urban regions.

Probabilistic database summarization for interactive data exploration

We present a probabilistic approach to generate a small, query-able summary of a dataset for interactive data exploration. Departing from traditional summarization techniques, we use the Principle of Maximum Entropy to generate a probabilistic representation of the data that can be used to give approximate query answers. We develop the theoretical framework and formulation of our probabilistic representation and show how to use it to answer queries. We then present solving techniques and give three critical optimizations to improve preprocessing time and query accuracy. Lastly, we experimentally evaluate our work using a 5 GB dataset of flights within the United States and a 210 GB dataset from an astronomy particle simulation. While our current work only supports linear queries, we show that our technique can successfully answer queries faster than sampling while introducing, on average, no more error than sampling and can better distinguish between rare and nonexistent values.